Self-adaptive motion control method and system for cleaning mechanical arm based on reinforcement learning
By adopting an adaptive motion control method based on reinforcement learning, the problems of data processing, scene adaptation, and collaborative execution of the cleaning robot arm were solved, realizing efficient and safe cleaning operations and adapting to the needs of complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN DONGFANG WATER CONSERVANCY MASCH CO LTD
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-12
AI Technical Summary
Existing motion control methods for cleaning robotic arms suffer from problems such as a lack of systematic data processing, poor scene adaptability, a single training method for reinforcement learning models, imperfect collaborative execution logic, simple anomaly handling mechanisms, and fixed reward calculation weights. These issues result in low operating efficiency, poor adaptability, and significant safety hazards, making it difficult to meet the intelligent cleaning needs in complex scenarios.
An adaptive motion control method based on reinforcement learning is adopted. Through multi-source data acquisition and parameterization, scene classification and sub-policy matching, collaborative execution feedback acquisition, dynamic adjustment of motion parameters, and construction of a hierarchical anomaly response mechanism, the model policy is optimized.
The improved adaptability of the cleaning robot arm enhances its operational efficiency, stability, and safety, enabling it to adapt to different work objects and dynamic environments, and achieving multi-objective optimization balance and operational continuity.
Smart Images
Figure CN122018437A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of adaptive motion control technology for robotic arms, specifically to an adaptive motion control method and system for a cleaning robotic arm based on reinforcement learning. Background Technology
[0002] With the increasing demand for environmental protection, robotic arms for cleaning are being used more and more widely in fields such as water cleaning and environmental management. Therefore, there is a need for an adaptive motion control method and system for robotic arms for cleaning based on reinforcement learning.
[0003] Existing motion control methods for cleaning robotic arms mostly rely on fixed parameter programming or simple adaptive logic.
[0004] The existing technology has the following technical problems: 1. Data processing lacks systematicity, and multi-source data has not been scientifically parameterized and fused, resulting in insufficient accuracy of state representation and affecting the reliability of decision-making.
[0005] 2. Poor scenario adaptability; no effective scenario classification and sub-strategy matching mechanism has been established, making it difficult to cope with complex working conditions involving diverse types of work objects and dynamic environmental changes.
[0006] 3. Reinforcement learning models have a single training method, relying solely on offline training, which makes it difficult to adapt to the different scenarios in actual tasks, resulting in insufficient model generalization ability.
[0007] 4. The collaborative execution logic is imperfect. The movement linkage between the robotic arm and the storage unit, navigation system and auxiliary mechanism lacks precise parameter matching, which can easily lead to problems such as dirt falling and work omissions.
[0008] 5. The abnormal handling mechanism is simple, mostly emergency shutdowns triggered by a single threshold, without forming a hierarchical response strategy, which poses a high safety risk and affects the continuity of operations.
[0009] 6. Fixed reward calculation weights make it impossible to dynamically balance multiple objectives such as cleaning efficiency, equipment safety, and energy consumption optimization. These issues result in low operating efficiency, poor adaptability, and significant safety hazards in existing robotic cleaning arms, making it difficult to meet the intelligent cleaning needs in complex scenarios. Summary of the Invention
[0010] To address the aforementioned technical shortcomings, the present invention aims to provide an adaptive motion control method and system for a cleaning robotic arm based on reinforcement learning.
[0011] To solve the above technical problems, the present invention adopts the following technical solution: The present invention provides an adaptive motion control method for a cleaning robot arm based on reinforcement learning, including the following steps: Step 1, data acquisition, processing and analysis: acquire multi-source associated data, analyze the multi-source associated data, and generate parameterized state vectors.
[0012] Step 2, Decision Generation Strategy Matching: Scene classification is completed through feature parameter analysis, the adaptor sub-policy is called as a constraint, and the action parameter instruction is generated by reasoning and operating on the state vector based on the reinforcement learning model.
[0013] Step 3: Collaborative Execution Feedback Acquisition: Drive the robotic arm and associated systems to collaboratively execute the operation according to the parameters. After the operation is executed, collect the feedback parameters and calculate the reward value.
[0014] Step 4: Strategy optimization task termination: Iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.
[0015] Preferably, the generation of action parameter instructions is specifically generated as follows: S1, setting scene classification parameter thresholds: when the proportion of the type of work object reaches the preset type proportion threshold, it is a single type scene; when the obstacle density is lower than the preset obstacle density threshold, it is a simple scene; when the dynamic change rate is lower than the preset change rate threshold, it is a stable scene; when the remaining capacity of the equipment is higher than the preset capacity threshold, it is a normal scene.
[0016] S2. Construct a core scenario sub-strategy parameter library containing a preset number of scenarios: Configure a preset number of parameter items for each scenario. Use the cosine similarity algorithm to calculate the matching degree between the current state parameter vector and the sub-strategy parameter library. If the matching degree reaches the preset high matching degree threshold, directly call the corresponding sub-strategy parameter. If the matching degree is in the preset medium matching degree range, fine-tune the parameter. If the matching degree is lower than the preset low matching degree threshold, call the default sub-strategy parameter.
[0017] S3. Divide the motion parameter dimension system. Discrete motion parameters include preset operation modes and preset operation priorities. Continuous motion parameters include joint angle adjustment and end-effector velocity preset parameter ranges. Classify them according to preset quantization units. Generate confidence scores for various discrete motions and various continuous parameters through correlation analysis. Record a certain type of discrete motion and a certain type of continuous parameter with a confidence score greater than a preset threshold as a pair of correlated parameters. Establish a parameter correlation table to obtain the mapping relationship between discrete and continuous parameters. Input the parameterized state vector into the reinforcement learning model. Determine the discrete parameters through the Softmax function. Use Gaussian distribution sampling combined with the characteristics of the operation object to correct the continuous parameters according to preset correction rules. Generate motion parameter instructions in this way.
[0018] Preferably, feedback parameters are collected after the action is executed. The specific collection process is as follows: T1, linkage between the robotic arm and the storage unit: when the difference between the conveying speed parameter of the robotic arm and the receiving speed parameter of the storage unit exceeds the preset speed difference threshold, the corrected speed parameter is calculated and adjusted synchronously according to the corrected speed parameter. The inlet height parameter of the storage unit is added to the preset height margin to obtain the robotic arm lifting height parameter threshold. When the robotic arm lifting height parameter is greater than the robotic arm lifting height parameter threshold, coordination is performed.
[0019] T2. Robotic arm and navigation system coordination: Divide the preset maximum operating efficiency parameter by the preset robotic arm operating width parameter, and then divide by the preset operating depth parameter to obtain the navigation speed parameter threshold. When the navigation speed parameter is less than or equal to the navigation speed parameter, and the force on one side of the robotic arm exceeds the preset force threshold, the navigation system outputs thrust in the opposite direction, and the adjustment time does not exceed the preset adjustment time.
[0020] T3. Synchronization of robotic arm and auxiliary mechanism: Divide the preset cutting length parameter by the robotic arm conveying speed parameter to obtain the gripping interval of the cutting mechanism. Use the cutting mechanism according to the gripping interval of the cutting mechanism. At the same time, multiply the robotic arm action frequency by the preset frequency multiple to obtain the update frequency of the auxiliary detection mechanism. Use the auxiliary detection mechanism according to the update frequency of the auxiliary detection mechanism.
[0021] T4. Construct a feedback parameter system: Normalize and weight the various data of the cleaning effect parameters to obtain the cleaning effect reward; normalize and weight the various data of the equipment safety parameters to obtain the equipment safety reward; normalize and weight the various data of the energy consumption optimization parameters to obtain the energy consumption optimization reward; normalize and weight the various data of the task progress parameters to obtain the task progress reward. After dynamically adjusting the weights using the entropy weight method, sum them up to obtain the reward value.
[0022] Preferably, the abnormal response mechanism is used to handle the abnormality, and the specific processing procedure is as follows: G1, Adaptive adjustment of action parameters: The action parameters are adjusted according to the size of the work object, the weight of the work object, the distribution density of the work object, the dynamic parameters of the environment, the remaining capacity of the storage unit, the remaining power of the power system, the task completion rate and the priority of the work object.
[0023] G2. The abnormal response mechanism is determined according to the preset abnormal threshold: jamming occurs when the load of the drive unit reaches the preset load threshold and the duration exceeds the preset abnormal duration; collision occurs when the distance detection parameter is lower than the preset safe distance threshold; and overload occurs when the grasping force reaches the preset force threshold. The abnormal index is calculated by normalizing the abnormal duration and the range of influence. The state level is obtained according to the abnormal index range corresponding to the three levels of mild, moderate and severe in the database. When the abnormality is mild, the motion parameters are adjusted according to the preset reverse adjustment ratio. When the abnormality is moderate, the robotic arm is reset to the initial posture. When the abnormality is severe, the motion parameters are reset to zero and an abnormal data packet is sent.
[0024] G3. Model Strategy Iterative Optimization: Samples whose reward values in the feedback parameters reach the preset high reward threshold or fall below the preset low reward threshold are recorded as high-value samples. Samples whose cleaning efficiency improvement reaches the preset efficiency improvement threshold or whose abnormality rate decreases reach the preset abnormality reduction threshold are recorded as valid samples. The sample retention rate is the preset sample retention rate.
[0025] G4. Parameter Iteration: Iteration is initiated when the number of newly added valid samples reaches the preset sample increment threshold, the number of consecutive preset decay cycles of cleaning efficiency decreases to the preset efficiency decay threshold, or the scene parameter similarity is lower than the preset scene similarity threshold. The model weight parameters are updated using the mini-batch gradient descent algorithm to verify the iteration effect. A preset evaluation threshold system is set, and the parameters are saved if the threshold is met.
[0026] On the other hand, the present invention provides an adaptive motion control system for a cleaning robot arm based on reinforcement learning, including the following modules: a data acquisition, processing and analysis module, used to acquire multi-source correlated data, analyze the multi-source correlated data and generate parameterized state vectors.
[0027] The decision generation strategy matching module is used to complete scene classification through feature parameter analysis, call the adaptor sub-policy as a constraint, and generate action parameter instructions based on the reinforcement learning model to perform inference operations on the state vector.
[0028] The collaborative execution feedback acquisition module is used to drive the robotic arm and associated systems to perform work actions in coordination according to parameters. After the action is executed, feedback parameters are collected and reward values are calculated.
[0029] The strategy optimization task termination module is used to iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.
[0030] The beneficial effects of this invention are as follows: 1. This invention first collects multi-source associated data and generates a state vector through parameterization; secondly, it calls the adaptor sub-strategy through scene classification and generates action parameter instructions based on reinforcement learning model inference; then, it drives the robotic arm to work collaboratively with associated systems such as storage units and navigation systems, collects feedback parameters and calculates reward values; finally, it iteratively optimizes the model strategy based on the reward values, dynamically adjusts the action parameters and handles anomalies through a hierarchical anomaly response mechanism. This invention can adapt to different work objects and dynamic environments, improving cleaning efficiency, stability and safety reliability.
[0031] 2. This invention collects multi-source related data such as the state of the robotic arm, the characteristics of the work object, and the state of the equipment. It uses Min-Max normalization, independent coding, and analytic hierarchy process to allocate weights and generate parameterized state vectors. This achieves the systematic fusion of multi-dimensional data, effectively eliminates differences in data dimensions and redundant interference, provides accurate and comprehensive state representation for reinforcement learning model decision-making, and ensures the scientific nature of decision instructions.
[0032] 3. By setting thresholds for multi-dimensional scene classification parameters, a core scene sub-policy parameter library is constructed. Combined with the cosine similarity algorithm, sub-policy matching and fine-tuning are achieved, which significantly reduces the decision search space of the reinforcement learning model and improves the decision response speed. At the same time, the design of the association mapping between discrete and continuous action parameters enables action instructions to be accurately adapted to different work scenarios and object characteristics.
[0033] 4. The reinforcement learning model adopts a two-stage training mode of offline pre-training and online fine-tuning. Offline pre-training lays the foundation for basic operation capabilities through large-scale simulation samples, while online fine-tuning dynamically optimizes parameters based on actual operation data, updating only the parameters of the fully connected layer of the Actor network. This balances model stability and scenario adaptability, effectively improving the model's generalization ability in complex dynamic environments and ensuring cleaning efficiency and operation stability.
[0034] 5. A precise parameter linkage logic was established between the robotic arm and the storage unit, navigation system, and auxiliary mechanisms. Through collaborative mechanisms such as speed synchronization, thrust compensation, and frequency matching, problems such as dirt falling, work omissions, and equipment overturning caused by the independent operation of a single device were avoided. This enabled continuous and efficient cleaning operations, significantly improving the amount of dirt cleaned per unit time and the completeness of work coverage.
[0035] 6. The entropy weight method is used to dynamically adjust the reward weights of four dimensions: cleaning effect, equipment safety, energy consumption optimization, and task progress. The priority of each objective can be flexibly adjusted according to actual working conditions such as remaining power and density of work objects. This ensures cleaning efficiency while reducing energy consumption and equipment malfunction risks, achieving multi-objective optimization balance and adapting to the core operational needs of different scenarios.
[0036] 7. Construct a graded anomaly response mechanism. Through anomaly threshold determination and anomaly index calculation, it achieves accurate classification of mild, moderate and severe anomalies. Differentiated handling strategies are adopted for different levels of anomalies, avoiding the limitations of the traditional single shutdown mode. Under the premise of ensuring equipment and operation safety, it minimizes operation interruption and improves operation continuity and reliability. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention.
[0039] Figure 2 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] according to Figure 1 As shown, the present invention provides an adaptive motion control method for a cleaning robot arm based on reinforcement learning, including the following steps: Step 1, data acquisition, processing and analysis: acquire multi-source associated data, analyze the multi-source associated data, and generate parameterized state vectors.
[0042] In one specific embodiment, the analysis of multi-source correlated data is carried out as follows: collecting robotic arm status parameters, work object feature parameters, and equipment status parameters.
[0043] It should be noted that the state parameters of the robotic arm include, but are not limited to, the angles of each joint, the joint angular velocity, the joint angular acceleration, and the cumulative value of the joint rotation angle.
[0044] The characteristic parameters of the work object include, but are not limited to, the size, weight, density, and hardness of the work object.
[0045] Device status parameters include, but are not limited to, remaining battery power, remaining driving time, charging status, and battery temperature.
[0046] Continuous parameters are mapped to a preset numerical range through Min-Max normalization, while discrete parameters are independently encoded. The weights of the robotic arm state parameters, the work object characteristic parameters, and the equipment state parameters are determined by the analytic hierarchy process, and then weighted summation is used to generate a state vector of a preset dimension.
[0047] It should be noted that the preset value range is set by the staff.
[0048] Step 2, Decision Generation Strategy Matching: Scene classification is completed through feature parameter analysis, the adaptor sub-policy is called as a constraint, and the action parameter instruction is generated by reasoning and operating on the state vector based on the reinforcement learning model.
[0049] In a specific embodiment, the generation process of the action parameter instruction is as follows: S1, setting scene classification parameter thresholds: when the proportion of the operation object type reaches the preset type proportion threshold, it is a single type scene; when the obstacle density is lower than the preset obstacle density threshold, it is a simple scene; when the dynamic change rate is lower than the preset change rate threshold, it is a stable scene; when the remaining capacity of the equipment is higher than the preset capacity threshold, it is a normal scene.
[0050] It should be noted that the threshold is set by the staff.
[0051] S2. Construct a core scenario sub-strategy parameter library containing a preset number of scenarios: Configure a preset number of parameter items for each scenario. Use the cosine similarity algorithm to calculate the matching degree between the current state parameter vector and the sub-strategy parameter library. If the matching degree reaches the preset high matching degree threshold, directly call the corresponding sub-strategy parameter. If the matching degree is in the preset medium matching degree range, fine-tune the parameter. If the matching degree is lower than the preset low matching degree threshold, call the default sub-strategy parameter.
[0052] It should be noted that the parameters are adjusted according to the preset ratio, which is set by the staff. The high matching threshold, medium matching range and low matching threshold are also set by the staff.
[0053] S3. Divide the motion parameter dimension system. Discrete motion parameters include preset operation modes and preset operation priorities. Continuous motion parameters include joint angle adjustment and end-effector velocity preset parameter ranges. Classify them according to preset quantization units. Generate confidence scores for various discrete motions and various continuous parameters through correlation analysis. Record a certain type of discrete motion and a certain type of continuous parameter with a confidence score greater than a preset threshold as a pair of correlated parameters. Establish a parameter correlation table to obtain the mapping relationship between discrete and continuous parameters. Input the parameterized state vector into the reinforcement learning model. Determine the discrete parameters through the Softmax function. Use Gaussian distribution sampling combined with the characteristics of the operation object to correct the continuous parameters according to preset correction rules. Generate motion parameter instructions in this way.
[0054] It should be noted that the preset correction rule is to correct parameters according to a preset ratio, and both the preset threshold and the preset ratio are set by the staff.
[0055] In one specific embodiment, the reinforcement learning model adopts a two-stage training process, which is as follows: a simulation parameter model is constructed, a sample of a preset number of typical scenarios is generated, and the PPO algorithm is used for training. The average reward value and cleaning efficiency parameter of the validation set are calculated every preset number of iterations at a preset validation interval. When the average reward value of the validation set reaches the preset validation reward threshold and the cleaning efficiency parameter reaches the preset validation efficiency threshold, the training is stopped and the model parameters are saved.
[0056] It should be noted that the preset threshold is set by staff.
[0057] For each job that completes a preset number of collection cycles, one set of samples is collected. Fine-tuning is initiated when the sample increment reaches a preset fine-tuning sample threshold, the reward value for a consecutive preset number of low-reward cycles falls below a preset fine-tuning reward threshold, or the rate of change of state parameters reaches a preset state change rate threshold. An incremental learning algorithm is used, and the learning rate is reduced to a preset learning rate range. Only the fully connected layer parameters of the Actor network are updated, while the feature extraction layer parameters are frozen. After fine-tuning, a preset number of verification sample sets of job data are collected continuously. The cleaning efficiency improvement rate and the anomaly rate reduction rate are calculated. When the preset fine-tuning efficiency improvement threshold and the anomaly rate reduction rate reach the preset fine-tuning anomaly reduction threshold are reached, the updated parameters are saved; otherwise, the parameters before adjustment are reverted.
[0058] It should be noted that the parameters are adjusted according to the preset ratio, which is set by the staff. The preset range and the preset threshold are also set by the staff.
[0059] In one specific embodiment, the reinforcement learning model is implemented as follows: the parameterized state vector is input into the Actor network of the reinforcement learning model, the probability distribution of discrete action parameters is calculated by the Softmax function, the parameter combination with the highest probability is selected as the discrete decision result parameter, the initial values of continuous action parameters are generated by Gaussian distribution sampling, and the continuous parameter sequence is obtained by summarizing. The action instruction includes discrete decision results, continuous parameter sequence and temporal parameters.
[0060] Step 3: Collaborative Execution Feedback Acquisition: Drive the robotic arm and associated systems to collaboratively execute the operation according to the parameters. After the operation is executed, collect the feedback parameters and calculate the reward value.
[0061] In a specific embodiment, feedback parameters are collected after the action is executed. The specific collection process is as follows: T1, linkage between the robotic arm and the storage unit: when the difference between the robotic arm conveying speed parameter and the storage unit receiving speed parameter exceeds the preset speed difference threshold, the corrected speed parameter is calculated and adjusted synchronously according to the corrected speed parameter. The storage unit entrance height parameter is added to the preset height margin to obtain the robotic arm lifting height parameter threshold. When the robotic arm lifting height parameter is greater than the robotic arm lifting height parameter threshold, coordination is performed.
[0062] It should be noted that both the preset margin and the preset threshold are set by the staff.
[0063] T2. Robotic arm and navigation system coordination: Divide the preset maximum operating efficiency parameter by the preset robotic arm operating width parameter, and then divide by the preset operating depth parameter to obtain the navigation speed parameter threshold. When the navigation speed parameter is less than or equal to the navigation speed parameter, and the force on one side of the robotic arm exceeds the preset force threshold, the navigation system outputs thrust in the opposite direction, and the adjustment time does not exceed the preset adjustment time.
[0064] It should be noted that the force parameters on one side of the robotic arm are multiplied by the preset thrust coefficient to obtain the reverse output thrust. The preset maximum parameter and the preset thrust coefficient are both set by the staff.
[0065] T3. Synchronization of robotic arm and auxiliary mechanism: Divide the preset cutting length parameter by the robotic arm conveying speed parameter to obtain the gripping interval of the cutting mechanism. Use the cutting mechanism according to the gripping interval of the cutting mechanism. At the same time, multiply the robotic arm action frequency by the preset frequency multiple to obtain the update frequency of the auxiliary detection mechanism. Use the auxiliary detection mechanism according to the update frequency of the auxiliary detection mechanism.
[0066] T4. Construct a feedback parameter system: Normalize and weight the various data of the cleaning effect parameters to obtain the cleaning effect reward; normalize and weight the various data of the equipment safety parameters to obtain the equipment safety reward; normalize and weight the various data of the energy consumption optimization parameters to obtain the energy consumption optimization reward; normalize and weight the various data of the task progress parameters to obtain the task progress reward. After dynamically adjusting the weights using the entropy weight method, sum them up to obtain the reward value.
[0067] The various data of the cleaning effect parameters include, but are not limited to, the weight and volume of dirt captured in a single operation, the amount of dirt removed per unit time, and the coverage area of the operation area.
[0068] Before the operation starts, record the empty weight of the storage box. During the operation, collect the real-time total weight of the storage box at a preset sampling frequency. Divide the cleaning weight of the storage box within a certain time period by the duration of the time period to obtain the cleaning amount per unit time.
[0069] The various data of equipment safety parameters include, but are not limited to, the number of jamming occurrences, jamming duration, number of collisions, collision contact force, and overload operation duration.
[0070] The various data for energy consumption optimization parameters include, but are not limited to, total energy consumption per operation, energy consumption per unit of cleaning volume, energy consumption ratio of robotic arm movement, remaining power battery capacity, power consumption rate, estimated runtime, and charging efficiency.
[0071] The various data for mission progress parameters include, but are not limited to, the completion rate of the work area and the distribution ratio of the unfinished area.
[0072] In one specific embodiment, the entropy weight method dynamically adjusts the weights, and the specific adjustment process is as follows: when the remaining power is lower than the preset low power threshold, the energy consumption optimization reward weight factor is adjusted according to the preset energy consumption weight increase ratio; when the density of the work object reaches the preset high density threshold, the cleaning effect reward is adjusted according to the preset cleaning weight increase ratio; and the weights are recalculated every preset weight recalculation cycle.
[0073] It should be noted that the preset thresholds are all set by staff.
[0074] Step 4: Strategy optimization task termination: Iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.
[0075] In one specific embodiment, the abnormal response mechanism is used to handle the abnormality. The specific processing procedure is as follows: G1, Adaptive adjustment of action parameters: The action parameters are adjusted according to the size of the work object, the weight of the work object, the distribution density of the work object, the dynamic parameters of the environment, the remaining capacity of the storage unit, the remaining power of the power system, the task completion rate and the priority of the work object.
[0076] It should be noted that when the size and weight of the work object reach the preset large and heavy load judgment conditions, the gripping force is adjusted according to the preset force increase ratio, and the moving speed is adjusted according to the preset speed decrease ratio. When the size and distribution density of the work object reach the preset small and dense judgment conditions, the work area is adjusted according to the preset range expansion ratio, and the conveying speed is adjusted according to the preset conveying speed increase ratio. When the environmental dynamic parameters exceed the preset dynamic parameter threshold, the action parameters are adjusted according to the preset compensation coefficient. When the rate of change of dynamic parameters exceeds the preset rate of change threshold, a smooth transition with a preset transition time is initiated. When the remaining capacity of the storage unit is lower than the preset low capacity threshold, the conveying interval is adjusted according to the preset interval shortening ratio. When the remaining power of the power system is lower than the preset low power threshold, the action speed is adjusted according to the preset energy saving and speed reduction ratio, and unnecessary action parameters are turned off. When the task completion rate reaches the preset completion rate threshold, the work coverage accuracy is adjusted according to the preset accuracy increase ratio. When the work object is of the highest priority, the cleaning effect weight is adjusted according to the preset priority weight increase ratio. The adjustment requirements are reassessed every preset adjustment evaluation cycle.
[0077] G2. The abnormal response mechanism is determined according to the preset abnormal threshold: jamming occurs when the load of the drive unit reaches the preset load threshold and the duration exceeds the preset abnormal duration; collision occurs when the distance detection parameter is lower than the preset safe distance threshold; and overload occurs when the grasping force reaches the preset force threshold. The abnormal index is calculated by normalizing the abnormal duration and the range of influence. The state level is obtained according to the abnormal index range corresponding to the three levels of mild, moderate and severe in the database. When the abnormality is mild, the motion parameters are adjusted according to the preset reverse adjustment ratio. When the abnormality is moderate, the robotic arm is reset to the initial posture. When the abnormality is severe, the motion parameters are reset to zero and an abnormal data packet is sent.
[0078] G3. Model Strategy Iterative Optimization: Samples whose reward values in the feedback parameters reach the preset high reward threshold or fall below the preset low reward threshold are recorded as high-value samples. Samples whose cleaning efficiency improvement reaches the preset efficiency improvement threshold or whose abnormality rate decreases reach the preset abnormality reduction threshold are recorded as valid samples. The sample retention rate is the preset sample retention rate.
[0079] G4. Parameter Iteration: Iteration is initiated when the number of newly added valid samples reaches the preset sample increment threshold, the number of consecutive preset decay cycles of cleaning efficiency decreases to the preset efficiency decay threshold, or the scene parameter similarity is lower than the preset scene similarity threshold. The model weight parameters are updated using the mini-batch gradient descent algorithm to verify the iteration effect. A preset evaluation threshold system is set, and the parameters are saved if the threshold is met.
[0080] In one specific embodiment, the iteration initiation triggering condition further includes: pre-setting other triggering scenarios, when updating weight parameters using the mini-batch gradient descent algorithm, the loss function adopts the mean square error formula, the difference between the predicted reward value and the actual reward value is squared and then the mean is calculated to obtain the updated reward offset rate. If the reward offset rate is less than a preset threshold, the updated parameters are saved. If the threshold is not met, the iteration parameters are adjusted or high-value samples are added, and the iteration is re-executed.
[0081] according to Figure 2 As shown, the present invention provides an adaptive motion control system for a cleaning robot arm based on reinforcement learning, comprising the following modules: a data acquisition, processing and analysis module, a decision generation and strategy matching module, a collaborative execution feedback acquisition module, and a strategy optimization and task termination module.
[0082] The decision generation strategy matching module is connected to the data acquisition, processing and analysis module and the collaborative execution feedback acquisition module, respectively, and the strategy optimization task termination module is connected to the collaborative execution feedback acquisition module.
[0083] The data acquisition, processing, and analysis module is used to collect multi-source correlated data, analyze the multi-source correlated data, and generate parameterized state vectors.
[0084] The decision generation strategy matching module is used to complete scene classification through feature parameter analysis, call the adaptor sub-policy as a constraint, and generate action parameter instructions based on the reinforcement learning model to perform inference operations on the state vector.
[0085] The collaborative execution feedback acquisition module is used to drive the robotic arm and associated systems to perform work actions in coordination according to parameters. After the action is executed, feedback parameters are collected and reward values are calculated.
[0086] The strategy optimization task termination module is used to iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.
[0087] The Min-Max normalization, analytic hierarchy process, correlation analysis, Softmax function, and PPO algorithm training described in this invention are all existing technologies that can be found on the Internet, and therefore will not be elaborated further.
[0088] The examples described in this invention are not limited to the specific embodiments listed above. The examples are merely illustrative to facilitate understanding of the invention and do not constitute a limitation on the scope of protection of this invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of this invention should be included within the scope of protection.
[0089] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in this specification, they should all fall within the protection scope of the present invention.
Claims
1. An adaptive motion control method for a cleaning robot arm based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Data Acquisition, Processing and Analysis: Collect multi-source correlated data, analyze the multi-source correlated data, and generate parameterized state vectors; Step 2, Decision Generation Strategy Matching: Scene classification is completed through feature parameter analysis, the adaptor sub-policy is called as a constraint, and the action parameter instruction is generated by reasoning and operating on the state vector based on the reinforcement learning model. Step 3: Collaborative Execution Feedback Acquisition: Drive the robotic arm and associated systems to collaboratively execute the operation according to the parameters. After the operation is executed, collect feedback parameters and calculate the reward value. Step 4: Strategy optimization task termination: Iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.
2. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 1, characterized in that, The analysis of multi-source correlated data is performed as follows: Collect robotic arm status parameters, work object characteristic parameters, and equipment status parameters; Continuous parameters are mapped to a preset numerical range through Min-Max normalization, while discrete parameters are independently encoded. The weights of the robotic arm state parameters, the work object characteristic parameters, and the equipment state parameters are determined by the analytic hierarchy process, and then weighted summation is used to generate a state vector of a preset dimension.
3. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 2, characterized in that, The specific process for generating the action parameter command is as follows: S1. Set the threshold for scene classification parameters: When the proportion of work object type reaches the preset type proportion threshold, it is a single type scene; when the obstacle density is lower than the preset obstacle density threshold, it is a simple scene; when the dynamic change rate is lower than the preset change rate threshold, it is a stable scene; when the remaining equipment capacity is higher than the preset capacity threshold, it is a normal scene. S2. Construct a core scenario sub-strategy parameter library containing a preset number of scenarios: Configure a preset number of parameter items for each scenario. Use the cosine similarity algorithm to calculate the matching degree between the current state parameter vector and the sub-strategy parameter library. If the matching degree reaches the preset high matching degree threshold, directly call the corresponding sub-strategy parameter. If the matching degree is in the preset medium matching degree range, fine-tune the parameter. If the matching degree is lower than the preset low matching degree threshold, call the default sub-strategy parameter. S3. Divide the motion parameter dimension system. Discrete motion parameters include preset operation modes and preset operation priorities. Continuous motion parameters include joint angle adjustment and end-effector velocity preset parameter ranges. Classify them according to preset quantization units. Generate confidence scores for various discrete motions and various continuous parameters through correlation analysis. Record a certain type of discrete motion and a certain type of continuous parameter with a confidence score greater than a preset threshold as a pair of correlated parameters. Establish a parameter correlation table to obtain the mapping relationship between discrete and continuous parameters. Input the parameterized state vector into the reinforcement learning model. Determine the discrete parameters through the Softmax function. Use Gaussian distribution sampling combined with the characteristics of the operation object to correct the continuous parameters according to preset correction rules. Generate motion parameter instructions in this way.
4. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 3, characterized in that, The reinforcement learning model employs a two-stage training process, as follows: Construct a simulation parameter model, generate a sample of a preset number of typical scenarios with preset scenario combinations, train the PPO algorithm, calculate the average reward value and cleaning efficiency parameter of the validation set at preset validation intervals in each iteration, and stop training and save the model parameters when the average reward value of the validation set reaches the preset validation reward threshold and the cleaning efficiency parameter reaches the preset validation efficiency threshold. For each job that completes a preset number of collection cycles, one set of samples is collected. Fine-tuning is initiated when the sample increment reaches a preset fine-tuning sample threshold, the reward value for a consecutive preset number of low-reward cycles falls below a preset fine-tuning reward threshold, or the rate of change of state parameters reaches a preset state change rate threshold. An incremental learning algorithm is used, and the learning rate is reduced to a preset learning rate range. Only the fully connected layer parameters of the Actor network are updated, while the feature extraction layer parameters are frozen. After fine-tuning, a preset number of verification sample sets of job data are collected continuously. The cleaning efficiency improvement rate and the anomaly rate reduction rate are calculated. When the preset fine-tuning efficiency improvement threshold and the anomaly rate reduction rate reach the preset fine-tuning anomaly reduction threshold are reached, the updated parameters are saved; otherwise, the parameters before adjustment are reverted.
5. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 3, characterized in that, The reinforcement learning model is implemented as follows: The parameterized state vector is input into the Actor network of the reinforcement learning model. The probability distribution of discrete action parameters is calculated by the Softmax function. The parameter combination with the highest probability is selected as the discrete decision result parameter. The initial values of continuous action parameters are generated by sampling through Gaussian distribution. The continuous parameter sequence is obtained by summarizing. The action command includes discrete decision results, continuous parameter sequence and time sequence parameters.
6. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 1, characterized in that, After the action is performed, feedback parameters are collected. The specific collection process is as follows: T1. Linkage between robotic arm and storage unit: When the difference between the conveying speed parameter of the robotic arm and the receiving speed parameter of the storage unit exceeds the preset speed difference threshold, the corrected speed parameter is calculated and adjusted synchronously according to the corrected speed parameter. The preset height margin is added to the entrance height parameter of the storage unit to obtain the lifting height parameter threshold of the robotic arm. When the lifting height parameter of the robotic arm is greater than the lifting height parameter threshold of the robotic arm, coordination is performed. T2. Robotic arm and navigation system coordination: Divide the preset maximum operating efficiency parameter by the preset robotic arm operating width parameter, and then divide by the preset operating depth parameter to obtain the navigation speed parameter threshold. When the navigation speed parameter is less than or equal to the navigation speed parameter, and the force on one side of the robotic arm exceeds the preset force threshold, the navigation system outputs thrust in the opposite direction, and the adjustment time does not exceed the preset adjustment time. T3. Synchronization between the robotic arm and the auxiliary mechanism: Divide the preset cutting length parameter by the robotic arm conveying speed parameter to obtain the gripping interval of the cutting mechanism. Use the cutting mechanism according to the gripping interval of the cutting mechanism. At the same time, multiply the robotic arm action frequency by the preset frequency multiple to obtain the update frequency of the auxiliary detection mechanism. Use the auxiliary detection mechanism according to the update frequency of the auxiliary detection mechanism. T4. Construct a feedback parameter system: Normalize and weight the various data of the cleaning effect parameters to obtain the cleaning effect reward; normalize and weight the various data of the equipment safety parameters to obtain the equipment safety reward; normalize and weight the various data of the energy consumption optimization parameters to obtain the energy consumption optimization reward; normalize and weight the various data of the task progress parameters to obtain the task progress reward. After dynamically adjusting the weights using the entropy weight method, sum them up to obtain the reward value.
7. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 6, characterized in that, The entropy weight method dynamically adjusts the weights, and the specific adjustment process is as follows: When the remaining power is lower than the preset low power threshold, the energy consumption optimization reward weight factor is adjusted according to the preset energy consumption weight increase ratio. When the density of the work object reaches the preset high density threshold, the cleaning effect reward is adjusted according to the preset cleaning weight increase ratio. The weight is recalculated every preset weight recalculation cycle.
8. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 1, characterized in that, The exception response mechanism handles exceptions, and the specific processing procedure is as follows: G1. Adaptive adjustment of motion parameters: Adjust motion parameters based on the size of the work object, the weight of the work object, the distribution density of the work object, environmental dynamic parameters, the remaining capacity of the storage unit, the remaining power of the power system, the task completion rate, and the priority of the work object. G2. The abnormal response mechanism is determined according to the preset abnormal threshold: jamming is when the load of the drive unit reaches the preset load threshold and the duration exceeds the preset abnormal duration; collision is when the distance detection parameter is lower than the preset safe distance threshold; overload is when the grasping force reaches the preset force threshold. The abnormal index is calculated by normalizing the abnormal duration and the impact range. The state level is obtained according to the abnormal index range corresponding to the three levels of mild, moderate and severe in the database. When the abnormality is mild, the motion parameters are adjusted according to the preset reverse adjustment ratio. When the abnormality is moderate, the robotic arm is reset to the initial posture. When the abnormality is severe, the motion parameters are reset to zero and an abnormal data packet is sent. G3. Model strategy iterative optimization: Samples whose reward values in the feedback parameters reach the preset high reward threshold or are lower than the preset low reward threshold are recorded as high-value samples. Samples whose cleaning efficiency improvement reaches the preset efficiency improvement threshold or whose abnormality rate decreases reach the preset abnormality reduction threshold are recorded as valid samples. The sample retention rate is the preset sample retention rate. G4. Parameter Iteration: Iteration is initiated when the number of newly added valid samples reaches the preset sample increment threshold, the number of consecutive preset decay cycles of cleaning efficiency decreases to the preset efficiency decay threshold, or the scene parameter similarity is lower than the preset scene similarity threshold. The model weight parameters are updated using the mini-batch gradient descent algorithm to verify the iteration effect. A preset evaluation threshold system is set, and the parameters are saved if the threshold is met.
9. The adaptive motion control method for a cleaning robot based on reinforcement learning according to claim 1, characterized in that, The iteration initiation triggering conditions also include: In other pre-defined trigger scenarios, when updating weight parameters using the mini-batch gradient descent algorithm, the loss function uses the mean squared error formula. The difference between the predicted reward value and the actual reward value is squared and then averaged to obtain the updated reward offset rate. If the reward offset rate is less than a preset threshold, the updated parameters are saved. If the threshold is not met, the iteration parameters are adjusted or high-value samples are added, and the iteration is re-executed.
10. A motion control system utilizing the reinforcement learning-based adaptive motion control method for a cleaning robot arm according to any one of claims 1-9, characterized in that, Includes the following modules: The data acquisition, processing, and analysis module is used to collect multi-source correlated data, analyze the multi-source correlated data, and generate parameterized state vectors. The decision generation strategy matching module is used to complete scene classification through feature parameter analysis, call the adaptor sub-policy as a constraint, and generate action parameter instructions based on the reinforcement learning model to reason about the state vector. The collaborative execution feedback acquisition module is used to drive the robotic arm and associated systems to collaboratively execute work actions according to parameters. After the action is executed, feedback parameters are collected and reward values are calculated. The strategy optimization task termination module is used to iteratively optimize the model strategy based on the reward value and feedback results, dynamically adjust the action parameters based on changes in feature parameters, and handle anomalies through an anomaly response mechanism.