Control method of chemical powder grinding robot based on reinforcement learning

Abstract

The application discloses a control method of a chemical powder grinding mechanical arm based on reinforcement learning, and belongs to the field of grinding robot control, and comprises the following steps: step 1, establishing a mortar inner surface model of a grinding platform and a mobility control module, and collecting powder information; step 2, constructing a heuristic strategy and a reward function for a powder grinding scene, and executing the heuristic strategy to obtain an offline data set; step 3, preliminarily training a strategy network of a strategy model of a reinforcement learning module by using the offline data set and an offline reinforcement learning algorithm; step 4, deploying the strategy model on a mechanical arm controller, continuously collecting real-time vision and torque feedback by using a visual feedback module and the mobility control module, and incrementally updating and optimizing the strategy by using an online reinforcement learning algorithm; and step 5, generating a grinding track according to the optimized optimal strategy, and driving the mechanical arm to execute powder grinding, and stopping until a visual feedback target is reached. The method can efficiently and adaptively control the mechanical arm to grind powder.

Description

Technical Field

[0001] This invention relates to the field of control technology for grinding robots in powder preparation, and more particularly to a control method for a chemical powder grinding robotic arm based on reinforcement learning. Background Technology

[0002] With the rapid growth in demand for high-throughput experiments and complex chemical synthesis, automation in chemical laboratories has become an inevitable trend for improving experimental efficiency, reducing labor costs, and ensuring operational safety. In the preparation of solid reagents, the grinding process has a crucial impact on particle size uniformity and subsequent reactivity: traditional ball mills, due to their large minimum grinding jar volume (50–200 mL), are insufficient for grinding milligram-level or smaller samples; while manual grinding, although suitable for small sample sizes, is time-consuming, inefficient, and has poor repeatability, making it difficult to integrate with automated platforms. In recent years, some studies have attempted to introduce robotic arm platforms, acquiring grinding status through multimodal sensing such as vision and acoustics, and utilizing elastic clamps and flexible tools for safety protection. However, their optimization strategies often rely on experience or online manual parameter tuning, lacking a performance feedback loop directly coupled with actual grinding quality (such as particle size distribution and fineness indicators). This not only restricts continuous strategy improvement but also makes it difficult to address the complex kinetic challenges brought about by changes in the characteristics of the ground materials and operating conditions.

[0003] Reinforcement learning-based methods can achieve adaptive control and real-time optimization of nonlinear, coupled systems by pre-training initial policies in an offline environment and combining them with online fine-tuning. However, there is currently no reinforcement learning-based control method for using robotic arms in chemical laboratories for powder grinding tasks, resulting in low grinding efficiency and poor generalization ability in chemical powder grinding experiments.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a control method for a chemical powder grinding robotic arm based on reinforcement learning. Under the premise of ensuring grinding safety, this method can achieve efficient, precise and adaptive control of the robotic arm for small-batch powder grinding tasks by combining offline pre-training and online fine-tuning. This improves the grinding efficiency and generalization ability of the robotic arm in chemical powder grinding experiments, thereby solving the technical problems of low strategy optimization efficiency, lack of performance closed-loop feedback and difficulty in adapting to changes in materials and working conditions in existing methods.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A control method for a chemical powder grinding robotic arm based on reinforcement learning, comprising:

[0008] Step 1: Establish a model of the inner surface of the mortar of the grinding platform, construct the admittance control module of the controller for the robotic arm based on the model of the inner surface of the mortar, and collect powder information in the grinding space inside the mortar.

[0009] Step 2: Construct a heuristic policy and reward function for the powder grinding scenario, and use the admittance control module to control the robotic arm to execute the heuristic policy to obtain an offline dataset;

[0010] Step 3: Based on the offline dataset obtained in Step 2, use the offline reinforcement learning algorithm to perform preliminary training on the policy network of the policy model of the reinforcement learning module to obtain a policy model with initial polishing capability.

[0011] Step 4: Deploy the strategy model with initial grinding capability obtained in Step 3 onto the controller of the robotic arm. The visual feedback module and admittance control module of the controller of the robotic arm continuously collect real-time visual feedback and torque feedback, respectively. The strategy network of the strategy model of the reinforcement learning module is incrementally updated and optimized using an online reinforcement learning algorithm.

[0012] Step 5: The admittance control module generates a grinding trajectory in real time and drives the robotic arm to perform powder grinding operation based on the optimal strategy of the strategy model obtained after online fine-tuning, until the visual feedback reaches the set target and then stops.

[0013] Compared with existing technologies, the control method for a chemical powder grinding robotic arm based on reinforcement learning provided by this invention has the following advantages:

[0014] By combining offline reinforcement learning with online reinforcement learning fine-tuning, the task execution effect can be improved as much as possible while ensuring the efficiency of policy training. Online reinforcement learning can respond to changes in material characteristics and working conditions in real time, and achieve adaptive policy optimization. Finally, the grinding effect of the robotic arm is significantly better than that of pure heuristic or offline reinforcement learning methods in terms of particle size uniformity and fineness. Moreover, it is convenient for modular design of the framework, which can be flexibly deployed on grinding platforms of different robotic arms, and has good versatility and engineering promotion value. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a control method for a chemical powder grinding robotic arm based on reinforcement learning, provided in an embodiment of the present invention.

[0017] Figure 2 This is a flowchart illustrating the control method for a chemical powder grinding robotic arm based on reinforcement learning provided in this embodiment of the invention.

[0018] Figure 3 This is a side view of the inner surface model of the mortar in the control method provided in this embodiment of the invention.

[0019] Figure 4 This is a top view of the inner surface model of the mortar in the control method provided in this embodiment of the invention.

[0020] Figure 5 This is a schematic diagram of the system architecture used in the control method of the chemical powder grinding robot arm based on reinforcement learning provided in the embodiments of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] First, the following explanations are provided for the terms that may be used in this article:

[0023] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0024] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0025] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0026] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0027] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0028] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0029] The solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0030] like Figure 1 , Figure 2 As shown, this invention provides a control method for a chemical powder grinding robotic arm based on reinforcement learning, used to control the robotic arm on a grinding platform to hold a grinding pestle and perform chemical powder grinding, including:

[0031] Step 1: Establish a model of the inner surface of the mortar of the grinding platform, construct the admittance control module of the controller for the robotic arm based on the model of the inner surface of the mortar, and collect powder information in the grinding space inside the mortar.

[0032] Step 2: Construct a heuristic policy and reward function for the powder grinding scenario, and use the admittance control module to control the robotic arm to execute the heuristic policy to obtain an offline dataset;

[0033] Step 3: Based on the offline dataset obtained in Step 2, use the offline reinforcement learning algorithm to perform preliminary training on the policy network of the policy model of the reinforcement learning module to obtain a policy model with initial polishing capability.

[0034] Step 4: Deploy the strategy model with initial grinding capability obtained in Step 3 onto the controller of the robotic arm. The visual feedback module and admittance control module of the controller of the robotic arm continuously collect real-time visual feedback and torque feedback, respectively. The strategy network of the strategy model of the reinforcement learning module is incrementally updated and optimized using an online reinforcement learning algorithm.

[0035] Step 5: The admittance control module generates a grinding trajectory in real time and drives the robotic arm to perform powder grinding operation based on the optimal strategy of the strategy model obtained after online fine-tuning, until the visual feedback reaches the set target and then stops.

[0036] Preferably, in step 1 of the above method, the inner surface model of the mortar of the grinding platform is established in the following manner, the admittance control module of the controller for controlling the robotic arm is constructed based on the inner surface model of the mortar, and powder information in the grinding space inside the mortar is collected, including:

[0037] Step 11: Use a semi-circular curved surface to approximate the inner surface of the mortar on the grinding platform to establish a model of the inner surface of the mortar.

[0038] Step 12: Construct the admittance control module of the controller for the robotic arm by establishing the inner surface model of the mortar. The admittance control module adjusts the grinding trajectory of the robotic arm through real-time force feedback based on the grinding reference trajectory.

[0039] Step 13: Construct a color range threshold for the powder by using the local color features of the powder image in the grinding space of the mortar. Perform binarization processing on the powder image according to the color range threshold to obtain the powder distribution image, and save the saturation of the powder distribution area in the powder distribution image.

[0040] Step 14: Divide the powder distribution image into several non-overlapping regions according to the row and column directions with a preset grid size, and calculate the average gray value of each region as the scoring index of the corresponding region. Combine the saturation of each region as the powder information in the grinding space.

[0041] See Figure 3 , Figure 4 Preferably, the mortar inner surface model established in step 11 above Represented as:

[0042] ;

[0043] Among them, the inner surface model of the mortar coordinates on; The radius of the model of the inner surface of the mortar; The depth inside the mortar;

[0044] During the grinding process where the robotic arm holds the grinding pestle, the end of the pestle remains on and perpendicular to the inner surface of the mortar to achieve thorough grinding. In this grinding motion, for a grinding pestle initially positioned with its centerline parallel to the z-axis, the Euler angle of the x-axis... Euler angles with the y-axis The calculations are as follows:

[0045] ;

[0046] ;

[0047] Angle between the center line of the grinding pestle and the z-axis ; where r j The radius of the robot arm's end position in the top view of the mortar is the polar coordinate radius, which is the distance between the robot arm's end position and the center of the circle in the top view of the mortar; q represents the polar coordinate angle of the robot arm's end position in the top view of the mortar, which is the angle between the projection of the grinding pestle's center line in the top view of the mortar and the positive x-axis.

[0048] Based on the angle between the centerline of the grinding pestle and the z-axis mortar inner surface model coordinates on , , The calculations are as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] Where L is the length of the grinding pestle;

[0053] The above approximates the grinding working space on the inner surface of the mortar as a circle, which can be determined by using the polar coordinate radius r as a polar coordinate parameter. j The polar coordinate angle q can uniquely determine the grinding point position of the grinding pestle in the mortar, within the polar coordinate radius r. j The reference grinding trajectory is calculated during the dynamic change of the polar coordinate angle q.

[0054] Preferably, in step 12 above, the expression for the constructed admittance control module is:

[0055] ;

[0056] Among them, parameters M, B, and K represent the mass, damping, and stiffness of the admittance control module, respectively; , These represent the components of the actual and desired positions of the robotic arm's end effector in Cartesian space along the x-axis, y-axis, or z-axis, respectively. , These represent the rate of change of the actual position and the rate of change of the desired position at the end of the robotic arm, respectively. , These represent the acceleration due to the actual position change and the acceleration due to the desired position change of the robotic arm's end effector, respectively. , These represent the actual contact force and the expected contact force at the end of the robotic arm, respectively.

[0057] make and Let represent the error in the position of the robotic arm's end effector and the force error, respectively. Then, in the backward difference form of the discrete system, the error acceleration of the robotic arm's end effector position at time j is... The calculation is as follows:

[0058] ;

[0059] and the actual position of the robotic arm's end effector at time j. The calculation is as follows:

[0060] ;

[0061] ;

[0062] ;

[0063] in, Let j represent the desired position of the robotic arm's end effector at time j; This represents the actual position of the robotic arm's end effector at time j; This represents the actual position of the robotic arm's end effector at time j-1; The error velocity of the robotic arm's end effector at time j-1 in the backward differential form is represented by T; T represents the control period. This represents the error velocity of the robotic arm's end effector at time j in the backward difference form.

[0064] By combining the admittance control module and the real-time acquisition of the contact force at the end of the robotic arm, the target position at the end can be solved, so that the contact between the end of the robotic arm and the outside world exhibits the properties of mass-spring-damping.

[0065] By adjusting the stiffness K of the admittance control module and the desired contact force It enables the end of the robotic arm to make contact, squeeze, and rub against the inner surface of the mortar in a predetermined state.

[0066] Preferably, in step 13 above, the color range threshold of the powder is constructed using the local color features of the powder image in the grinding space within the mortar, and the powder image is binarized based on the powder color range threshold to obtain a powder distribution image. The saturation of the powder distribution region in the powder distribution image is then saved, including:

[0067] The RGB image of the mortar on the grinding platform, captured by a color depth camera set on the grinding platform, is used as the powder image. Local color features of the powder are extracted from the HSV image converted from the RGB image. The powder aggregation area in the HSV image is sampled to obtain the sampling area. The hue mean of the powder color in the sampling area is calculated to obtain the color interval threshold of the powder in the current mortar. The HSV image is binarized according to the color interval threshold to obtain the powder distribution mask. The original RGB image is processed according to the powder distribution mask to obtain the powder distribution image, and the saturation of the powder distribution area in the powder distribution image is saved.

[0068] Further, in step 13 above, local color features of the powder are extracted from the HSV image after conversion from the RGB image as follows: sampling is performed on the powder aggregation area in the HSV image to obtain the sampling area; the hue mean of the powder color within the sampling area is calculated to obtain the color interval threshold of the powder in the current mortar; the HSV image is binarized according to the color interval threshold to obtain the powder distribution mask; the original RGB image is processed according to the powder distribution mask to obtain the powder distribution image; and the saturation of the powder distribution area in the powder distribution image is saved, including:

[0069] Normalize the (R,G,B) value of each pixel in an RGB image to a (r,g,b) value using the following formula:

[0070] ;

[0071] Calculate the following intermediate parameters using the obtained (r, g, b) values:

[0072] ;

[0073] ;

[0074] ;

[0075] Using the intermediate parameters obtained above , , The values ​​of lightness (V), saturation (S), and hue (H) are calculated as follows:

[0076] ;

[0077] ;

[0078] ;

[0079] In HSV images Sampling was performed on the powder accumulation area. Let Q represent the set of real numbers, and let Q be the defined sampling region:

[0080] ;

[0081] in, p is the pixel coordinate of the center of the inner surface of the mortar in the HSV image; p is the pixel coordinate of the sampling point in the HSV image. ; Ω represents the pixel radius of the sampling region Q; Ω is the set of powder distribution masks within the sampling region Q, determined by the pixel coordinates within the sampling region Q. Powder distribution mask of corresponding sampling points composition;

[0082] The hue H at each sampling point Calculate the hue mean within the sampling region Q. :

[0083] ;

[0084] Set hue deviation threshold Saturation deviation threshold and brightness deviation threshold The hue range threshold that constitutes the color range threshold of the powder Saturation range threshold and brightness range threshold They are respectively:

[0085] ;

[0086] ;

[0087] ;

[0088] in, The average saturation value within the sampling region Q. The mean lightness value within the sampling region Q, and the hue interval threshold. In This represents the lower limit of the hue range. The upper limit of the hue range; the threshold of the saturation range. In This represents the lower limit of the saturation range. The upper limit of the saturation range; the threshold of the brightness range. In This represents the lower limit of the brightness range. This represents the upper limit of the brightness range;

[0089] The HSV image is binarized according to the above color range thresholds for the powder to obtain pixel coordinates. Powder distribution mask of corresponding sampling points for:

[0090] ;

[0091] in, Represents pixel coordinates The hue value of the corresponding sampling point; Represents pixel coordinates The saturation value of the corresponding sampling point; Represents pixel coordinates The corresponding brightness value of the sampling point;

[0092] The original image is processed based on the powder distribution mask obtained above to obtain a binarized powder distribution image;

[0093] All elements in the binarized powder distribution image that satisfy the powder distribution mask are... saturation value of pixels Saturation matrix is ​​formed by summing the results. The saturation of the powder distribution area in the powder distribution image is used as the basis for data storage in the visual data storage unit, and is used for subsequent reward calculation and policy training.

[0094] Preferably, in step 14 above, the powder distribution image obtained in step 13 is divided into several non-overlapping regions according to the row and column directions and a preset grid size, and the average gray value of each region is calculated as the scoring index for the corresponding region, including:

[0095] Let the grayscale value of a pixel in a region be... ,in, Let m be the number of rows and columns of pixels in the region. Then, the powder content score d, which reflects the powder content of the region and is used for subsequent target detection and evaluation, is defined as:

[0096] ;

[0097] in, This indicates that a white pixel is 255 and a black pixel is 0.

[0098] The average value of powder saturation at the corresponding position in the original powder distribution image for each region is calculated, and the powder saturation score s, which reflects the degree of powder grinding in that region, is calculated.

[0099] Record the center pixel coordinates of each region and transform the center pixel coordinates to the world coordinate system;

[0100] Based on the intrinsic and extrinsic parameters of the color depth camera set on the grinding platform, the pixel coordinates in the powder distribution image are mapped to their actual positions in three-dimensional space using the camera's perspective projection model. ;

[0101] With a predetermined width Divide the space inside the mortar into several concentric rings, then the ring number c corresponding to each region is obtained through... Calculate and obtain the information quadruples for each region in the binarized powder distribution image. Represents the coordinates of the i-th region within the c-th annulus. Powder content score and powder saturation score .

[0102] Preferably, in step 2 of the above method, a heuristic policy and reward function for the powder grinding scenario are constructed in the following manner, and the robotic arm is controlled by the admittance control module to execute the heuristic policy to obtain an offline dataset, including:

[0103] Step 21, construct a heuristic strategy for the powder grinding scenario as follows:

[0104] First, the action space is represented in the form of a quadruple. ,in, This represents the change in polar coordinate radius. This represents the change in polar coordinate angle. For the desired contact force, To improve the stiffness of the admittance control module, the position, speed, and admittance control characteristics of the grinding pestle at the end of the robotic arm are adjusted by adjusting the parameters of the motion space.

[0105] Based on the visual feedback module of the robotic arm's controller, the powder information in each region can be represented by the [missing information - likely a specific method or feature]. The first ring The quadruple of the coordinates, powder content score, and powder saturation score of each region. express;

[0106] Calculate the powder content of each ring. and degree of dispersion :

[0107] ;

[0108] ;

[0109] in, Let represent the mean of the coordinates of the region corresponding to the c-th annulus. This represents the degree of dispersion of the powder distribution in the c-th ring, and the standard deviation of the coordinates of the center of the region in the c-th ring;

[0110] Based on the powder content of each ring and degree of dispersion Calculate the grinding value of each ring ,in, Indicates the first Average powder saturation in each ring;

[0111] Select the maximum grinding value The corresponding radius of the annulus is used as the polar coordinate radius of the target. The change in polar coordinate radius is adjusted by the PD controller of the robotic arm. To gradually approach the polar coordinate radius of the target ;

[0112] Secondly, the change in polar coordinate angle during the grinding process is calculated based on the average saturation of the powder. Expected contact force Stiffness of the admittance control module The calculation formula is:

[0113] ;

[0114] ;

[0115] ;

[0116] in, , , , , , These represent the maximum value of the polar coordinate angle change, the minimum value of the polar coordinate angle change, the maximum value of the desired contact force, the minimum value of the desired contact force, the maximum value of the stiffness of the admittance control module, and the minimum value of the stiffness of the admittance control module, respectively.

[0117] The heuristic strategy determined by the above formula calculates the motion space and outputs it to the admittance control module of the robotic arm's controller to execute the grinding action.

[0118] Step 22, the constructed reward function R for reinforcement learning is:

[0119] ;

[0120] Among them, non-negative numbers , , These represent the visual coefficient, force feedback coefficient, and time coefficient, respectively. V represents the visual reward function; F represents the force feedback reward value of the force feedback reward function. Represents the time-reward function;

[0121] During actual training, the time term coefficient remains fixed, while the visual and force feedback coefficients are dynamically adjusted based on the magnitude of the force feedback reward function, as follows:

[0122] ;

[0123] ;

[0124] ;

[0125] Step 23: Execute the heuristic strategy designed in Step 21 in the actual robotic arm environment. During the execution of the action, collect and store the data of the visual feedback module and admittance control module of the robotic arm controller, the robotic arm's own motion data, and the corresponding reward value for each step as an offline dataset.

[0126] Preferably, in step 22 above, the visual reward function V is:

[0127] ;

[0128] in, This represents the visual reward value at time t. for:

[0129] ;

[0130] Where D represents the breadth of powder distribution in the current visual feedback. E represents the degree of dispersion of the powder distribution in the current visual feedback. ; Indicates the normalization parameter;

[0131] This is the initial value for the visual reward, and this initial value for the visual reward is saved during the initialization phase of each training round.

[0132] The force feedback reward value F of the force feedback reward function is:

[0133] ;

[0134] in, , This indicates the force feedback value at the end effector of the robotic arm; and These are the upper and lower threshold values ​​for force feedback at the robotic arm's end effector, respectively. The force feedback reward value represents the final force feedback reward value. Represents the normalized parameters of the force feedback reward function;

[0135] The time reward function for:

[0136] ;

[0137] in, This represents the normalized parameter of the time reward function; step represents the number of steps in this round.

[0138] It can be seen that the normalized parameters of the force feedback reward function mentioned above... Normalized parameters of the time reward function The appropriate value can be determined through debugging based on the training effect.

[0139] Preferably, in step 2 of the above method, the offline reinforcement learning algorithm adopts the CQL algorithm, i.e., conservative Q-learning.

[0140] Preferably, in step 4 of the above method, the strategy model with initial grinding capability obtained in step 3 is deployed on the controller of the robotic arm in the following manner: the visual feedback module and admittance control module of the robotic arm controller continuously collect real-time visual feedback and torque feedback, respectively; and the strategy network of the strategy model of the reinforcement learning module is incrementally updated and optimized using an online reinforcement learning algorithm, including:

[0141] Step 41: Load the pre-trained policy model output after training in Step 3 into the online sub-module of the reinforcement learning module in the controller of the robotic arm; synchronously collect real-time status and action data provided by the visual feedback module and the admittance control module on the real grinding platform, and calculate the instant reward according to the reward function in Step 22.

[0142] Step 42: The reinforcement learning module calls the SAC algorithm to incrementally update the policy network and value network of the policy model using the real-time state and action data provided by the visual feedback module and admittance control module collected in step 41. Finally, the updated policy model is output after completing online fine-tuning with a preset number of fine-tuning rounds as the termination condition.

[0143] Preferably, the system framework corresponding to the above method is as follows: Figure 5As shown, it includes:

[0144] The robotic arm consists of a robotic arm body and a controller. The controller includes a reinforcement learning module, an admittance control module, and a visual feedback module. The reinforcement learning module is divided into offline and online sub-modules. The robotic arm body is a seven-DOF robotic arm responsible for executing the actual movements. The admittance control module is connected to the robotic arm body and the reinforcement learning module, responsible for sending force feedback information to the reinforcement learning module, receiving the desired trajectory and controller parameters from the reinforcement learning module, and sending the action to be executed to the robotic arm body. The reinforcement learning module is connected to the admittance control module and the visual feedback module, responsible for sending the desired trajectory and controller parameters to the admittance control module, receiving force feedback information from the admittance control module, and receiving visual statistical information from the visual feedback module. The visual feedback module is connected to the reinforcement learning module and is responsible for sending visual statistical information to the reinforcement learning module. The offline sub-module of the reinforcement learning module is responsible for executing heuristic policies, collecting data, and offline training, while the online sub-module is responsible for online training and executing the final policy.

[0145] In summary, the control method of this invention, by combining the offline reinforcement learning algorithm of CQL (Conservative Q-Learning) with the online fine-tuning of the SAC algorithm (Flexible Action-Evaluation Algorithm), can maximize task execution efficiency while ensuring policy training efficiency. Online reinforcement learning can respond to material characteristics and working condition changes in real time, achieving adaptive policy optimization. The final grinding effect is significantly better than pure heuristic or offline reinforcement learning methods in terms of particle size uniformity and fineness. Furthermore, the applied framework can be modularly designed and flexibly deployed on different robotic arm platforms, demonstrating good versatility and engineering promotion value.

[0146] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0147] Example 1

[0148] like Figure 1 As shown, this embodiment provides a control method for a chemical powder grinding robotic arm based on reinforcement learning, used to control the robotic arm to perform efficient and highly generalizable grinding tasks of chemical powders, including:

[0149] Step 1: An approximate model of the inner surface of the mortar on the grinding platform is created using a semi-circular curved surface. This model determines the spatial mapping relationship of the mortar's motion. Based on the mortar's inner surface model, an admittance control module for the robotic arm controller is constructed. An RGBD camera on the grinding platform acquires images of the powder distribution within the grinding space of the mortar. The powder distribution images are binarized and quantified to statistically analyze the powder distribution information. This includes the following steps:

[0150] Step 11, First, construct the inner surface model of the mortar. Represented as:

[0151] ;

[0152] in, The depth of the mortar; The radius of the semicircular model; For the model The coordinate points on the graph; if the grinding workspace is approximated as a circle, then the polar coordinate radius can be used. and polar coordinate angle The position of the grinding pestle in the mortar is uniquely determined. During the grinding process, the center line of the grinding pestle must always pass through the center point I of the inscribed circle to ensure that the grinding pestle remains perpendicular to the surface of the mortar for thorough grinding. Therefore, for a grinding pestle whose initial state is that its center line is parallel to the z-axis, the Euler angles for the x and y axes are calculated as follows:

[0153] ;

[0154] ;

[0155] In addition, the angle between the center line of the grinding pestle and the z-axis Finally, the coordinates of the robotic arm's end effector are calculated as follows:

[0156] ;

[0157] ;

[0158] ;

[0159] Based on the above grinding motion method, only the polar coordinate radius is needed. The grinding point in the mortar can be uniquely determined by the polar coordinate angle q, thus determining the polar coordinate radius. The reference grinding trajectory is calculated during the dynamic change of the polar coordinate angle q.

[0160] Step 12: Using the constructed admittance control module, adjust the grinding trajectory of the grinding robot arm according to the input grinding reference trajectory through real-time force feedback in the following manner. The expression of the admittance control module is:

[0161] ;

[0162] Among them, parameters M, B, and K represent the mass, damping, and stiffness of the admittance control module, respectively; , These represent the components of the actual and desired positions of the robotic arm's end effector in Cartesian space along the x, y, or z axes, respectively. , These represent the rate of change of the actual position and the rate of change of the desired position, respectively. , These represent the acceleration due to the actual position change and the acceleration due to the desired position change, respectively. , Let the actual contact force and the desired contact force at the end of the robotic arm be represented respectively; , Let the position error and the end force error be represented by the error acceleration at time j in the backward difference form of the discrete system. The calculation is as follows:

[0163] ;

[0164] Then the actual position at time j The calculation is as follows:

[0165] ;

[0166] ;

[0167] ;

[0168] in, Let j represent the desired position of the robotic arm's end effector at time j; This represents the actual position of the robotic arm's end effector at time j; This represents the actual position of the robotic arm's end effector at time j-1; The error velocity of the robotic arm's end effector at time j-1 in the backward differential form is represented by T; T represents the control period. This represents the error velocity of the robotic arm's end effector at time j in the backward difference form. Combining the above formula with the real-time acquired contact force of the robotic arm's end effector, the target position of the end effector can be solved, making the contact between the robotic arm's end effector and the external environment exhibit mass-spring-damped properties. This is achieved by adjusting the stiffness K of the admittance control module and the desired contact force. By adjusting the position, the end of the robotic arm can make contact, squeeze, and rub against the inner wall of the mortar in a specific state.

[0169] Step 13: Extract local color features of powder from the HSV image converted from the RGB image captured by the RGBD camera (i.e., color depth camera) to construct the color range threshold of powder, i.e., the visual threshold. Sample the powder aggregation area in the HSV image to obtain the sampling area. Calculate the hue mean of the powder color in the sampling area to obtain the color range of the powder in the current mortar. Binarize the image according to the color range to obtain the powder distribution image and save the saturation of the powder distribution area in the image.

[0170] For each pixel (R, G, B) value, calculate the normalized (r, g, b) value using the following formula:

[0171] ;

[0172] Calculate the following intermediate parameters respectively:

[0173] ;

[0174] ;

[0175] ;

[0176] Using the intermediate parameters obtained from the above calculations , , Calculate the hue (H), saturation (S), and lightness (V) respectively:

[0177] ;

[0178] ;

[0179] ;

[0180] In HSV images Sampling was performed on the powder accumulation area. Given the set of real numbers, the determined sampling region Q is:

[0181] ;

[0182] in, p is the pixel coordinate of the center of the inner surface of the mortar in the HSV image; p is the pixel coordinate of the sampling point in the HSV image. ; Ω represents the pixel radius of the sampling region Q; Ω is the set of powder distribution masks within the sampling region Q, determined by the pixel coordinates within the sampling region Q. Powder distribution mask of corresponding sampling points composition;

[0183] The hue H at each sampling point Calculate the hue mean within the sampling region Q. :

[0184] ;

[0185] Set hue deviation threshold Saturation deviation threshold and brightness deviation threshold The hue range threshold that constitutes the color range threshold of the powder Saturation range threshold and brightness range threshold They are respectively:

[0186] ;

[0187] ;

[0188] ;

[0189] in, The average saturation value within the sampling region Q. The mean lightness value within the sampling region Q, and the hue interval threshold. In This represents the lower limit of the hue range. The upper limit of the hue range; the threshold of the saturation range. In This represents the lower limit of the saturation range. The upper limit of the saturation range; the threshold of the brightness range. In This represents the lower limit of the brightness range. This represents the upper limit of the brightness range;

[0190] The HSV image is binarized according to the above-mentioned powder color range thresholds to obtain pixel coordinates. Powder distribution mask of corresponding sampling points :

[0191] ;

[0192] The original image is processed based on the powder distribution mask described above to obtain a binarized powder distribution image. All elements in the binarized powder distribution image that satisfy the above conditions are then processed. saturation value of pixels Saturation matrix is ​​formed by summing the results. The saturation of the powder distribution area in the powder distribution image is used as the basis for calculation of rewards and training of policies.

[0193] Step 14: Divide the binarized powder distribution image obtained in Step 13 into several non-overlapping regions according to the row and column directions and a preset grid size, and calculate the average gray value of each region as the scoring index for that region. Let the gray value of each pixel in each region be... ,in The powder content score d for this region is defined as follows:

[0194] ;

[0195] in, White pixels are defined as 255, and black pixels as 0. The powder content score d for each region reflects the powder content of that region and can be used for subsequent target detection and evaluation. In addition, for each region, the powder saturation value at the corresponding position in the original image is averaged to calculate the powder saturation score s for that region. The powder saturation score reflects the degree of powder abrasion in that region.

[0196] Furthermore, the center pixel coordinates of each region are recorded and transformed into the world coordinate system; based on the camera's intrinsic and extrinsic parameters, the pixel coordinates in the image are mapped to their actual positions in three-dimensional space using the camera's perspective projection model. For the space inside the mortar, with a predetermined width Dividing it into several concentric rings, the ring number c corresponding to each region can be obtained through... Calculation; this allows us to obtain the information quadruple for each region in the corresponding image. , representing the coordinates, powder content score, and powder saturation score of the i-th region in the c-th ring;

[0197] Step 2 involves constructing a heuristic policy and reward function for the powder grinding scenario, and executing the heuristic policy through the admittance control module to obtain an offline dataset, including the following steps:

[0198] Step 21, first, represent the action space in the form of a quadruple as follows: ,in, This represents the change in polar coordinate radius. This represents the change in polar coordinate angle. For the desired contact force, To adjust the stiffness of the admittance control module, the movement position, speed, and admittance control characteristics of the grinding pestle held at the end of the robotic arm can be adjusted by adjusting the parameters of the motion space.

[0199] According to the visual feedback module of the robotic arm's controller, the information of particles in each small region can be obtained from quaternions. Description, indicating the first The first ring The coordinates, powder content score, and powder saturation score of each region are calculated; first, the powder content of each ring is calculated. and the degree of dispersion of powder distribution :

[0200] ;

[0201] ;

[0202] in, This represents the mean of the coordinates of the region corresponding to the annulus c. The dispersion of powder distribution in the c-th ring is represented by , which is the standard deviation of the coordinates of the center region of the c-th ring. Subsequently, the for each ring is calculated separately. To evaluate its grinding value, among which Indicates the first The average saturation of each ring, then select the largest one. The corresponding radius of the annulus is used as the polar coordinate radius of the target. And adjust via PD control To gradually approach the polar coordinate radius of the target ;

[0203] Secondly, throughout the polishing process, a crucial metric for evaluating the polishing progress is the average saturation of all regions in the image. Therefore, the remaining three parameters are calculated based on the average powder saturation. The overall idea is that as the average powder saturation decreases during the grinding process, the angle change gradually increases, while the desired contact force and stiffness gradually decrease. The specific calculations are as follows:

[0204] ;

[0205] ;

[0206] ;

[0207] in, , , , , , These represent the maximum and minimum values ​​of the angle change, the maximum and minimum values ​​of the desired contact force, the maximum and minimum values ​​of the stiffness of the admittance control module, respectively. After calculating the motion space according to the above formulas, the output is sent to the force-controlled grinding module of the robotic arm to execute the grinding action.

[0208] Step 22, construct the reward function for reinforcement learning as follows:

[0209] Firstly, regarding the above powder content and the degree of dispersion of powder distribution Based on this, calculate the breadth D and dispersion E of the powder distribution in the current visual feedback:

[0210] ;

[0211] ;

[0212] Based on the above evaluation metrics, the average saturation information of particles in the image is then introduced. This is used to measure the completion level of the current task; based on the above evaluation metrics, the visual reward function is designed as follows:

[0213] ;

[0214] in This represents the visual reward value at the current moment. This represents the normalization parameter; additionally, the initial value of the visual reward is saved during the initialization phase of each training round. and calculate This serves as the actual visual reward value; by using the ratio of the current value to the initial value as the final reward value, the bias between training rounds caused by different environments during initialization can be effectively avoided.

[0215] A force sensor is installed at the robot's end effector to achieve admittance control and end-effector force feedback during the grinding process. End-effector force feedback is incorporated into the reward function design to ensure effective contact and operational safety during grinding. The force feedback reward function is shown below:

[0216] ;

[0217] in, , This indicates the end force feedback value. and These are the lower and upper thresholds for force feedback, respectively. This represents the final force feedback reward value; the reward function is designed to effectively guide the contact force between the robot end effector and the environment, ensuring that the applied force is neither too low to result in insufficient contact, nor too high to result in damage to the equipment or workpiece.

[0218] To prevent the reinforcement learning module from becoming excessively slow or generating unnecessary repetitive behaviors due to every positive reward function during training, a time penalty term is introduced into the reward function. ;step represents the steps in this round. By imposing penalties on each step, with the penalties increasing the longer the time, the more severe the penalty, the more the reinforcement learning module is prevented from deliberately procrastinating.

[0219] Combining the visual reward function, force feedback reward function, and time reward function, the overall reward function is calculated as follows:

[0220] ;

[0221] Among them, non-negative numbers , , These represent the visual coefficient, force feedback coefficient, and time coefficient, respectively. During actual training, the visual coefficient and force feedback coefficient are dynamically adjusted based on the magnitude of the force feedback reward value of the force feedback reward function, while the time term coefficient remains fixed.

[0222] ;

[0223] ;

[0224] ;

[0225] Step 23: Execute the above heuristic strategy in the actual robotic arm environment. During the execution of the action, collect and store the data of the visual feedback module, the admittance control module, the robotic arm's own motion, and the corresponding reward value for each step as an offline dataset.

[0226] Step 3: Read training samples containing visual feedback, force control feedback, end position and corresponding action space information from the offline data storage unit; call the CQL algorithm as an offline reinforcement learning algorithm, and use the above samples to iteratively update the policy network and value network; output the offline pre-trained policy model after completing batch training with a preset number of training rounds as the termination condition.

[0227] Step 4: Deploy the initial policy obtained in Step 3 on a real grinding platform, and use an online reinforcement learning algorithm to incrementally update the policy network, thereby optimizing the policy. This includes the following steps:

[0228] Step 41: Load the pre-trained policy model output after training in Step 3 into the online sub-module; synchronously collect real-time status and action data provided by the visual feedback module and the force control grinding module on the real grinding platform, and calculate the instant reward according to the reward function in Step 22;

[0229] Step 42: Call the SAC algorithm to incrementally update the policy network and value network using the online interaction data mentioned above; finally, output the updated policy model after completing the online fine-tuning by using the preset number of fine-tuning rounds as the termination condition.

[0230] Step 5: The admittance control module and the visual feedback module acquire the status information fed back by the end torque sensor and the vision system in real time according to the optimal strategy obtained after online fine-tuning. They input the information into the strategy network to generate motion space information and send it to the grinding reference trajectory generation submodule. This drives the end of the robotic arm to perform appropriate grinding actions according to the current powder grinding situation and the end contact force until the visual feedback information reaches the set target and then stops.

[0231] The control method in this embodiment combines offline reinforcement learning algorithms with online fine-tuning, which can maximize task execution efficiency while ensuring policy training efficiency. Online reinforcement learning can respond to material characteristics and working condition changes in real time, achieving adaptive policy optimization. Finally, the grinding effect is significantly better than pure heuristic or offline reinforcement learning methods in terms of particle size uniformity and fineness.

[0232] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0233] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A control method of a chemical powder grinding robot based on reinforcement learning, characterized by, The method comprises the following steps: Step 1, establishing a mortar inner surface model of the grinding platform, constructing a mobility control module of a controller of the mechanical arm according to the mortar inner surface model, and collecting powder information in a grinding space in the mortar; In the step 1, the mortar inner surface model of the grinding platform is established in the following manner, the mobility control module of the controller of the mechanical arm is constructed according to the mortar inner surface model, and the powder information in the grinding space in the mortar is collected, comprising: Step 11, approximating the inner surface of the mortar of the grinding platform with a semicircular curved surface to establish the mortar inner surface model; Step 12, constructing a mobility control module of a controller of the mechanical arm through the established mortar inner surface model, and adjusting the grinding track of the mechanical arm in real time according to the grinding reference track through the mobility control module; Step 13, constructing a color interval threshold of the powder through the local color features of the powder image in the grinding space in the mortar, binarizing the powder image according to the color interval threshold of the powder to obtain a powder distribution image, and saving the saturation of the powder distribution area in the powder distribution image; Step 14, dividing the powder distribution image into a plurality of non-overlapping areas in a row-column direction with a preset grid size, calculating the average gray value of each area as a score index of the corresponding area, and combining the saturation of each area as the powder information in the grinding space; Step 2, constructing a heuristic strategy and a reward function for the powder grinding scene, and controlling the mechanical arm to execute the heuristic strategy through the mobility control module to obtain an offline data set; Step 3, preliminarily training a policy network of a policy model of a reinforcement learning module based on the offline data set obtained in the step 2, to obtain a policy model with initial grinding capacity; Step 4, deploying the policy model with initial grinding capacity obtained in the step 3 to the controller of the mechanical arm, continuously collecting real-time visual feedback and torque feedback through the visual feedback module and the mobility control module of the controller of the mechanical arm, and incrementally updating the policy network of the policy model of the reinforcement learning module through an online reinforcement learning algorithm to optimize the policy; Step 5, the mobility control module generates a grinding track in real time according to the optimal policy of the policy model obtained after online fine-tuning, and drives the mechanical arm to execute the powder grinding operation until the visual feedback reaches the set target.

2. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 1, wherein, In step 11, the established model of the inner surface of the mortar is represented as: ； in, Model of the inner surface of the mortar coordinates on; The radius of the model of the inner surface of the mortar; The depth inside the mortar; During the grinding process where the robotic arm holds the grinding pestle, the end of the pestle is always on and perpendicular to the inner surface of the mortar. In this grinding motion, for a grinding pestle whose initial state is that its centerline is parallel to the z-axis, the Euler angle of the x-axis is... Euler angles with the y-axis The calculations are as follows: ； ； Angle between grinding pest center line and z axis ; wherein r j represents the polar coordinate radius of the mechanical arm end position in the mortar top view, which is the distance between the mechanical arm end position and the center of the circle in the mortar top view; q represents the polar coordinate angle of the mechanical arm end position in the mortar top view, which is the angle between the projection of the grinding pest center line in the mortar top view and the positive direction of the x axis; According to the included angle between the center line of the grinding pestle and the z-axis , the coordinate point on the inner surface model of the mortar , , , are calculated as follows, respectively: ； ； ； Wherein, L is the length of the grinding pestle; By the polar coordinate radius r as a polar coordinate parameter j and the polar coordinate angle q the grinding point of the grinding pestle in the mortar can be uniquely determined, in the course of a dynamic change of the polar coordinate radius r j and the polar coordinate angle q the reference grinding trajectory is calculated.

3. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 1, wherein, In the step 12, the expression of the constructed mobility control module is: ； Wherein, parameters M, B, K represent the mass, damping and stiffness of the admittance control module respectively; 、 respectively represent the components of the actual position and the desired position of the end of the mechanical arm in the x-axis, y-axis or z-axis in the Cartesian space; 、 respectively represent the changing speed of the actual position and the desired position of the end of the mechanical arm; 、 respectively represent the changing acceleration of the actual position and the desired position of the end of the mechanical arm; 、 respectively represent the actual contact force and the desired contact force of the end of the mechanical arm; Let and denote the error of the end position of the robot arm and the force error, respectively. The error acceleration of the end position of the robot arm at time j in the backward difference form in the discrete system is calculated as: ； wherein, represents a force error of the robot end position at time j; represents an error velocity of the robot end position at time j-1 in a backward difference form; represents an error of the robot end position at time j-1. and the actual position of the robot end at time j The calculation is: ； ； ； wherein, denotes the error velocity of the j-th moment of the end position of the robot arm in the backward difference form; T denotes the control period; denotes the error of the j-th moment of the end position of the robot arm; denotes the desired position of the j-th moment of the end of the robot arm; The end target position can be solved by combining the expression of the mobility control module and the real-time collected contact force of the mechanical arm end, so that the contact between the mechanical arm end and the outside world presents the mass-spring-damper property; By adjusting the stiffness K and the desired contact force of the admittance control module expression , the end of the mechanical arm can produce contact, extrusion and friction with the inner surface of the mortar in a predetermined state.

4. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 3, wherein, In the step 13, the color interval threshold of the powder is constructed through the local color features of the powder image in the grinding space in the mortar in the following manner, the powder image is binarized according to the color interval threshold of the powder to obtain a powder distribution image, and the saturation of the powder distribution area in the powder distribution image is saved. The RGB image of the mortar of the grinding platform captured by the color depth camera arranged on the grinding platform is taken as a powder image, local color features of the powder are extracted from the HSV image converted from the RGB image, a sampling area is obtained by sampling the powder accumulation area in the HSV image, a hue mean value of the powder color in the sampling area is calculated to obtain a color interval threshold of the powder in the current mortar, a powder distribution mask is obtained by binarizing the HSV image according to the color interval threshold, and a powder distribution image is obtained by processing the original RGB image according to the powder distribution mask, and the saturation of the powder distribution area in the powder distribution image is saved.

5. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 4, wherein, In step 13, the local color features of the powder are extracted from the HSV image converted from the RGB image in the following manner, the sampling area is obtained by sampling the powder accumulation area in the HSV image, the hue mean value of the powder color in the sampling area is calculated to obtain the color interval threshold of the powder in the current mortar, the powder distribution mask is obtained by binarizing the HSV image according to the color interval threshold, the powder distribution image is obtained by processing the original RGB image according to the powder distribution mask, and the saturation of the powder distribution area in the powder distribution image is saved. The (r, g, b) value of each pixel (R, G, B) in the RGB image is normalized according to the following calculation formula: ； The following intermediate parameters are calculated by the obtained (r, g, b) value: ； ； ； Using the above obtained intermediate parameters , , Luminance V, saturation S and hue H are calculated as follows, respectively: ； ； ； Sampling is performed in powder accumulation areas in HSV images denotes the set of real numbers, the determined sampling region Q is​ ； wherein, is the pixel coordinate of the center of the inner surface of the mortar in the HSV image; p is the pixel coordinate of the sampling point in the HSV image ; is the pixel radius of the sampling region Q; Ω is the set of powder distribution masks within the sampling region Q, corresponding to the pixel coordinates of the sampling points within the sampling region Q composition; By hue H of each sampling point The hue average is calculated in the sampling area Q : ； hue deviation threshold , a saturation deviation threshold , and a lightness deviation threshold , respectively, are set as follows: , a saturation interval threshold , and a lightness interval threshold , respectively, are set as follows: ； ； ； wherein is a mean value of saturation within the sampling region Q, is a mean value of lightness within the sampling region Q, hue interval threshold in which is a lower limit value of the hue interval, is an upper limit value of the hue interval; saturation interval threshold in which is a lower limit value of the saturation interval, is an upper limit value of the saturation interval; lightness interval threshold in which is a lower limit value of the lightness interval, is an upper limit value of the lightness interval; The HSV image is binarized according to the color interval threshold of the above powder to obtain pixel coordinates The powder distribution mask of the corresponding sampling point is: ； wherein, denotes the pixel coordinates hue value of the corresponding sample point; denotes the pixel coordinates saturation value of the corresponding sample point; denotes the pixel coordinates lightness value of the corresponding sample point; All the saturation values of the pixel points in the binarized powder distribution image that satisfy the powder distribution mask are aggregated to form a saturation matrix The saturation values of the pixel points in the powder distribution region in the powder distribution image are aggregated to form a saturation matrix and saved in the visual feedback module for subsequent reward calculation and policy training.

6. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 4 or 5, characterized in that, In step 14, the powder distribution image obtained in step 13 is divided into a plurality of non-overlapping areas in the row and column directions with a preset grid size, and the average gray value of each area is calculated as the score indicator of the corresponding area, including: Let the gray value of a pixel point in a region be wherein, m is the number of rows and columns of the pixel points in the region, and the powder content score d reflecting the powder content of the region for subsequent target detection and evaluation is defined as: ； wherein, represents a white pixel point with 255 and a black point with 0; The powder saturation score s reflecting the grinding degree of the powder in the area is calculated by averaging the values of the powder saturation of the corresponding position of each area in the original powder distribution image; The center pixel coordinates of each area are recorded, and the center pixel coordinates are converted to the world coordinate system; According to the intrinsic and extrinsic parameters of the color depth camera arranged on the grinding platform, a pixel coordinate in the powder distribution image is mapped to an actual position in a three-dimensional space by using a perspective projection model of the camera ; with a predetermined width Divide the space in the mortar into several concentric circular rings, then the corresponding circular ring number c of each region is obtained through Calculate to obtain the information quadruple of each region in the binary powder distribution image The coordinates of the i-th region in the c-th circular ring are represented as , powder content score , and powder saturation score .

7. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 6, wherein, In step 2, the heuristic strategy and reward function for the powder grinding scene are constructed in the following manner, and the heuristic strategy is executed by the admittance control module of the controller of the mechanical arm to obtain the offline data set, including: Step 21, the heuristic strategy for the powder grinding scene is constructed in the following manner, including: Firstly, the action space is expressed in the form of a four-tuple as wherein, is a polar coordinate radius variation, is a polar coordinate angle variation, is a desired contact force, is a stiffness of the admittance control module, and adjustment of the parameters of the action space enables adjustment of the motion position, speed and admittance control characteristics of the polishing pad at the end of the robot arm. According to the visual feedback module of the controller of the robot arm, the powder information in each region can be represented by a four-tuple indicating the coordinates of the first region in the first annulus, the powder content score and the powder saturation score . Calculating the powder content of each annulus and the degree of dispersion of the powder distribution : ； ； wherein n represents the total number of annuli; represents the mean value of the coordinates of the regions corresponding to the cth annulus, represents the dispersion of the powder distribution in the cth annulus, which is the standard deviation of the coordinates of the centers of the regions in the cth annulus. According to the powder content of each annulus and the dispersion of the powder distribution Calculating the grinding value of each annulus wherein, denotes the average saturation of powder in the annulus. Selecting the maximum grinding value The corresponding circular ring radius as the polar coordinate radius of the target Adjust the polar coordinate radius change amount through the PD controller of the mechanical arm Gradually approach the polar coordinate radius of the target ; Secondly, the change of polar coordinate angle during the grinding process is calculated according to the average saturation of the powder , the desired contact force , and the stiffness of the admittance control module The calculation formula is: ； ； ； wherein, , , , , , respectively represent a maximum value of the polar coordinate angle variation amount, a minimum value of the polar coordinate angle variation amount, a maximum value of the expected contact force, a minimum value of the expected contact force, a maximum value of the rigidity of the admittance control module, and a minimum value of the rigidity of the admittance control module; represents the average saturation of the powder; The action space calculated according to the heuristic strategy determined by the above formula is output to the admittance control module of the controller of the mechanical arm to execute the grinding action; The reward function R of the reinforcement learning constructed in step 22 is: ； wherein, non-negative , , respectively represent visual coefficient, force feedback coefficient and time term coefficient, and ; V represents visual reward function; F represents force feedback reward value of force feedback reward function; represents time reward function; In the actual training process, the time item coefficient remains a fixed value, and the visual coefficient and the force feedback coefficient are dynamically adjusted according to the size of the force feedback reward value F of the force feedback reward function in the following manner, which is: ； ； ； wherein F max represents an upper threshold of the force feedback reward value of the force feedback reward function; F min represents a lower threshold of the force feedback reward value of the force feedback reward function. Step 23, the heuristic strategy designed in step 21 is executed in the actual mechanical arm environment, and the visual feedback module, the admittance control module of the controller of the mechanical arm, the data of the movement of the mechanical arm itself, and the corresponding reward value of each step are collected and stored as the offline data set.

8. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 7, wherein, The visual reward function V in step 22 is: ； wherein, represents the visual reward value at time t, is: ； where D is a spread of the powder distribution in the current visual feedback, E is a dispersion of the powder distribution in the current visual feedback, ; denotes a normalization parameter; The visual reward initial value is initialized for each training round, and the visual reward initial value is saved in the initialization stage. The force feedback reward value F of the force feedback reward function is: ； wherein, , represents the mechanical arm end force feedback value; and are respectively the upper limit threshold and the lower limit threshold of the mechanical arm end force feedback; represents the normalization parameter of the force feedback reward function; The time reward function is: ； wherein, denotes a normalizing parameter of the temporal reward function; step denotes the step in the current episode.

9. The control method of a chemical powder grinding robot based on reinforcement learning according to claim 7, wherein, In step 4, the strategy model with initial grinding ability obtained in step 3 is deployed to the controller of the robot arm in the following manner: the visual feedback module and the admittance control module of the controller of the robot arm continuously collect real-time visual feedback and torque feedback respectively, and the policy network of the strategy model of the reinforcement learning module is incrementally updated using an online reinforcement learning algorithm for policy optimization, including: Step 41: load the pre-trained strategy model output after training in step 3 into the online sub-module of the reinforcement learning module of the controller of the robot arm; simultaneously collect real-time state and action data provided by the visual feedback module and the admittance control module on the real grinding platform, and calculate the immediate reward according to the reward function in step 22; Step 42: the reinforcement learning module calls the SAC algorithm to incrementally update the policy network and the value network of the strategy model with the real-time state and action data provided by the visual feedback module and the admittance control module collected in step 41; finally, the updated strategy model is output after completing online fine-tuning with the preset fine-tuning number as the termination condition.

Images