A water-based glue spraying path planning method and system based on reinforcement learning

Abstract

The application provides a water-based glue spraying path planning method and system based on reinforcement learning, acquires a three-dimensional model of a workpiece and spraying process parameters, generates trajectory data by interacting with a spraying environment through a strategy network, and calculates advantage estimation values and their variances by using a value network. When the spraying quality does not meet the standard, the system constructs a training data set according to the variances of the advantage estimation values, updates the value network and optimizes the strategy network parameters. Then, the system converts the surface of the workpiece into a discrete graph structure, establishes a landmark set and calculates the cost between points, and generates a node heat map by using the value network. According to the geometric properties and accuracy requirements of the current spraying area, the system determines a landmark subset and calculates landmark contribution values. When there is a deviation between the actual surface and the model, the system adjusts the landmark contribution values, calculates a global guide trajectory by using the landmark subset and the contribution values, and finally generates an executable spraying path instruction sequence.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a water-based adhesive spraying path planning method and system based on reinforcement learning. BACKGROUND

[0002] Water-based adhesive is an environmentally friendly adhesive. In order to ensure the quality of water-based adhesive spraying, there are high requirements for coating thickness, uniformity of coverage and the like during spraying. However, in the spraying of complex workpieces such as deep cavities and sharp edges, manual spraying has problems such as unstable glue coating amount and missed spraying, and cannot guarantee the quality of spraying. Spraying path planning can generate a spraying path according to the characteristics of the workpiece, and can accurately control the spraying quality compared with manual spraying, and is particularly suitable for water-based adhesive spraying of complex workpieces. However, one problem is that existing spraying path planning focuses on path coverage, and lacks support for how to dynamically adjust the spraying gun attitude, speed and distance from the workpiece to accurately control the uniformity of the coating thickness. Reinforcement learning has self-learning and optimization capabilities in complex decision-making problems, and can be applied to robot path planning. However, there are still many problems in directly applying reinforcement learning to actual spraying. The value function or strategy obtained by reinforcement learning is usually an evaluation of local state or action, how to convert it into a spraying path that guides the whole and meets the process constraints, and how to extract valuable information from it to improve the strategy and value judgment when the spraying quality does not meet the standard, and avoid repeated inefficient exploration in invalid areas, are the keys to improving the quality of water-based adhesive spraying. SUMMARY

[0003] In view of the problems of low learning efficiency, insufficient use of value information and insufficient guidance to spraying quality of reinforcement learning in spraying path planning, the application provides a water-based adhesive spraying path planning method based on reinforcement learning, comprising:

[0004] A three-dimensional model of a workpiece and preset spraying process parameters are obtained, a policy network interacts with a spraying environment simulation, and trajectory data containing state, action and reward are generated, and an advantage estimate value is calculated by a value network, and a variance of the advantage estimate value is calculated.

[0005] When the monitored spraying quality index does not reach the preset performance benchmark, the trajectory data corresponding to the quality not meeting the standard is obtained from the trajectory, a training data set is constructed according to the variance of the advantage estimate value, and the value network is updated using the error of the training data set; the parameters of the policy network are optimized based on the advantage estimate value calculated by the updated value network.

[0006] The workpiece surface is converted into a discrete graph structure, a landmark set is established, and the cost between each point is calculated in advance; a node heat map of the workpiece surface is obtained by using the value network; a landmark subset corresponding to the current spraying area is determined from the landmark set according to the geometric properties of the current spraying area and the spraying precision requirement, and the contribution value of the landmark in the landmark subset in the path search of the current spraying area is obtained by using the node heat map;

[0007] When the actual surface morphology of the workpiece deviates from the three-dimensional model by more than a preset deviation, the contribution value of the landmark related to the target spraying area is adjusted; the global guide trajectory is calculated on the graph structure by using the landmark subset and the contribution value, and the executable spraying path instruction sequence is generated according to the strategy network and the global guide trajectory.

[0008] Optionally, the training data set is constructed according to the variance of the advantage estimation value, comprising:

[0009] The variance of the advantage estimation value of each state-action pair in the time segment in the trajectory is calculated;

[0010] All trajectory data with a variance of the advantage estimation value greater than a variance threshold, or trajectory data ranked in the top N% of variances, are added to the training data set.

[0011] Optionally, the node heat map of the workpiece surface is obtained by using the value network, comprising:

[0012] The three-dimensional spatial position coordinates, surface normal vector, and preset spraying process parameters of each node are extracted to obtain a state vector;

[0013] Each state vector is input into the trained value network to obtain an evaluation value; the evaluation values of all nodes are normalized to obtain a normalized node heat map.

[0014] Optionally, the contribution value of the landmark in the landmark subset in the path search of the current spraying area is obtained by using the node heat map, comprising:

[0015] From the pre-set global landmark set, all landmarks whose geometric positions are located inside the current spraying area and whose distances from the current spraying area are less than a predetermined threshold are obtained to obtain a current landmark subset;

[0016] For each landmark in the landmark subset , a neighborhood node set N with the closest Euclidean distance is determined;

[0017] The average value of the evaluation values of the nodes in the neighborhood node set is calculated, and the average value is taken as the contribution value of the landmark in the path search process .

[0018] Optionally, the adjusting the contribution value of the landmark related to the target spraying area comprises:

[0019] obtaining an upper threshold value of positive deviation and a lower threshold value of negative deviation ; for each landmark in the target spraying area , calculating an average value Avg of the normal deviation amount of all actual surface points in an influence area with a center of and a radius of R;

[0020] if Avg is greater than , reducing the contribution value of the landmark according to the formula , wherein is a contribution value attenuation coefficient;

[0021] if Avg is less than , reducing the contribution value of the landmark according to the formula , wherein is a contribution value attenuation coefficient.

[0022] Optionally, the calculating the global guide track on the graph structure by using the subset of landmarks and the contribution values comprises:

[0023] according to a discretized graph structure G=(V,E) of the workpiece surface, wherein V is a node set representing each micro area on the surface, and E is an edge set representing edges between nodes that can be directly connected for spraying operation; each edge e(u,v) connects node u and node v and has a basic path cost value , which is determined according to the three-dimensional Euclidean distance or the estimated spraying time between node u and node v;

[0024] determining a start node and an end node of the current spraying task in the graph structure G;

[0025] for each edge e(u,v) in the graph structure G, obtaining all landmarks with a distance less than an influence radius R from the edge e(u,v), and calculating an adjustment factor of the edge according to the following formula , wherein is the contribution value of the adjacent landmark , and is the weight of the landmark on the edge e(u,v);

[0026] calculating the passing cost of each edge in the graph structure G , wherein is the basic path cost value of the edge e(u,v);

[0027] A path search method is used to search for a path from the starting node to the ending node with the minimum cumulative cost as the global guide trajectory.

[0028] Optionally, the generating of the executable spraying path instruction sequence according to the policy network and the global guide trajectory comprises:

[0029] The global guide trajectory is refined into a series of path points by an interpolation algorithm, each path point containing three-dimensional position coordinates, an expected spray gun end posture and a spraying start-stop state flag; the current state of the spray gun is initialized;

[0030] Starting from the first path point, in each discrete servo control cycle, the current state of the spray gun and the current path point information are combined into a state vector, the policy network receives the input state vector and outputs an action instruction vector; the target speed and posture instructions in the action instruction vector are converted into specific joint motion instructions; the generated specific joint motion instructions are added to a first-in-first-out instruction sequence buffer; after the robot motion controller executes the instructions of the current cycle, the current state of the spray gun is updated;

[0031] The distance between the current spray gun position and the current path point is calculated, and if the distance is less than the tolerance and the end point of the trajectory has not been reached, the next path point is taken as the current path point until the last path point.

[0032] The application also proposes a water-based adhesive spraying path planning system based on reinforcement learning, comprising:

[0033] A data generation unit is configured to obtain a three-dimensional model of a workpiece and preset spraying process parameters, the policy network interacts with the spraying environment simulation, generates trajectory data containing states, actions and rewards, and calculates the advantage estimation value by the value network, and calculates the variance of the advantage estimation value;

[0034] An iterative optimization unit is configured to obtain a trajectory that does not meet the quality standard from the trajectory when the monitored spraying quality index does not meet the preset performance benchmark, construct a training data set according to the variance of the advantage estimation value, update the value network using the error of the training data set, and optimize the parameters of the policy network based on the advantage estimation value calculated by the updated value network;

[0035] A graph construction unit is configured to convert the surface of the workpiece into a discrete graph structure, establish a landmark set and pre-calculate the cost between points, obtain a node heat map of the surface of the workpiece using the value network, determine a landmark subset corresponding to the current spraying area from the landmark set according to the geometric properties of the current spraying area and the spraying precision requirement, and obtain the contribution value of the landmarks in the landmark subset in the path search of the current spraying area using the node heat map.

[0036] The path generation unit is configured to adjust the contribution value of the landmark related to the target spraying area when the actual surface morphology of the workpiece deviates from the three-dimensional model by more than a preset deviation; calculate a global guide track on the graph structure by using the landmark subset and the contribution value, and generate an executable spraying path instruction sequence according to the strategy network and the global guide track.

[0037] Optionally, the training data set is constructed according to the variance of the advantage estimation value, and the method comprises:

[0038] The variance of the advantage estimation value of each state-action pair in the time segment in the track is calculated.

[0039] The track data with a variance of the advantage estimation value greater than a variance threshold value, or the track data with a variance ranking in the top N percent, is added to the training data set.

[0040] Optionally, the node heat map of the workpiece surface is obtained by using the value network, and the method comprises:

[0041] The three-dimensional spatial position coordinates, the surface normal vector, and the preset spraying process parameters of each node are extracted to obtain a state vector.

[0042] Each state vector is input into the value network to obtain an evaluation value, and the evaluation values of all nodes are normalized to obtain a normalized node heat map.

[0043] Optionally, the contribution value of the landmark in the landmark subset in the path search in the current spraying area is obtained by using the node heat map, and the method comprises:

[0044] From a preset global landmark set, all landmarks located inside the current spraying area and the landmarks with a distance less than a predetermined threshold value from the current spraying area are obtained to obtain a current landmark subset.

[0045] For each landmark in the landmark subset , a neighborhood node set N with the closest Euclidean distance is determined.

[0046] The average value of the evaluation values of the nodes in the neighborhood node set is calculated, and the average value is taken as the contribution value of the landmark in the path search process .

[0047] Optionally, the contribution value of the landmark related to the target spraying area is adjusted, and the method comprises:

[0048] A positive deviation upper threshold value and a negative deviation lower threshold value are obtained; for each landmark in the target spraying area , a positive deviation Avg is the average of the normal deviation of all actual surface points in the influence area with radius R centered at the point;

[0049] If Avg is greater than , the contribution value of the landmark is reduced according to the formula , wherein is the contribution value decay coefficient;

[0050] If Avg is less than , the contribution value of the landmark is reduced according to the formula , wherein is the contribution value decay coefficient.

[0051] Optionally, the calculating the global guide trajectory on the graph structure using the subset of landmarks and the contribution values comprises:

[0052] According to the discretized graph structure G=(V,E) of the workpiece surface, wherein V is a set of nodes representing each micro area on the surface, and E is a set of edges representing the connection between nodes that can be directly connected for spraying operation; each edge e(u,v) connects node u and node v, and has a basic path cost value , which is determined according to the three-dimensional Euclidean distance or the estimated spraying time between node u and v;

[0053] Determine the start node and the end node of the current spraying task in the graph structure G;

[0054] For each edge e(u,v) in the graph structure G, obtain all landmarks with a distance less than the influence radius R from the edge e(u,v) point, and calculate the adjustment factor of the edge according to the following formula , wherein is the contribution value of the adjacent landmark , and is the weight of the landmark on the edge e(u,v);

[0055] Calculate the passing cost of each edge in the graph structure G , wherein is the basic path cost value of the edge e(u,v);

[0056] A path search method is used to search a path from the start node to the end node with the minimum cumulative cost as the global guide trajectory.

[0057] Optionally, the generating the executable spraying path instruction sequence according to the strategy network and the global guide trajectory comprises:

[0058] The global guide trajectory is refined into a series of path points by an interpolation algorithm, each path point containing three-dimensional position coordinates, an expected spray gun end posture, and a spray start-stop state flag; the current state of the spray gun is initialized;

[0059] Starting from the first path point, in each discrete servo control cycle, the current state of the spray gun and the current path point information are combined into a state vector, the strategy network outputs an action instruction vector after receiving the input state vector; the target speed and posture instructions in the action instruction vector are converted into specific joint motion instructions; the generated specific joint motion instructions are added to a first-in-first-out instruction sequence buffer; after the robot motion controller executes the instructions of the current cycle, the current state of the spray gun is updated;

[0060] The distance between the current spray gun position and the current path point is calculated, and if the distance is less than the tolerance and the end point of the trajectory has not been reached, the next path point is taken as the current path point until the last path point.

[0061] The application constructs a training data set according to the variance of the advantage estimate value after the strategy network interacts with the environment, so that the value network can preferentially learn from those trajectory data with higher uncertainty or potential improvement space, improving the pertinence and efficiency of the learning process; the node heat map of the workpiece surface is obtained by using the trained value network, the contribution value of the landmarks in the landmark subset in the current spraying area path search is obtained by using the node heat map, the value judgment based on learning is integrated into the generation of the global guide trajectory, so that the path planning is no longer dependent on geometric information, and a global path that is more conducive to improving the spraying quality is planned. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the embodiment;

[0063] Figure 2 The visualization schematic diagram of the node heat map on the workpiece surface;

[0064] Figure 3 The schematic diagram of the global guide trajectory;

[0065] Figure 4 The schematic diagram of the global guide trajectory and the path segment. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0067] In specific embodiments, the present application provides a water-based glue spraying path planning method based on reinforcement learning, as shown in the accompanying drawings, comprising: Figure 1

[0068] Step 1, obtain the three-dimensional model of the workpiece and the preset spraying process parameters, the policy network interacts with the spraying environment simulation, and generates trajectory data containing state, action and reward, and the value network calculates the advantage estimate value, and calculates the variance of the advantage estimate value;

[0069] Obtain the three-dimensional model of the workpiece to be sprayed, for example, a standard format file generated by CAD software, and load the preset spraying process parameters, for example, the flow rate of the paint, the rated moving speed of the spray gun, the target coating thickness, etc. Reinforcement learning includes a policy network interacting with a spraying simulation environment. The simulation environment can simulate the real spraying physical process, including fluid dynamics effects and kinematics characteristics of the robot operating arm. During the interaction process, the policy network generates a series of experience trajectories through continuous trial and error. Each trajectory contains state information such as the coverage of the current workpiece surface, the real-time position and attitude of the spray gun, etc., such as the control instructions for the spray gun movement, and the actions taken by the policy network, and the feedback reward signal, such as the evaluation value quantified according to the spraying quality and efficiency. Among them, the policy network is a deep neural network, and preferably, the policy network adopts an Actor-Critic architecture.

[0070] The value network is used to evaluate the expected cumulative return that can be brought after a certain state or a certain action is executed. Based on the evaluation result of the value network, the advantage estimate value can be calculated for each "state-action" pair in the trajectory. Preferably, the generalized advantage estimation method is used, and the advantage value represents the superiority of the selected action relative to the average action in the current state. The statistical variance of the advantage estimate value is calculated on the entire trajectory data or within a preset segment. Among them, the value network preferably adopts an Actor-Critic architecture value network.

[0071] Step 2, when the monitored spraying quality indicators do not reach the preset performance benchmark, the trajectory corresponding to the substandard quality is obtained from the trajectory, the training data set is constructed according to the variance of the advantage estimate value, and the error of the training data set is used to update the value network; based on the advantage estimate value calculated by the updated value network and the parameters of the policy network are optimized;

[0072] ​When the spraying quality indicators fail to meet the preset performance benchmarks, such as the uniformity of the coating film, the effective utilization rate of the paint, or the coverage integrity of a specific area, which can be obtained through virtual sensors or analytical models, the learning and optimization mechanism is triggered. First, the experience trajectory that leads to substandard spraying results is identified, and a training dataset is constructed based on the variance of the advantage estimate calculated in the previous step to guide subsequent training. Preferably, the variance of the advantage estimate is used to prioritize the experience samples, implementing a prioritized experience replay mechanism, i.e., samples with larger variance are sampled with a higher probability for training.

[0073] The value network is updated by minimizing the loss function, and after the value network is improved, the parameters of the policy network are also further optimized, such as using the proximal policy optimization gradient algorithm, using the more accurate advantage estimate provided by the updated value network to guide the iterative improvement of the policy, thereby guiding the agent to learn better spraying behavior.

[0074] Step 3: Convert the workpiece surface to a discrete graph structure, establish a landmark set and pre-calculate the cost between points; use the value network to obtain the node heat map of the workpiece surface; determine the landmark subset corresponding to the current spraying area from the landmark set according to the geometric properties of the current spraying area and the spraying precision requirements, and use the node heat map to obtain the contribution value of the landmarks in the landmark subset in the path search of the current spraying area;

[0075] The complex three-dimensional surface of the workpiece is converted into a discrete graph structure representation. In one embodiment, the workpiece model is voxelized, or a surface mesh graph is generated directly from its CAD mesh data, where the graph nodes represent discrete units or patches on the surface, and the edges represent their adjacency relationship or sprayable connectivity. At the same time, a landmark set is defined, and the landmarks are selected according to the key points of the workpiece geometric features or the points of the areas that are difficult to spray well. The initial passage cost between these landmarks, i.e., the graph nodes, is pre-calculated, such as the surface geodesic distance or the estimated spraying time. The node heat map of the workpiece surface is generated using information in the reinforcement learning process. In one embodiment, the entropy value of the action probability distribution output by the policy network at each discrete surface point is calculated, and the high-entropy area is displayed as the key attention area on the heat map. In one embodiment, the node heat map is for the node graph, not a continuous graph, as shown in Figure 2 All nodes and node evaluation values constitute the node heat map.

[0076] For the current spraying area that needs to be path planned, a relevant landmark subset is selected from a global landmark set according to geometric range, shape characteristics and spraying precision requirements. The contribution value of each landmark in the landmark subset in the current spraying area path search is determined by using a node heat map. In an optional embodiment, the contribution value is calculated by using a kernel-based function. In another embodiment, the local heat map mode around the landmark, the geometric properties of the landmark itself and the like are input into a neural network, and the contribution value is output.

[0077] Step 4, when the actual surface morphology of the workpiece deviates from the three-dimensional model by more than a preset deviation, the contribution value of the landmark related to the target spraying area is adjusted; the global guide trajectory is calculated on the graph structure by using the landmark subset and the contribution value, and the executable spraying path instruction sequence is generated according to the strategy network and the global guide trajectory.

[0078] If it is found by online detection means such as three-dimensional scanning that the actual surface morphology of the workpiece deviates from the theoretical three-dimensional model by more than a preset tolerance, the contribution value of the landmark related to the target spraying area that deviates is adjusted. In an embodiment, Gaussian process regression is used for adjustment. Specifically, the influence on the spraying difficulty is predicted according to the specific information of the observed surface deviation, and the contribution value of the landmark is updated accordingly. A global guide trajectory is calculated on the discrete graph structure of the workpiece surface by using the updated landmark contribution value and the previously selected landmark subset, as shown in Figure 3 As shown in FIG. 6, the RRT is preferably used for searching. When constructing the search tree, the RRT preferentially explores and connects the areas affected by the landmarks with high contribution values by using a bias sampling process or a node connection strategy. The strategy network not only outputs the instantaneous action for a single waypoint on the global trajectory in each control cycle, but also predicts the optimal action sequence in a short time window in the future, so that the spraying path can be continuously adjusted and quickly respond to minor disturbances while following the global guide, thereby ensuring the spraying effect.

[0079] In an optional embodiment, the training data set is constructed according to the variance of the advantage estimation value, including:

[0080] The variance of the advantage estimation value of each state-action pair in the time segment in the trajectory is calculated.

[0081] The trajectory data with a variance greater than a variance threshold, or the trajectory data with a variance ranking in the top N percent, is added to the training data set.

[0082] In constructing the training dataset, each state-action pair is extracted from the trajectory generated by the policy network interacting with the environment, assuming that in the spraying task, the policy network selects the moving speed and direction of the spray gun as the action in a certain state and returns the corresponding reward. For each state-action pair, its advantage estimate value is calculated, which is the superiority of the action relative to the average action in the current state, and then the variance of the advantage estimate value in the preset time period or time slice is calculated. According to the preset variance threshold or ranking percentage, samples are screened, for example, if the variance threshold is set to 0.1, all state-action pairs with a variance greater than 0.1 will be added to the training dataset; or the top 10% of samples with the highest variance can be selected. These screened samples will be used to update the value network and optimize the policy network, thereby improving the performance of the model in complex spraying tasks. The model can better learn samples with high uncertainty, thereby improving the overall effect of spraying path planning.

[0083] In an optional embodiment, the node heat map of the workpiece surface obtained by using the value network comprises:

[0084] Extracting the three-dimensional spatial position coordinates, surface normal vector, and preset spraying process parameters of each node to obtain a state vector;

[0085] Inputting each state vector into the trained value network to obtain an evaluation value; and normalizing the evaluation values of all nodes to obtain a normalized node heat map.

[0086] Specifically, in the process of generating the node heat map, information of each node is extracted from the three-dimensional model of the workpiece. For example, it is assumed that the surface of the workpiece is discretized into multiple nodes, each node has its three-dimensional coordinates and surface normal vector. In addition, the preset spraying process parameters such as the flow rate of the coating and the moving speed of the spray gun are also included in the state vector, which together constitute a state vector. Each state vector is input into the trained value network. The value network outputs an evaluation value according to the input state vector, which reflects the expected effect of spraying in the current state. For example, if the evaluation value of a node is high, it indicates that this area may achieve better coverage effect during spraying. The evaluation values of all nodes are normalized to generate a normalized node heat map. This heat map shows the spraying priority of each region on the surface of the workpiece.

[0087] In an optional embodiment, the contribution value of the landmark in the landmark subset in the current spraying area path search is obtained by using the node heat map, comprising:

[0088] From a pre-set global landmark set, all landmarks whose geometric positions are located within the current spraying area and landmarks whose distance from the current spraying area is less than a predetermined threshold are obtained, thereby obtaining a current landmark subset;

[0089] For each landmark L_s in the landmark subset, determine the set of neighboring nodes N that are closest to it in Euclidean distance;

[0090] Calculate the average value of each node evaluation value in the neighborhood node set and use the average value as the landmark Contribution value in the path search process .

[0091] Specifically, in the process of calculating the landmark contribution value, the landmarks related to the current spraying area are screened from the pre-set global landmark set. For example, assuming that the current spraying area is a specific workpiece surface area, the geometric position of each landmark is checked to determine whether it is located in the area or whether the distance to the area is less than a preset threshold such as 5 mm, and a current landmark subset is constructed. For each landmark in the landmark subset, , determine the set of neighboring nodes N with the nearest Euclidean distance. For example, if the landmark The coordinates are (x, y, z), the system will calculate all nodes to The Euclidean distance is calculated and the k nodes with the smallest distance are selected as the neighborhood node set N. Then, the average evaluation value of each node in the neighborhood node set is calculated and the average value is used as the landmark The contribution value C_s during the path search process. For example, if the node evaluation values ​​in the neighborhood node set N are 0.8, 0.7, and 0.9, respectively, then the contribution value C_s of the landmark L_s will be (0.8+0.7+0.9) / 3=0.8. This contribution value can effectively guide the path search, ensure that the spray path covers key areas, and improve the overall spray effect.

[0092] In an optional embodiment, adjusting the contribution value of the landmark associated with the target spraying area includes:

[0093] Get the upper threshold of positive deviation and the negative deviation lower threshold ; For each landmark in the target spraying area , calculated with The average value Avg of the normal deviation of all actual surface points in the influence area with a radius of R as the center;

[0094] If Avg is greater than , then according to the formula To reduce the landmark The contribution value of a contribution value decay coefficient;

[0095] If Avg is less than , the contribution value of the landmark is reduced according to the formula , where is a contribution value decay coefficient.

[0096] Specifically, in the process of adjusting the contribution value of a landmark, first, the upper threshold of positive deviation and the lower threshold of negative deviation are needed. Assuming is 0.1, is -0.1, representing the maximum positive and negative deviation allowed. For each landmark within the target spraying area, the average value Avg of the normal deviation amount of all actual surface points within the influence area centered at with a radius of R, such as 10 mm, is calculated. For example, if there are 5 points within the influence area, their normal deviation amounts are 0.05, 0.08, 0.12, 0.15, and 0.18, respectively, then Avg is 0.116. If Avg is greater than , for example, , the contribution value of the landmark L_s is reduced using the formula , where is a contribution value decay coefficient, preferably 0.5. For example, if the initial value of is 0.8, the new contribution value will be 0.736. If Avg is less than , for example, , the contribution value of the landmark is reduced using the formula , where is a contribution value decay coefficient, also preferably 0.5. For example, if the initial value of is 0.8, the new contribution value is 0.736. In this way, the contribution value of the landmark is dynamically adjusted, ensuring the adaptability and accuracy of the spraying path.

[0097] In an optional embodiment, the calculating a global guide trajectory on the graph structure using the subset of landmarks and the contribution values comprises:

[0098] According to the discretized graph structure G = (V, E) of the workpiece surface, where V is a set of nodes representing each micro area on the surface, and E is a set of edges representing the connection between nodes that can be directly connected for spraying operation; each edge e(u, v) connects node u and node v, and has a basic path value , which is determined according to the three-dimensional Euclidean distance between nodes u and v or the estimated spraying time;

[0099] determining the start node and the end node of the current spraying task in the graph structure G;

[0100] for each edge e(u, v) in the graph structure G, obtaining all landmarks that are within an influence radius R from the midpoint of the edge e(u, v); wherein is the contribution value of the adjacent landmark , is the weight of the landmark on the influence of the edge e(u, v);

[0101] calculating the traversal cost of each edge in the graph structure G , wherein the traversal cost of the edge e(u, v) is is the base path cost of the edge e(u, v);

[0102] using a path search method to search for a path from the start node to the end node with the minimum cumulative cost as the global guide trajectory.

[0103] Specifically, in the process of calculating the global guide trajectory, according to the discretized graph structure G = (V, E) of the workpiece surface, wherein V is a set of nodes representing each micro area on the surface, and E is a set of edges representing the connection between nodes that can be directly connected for spraying operation. It is assumed that the workpiece surface is discretized into multiple nodes, each node representing a micro spraying area, and the edge represents the connection relationship between these areas. Each edge e(u, v) connects node u and node v and has a base path cost , which can be determined according to the three-dimensional Euclidean distance between nodes u and v or the estimated spraying time. For example, if the Euclidean distance between nodes u and v is 5 mm, the base path cost is 5. The start node and the end node of the current spraying task are determined in the graph structure G. For example, the start node is a specific position on the surface of the workpiece, and the end node is another position, which are the starting point and the ending point of the spraying task, respectively. For each edge e(u, v) in the graph structure G, the system obtains all landmarks that are within an influence radius R from the midpoint of the edge e(u, v). For example, if the influence radius R is 10 mm, find the landmarks within 10 mm from the midpoint of the edge. According to the contribution value and the weight of these landmarks, the adjustment factor of the edge is calculated. For example, if the contribution value of the adjacent landmark is 0.8, and the weight If the value of the parameter is 0.5, the adjustment factor AF is 0.714. The travel cost of each edge in the graph structure G is calculated A path search method is used to search for a path from the start node to the end node with the minimum cumulative cost as the global guide trajectory.

[0104] In an optional embodiment, the generation of the executable spraying path instruction sequence according to the policy network and the global guide trajectory comprises:

[0105] The global guide trajectory is refined into a series of path points by an interpolation algorithm, as shown in FIG. 2, each path point contains three-dimensional position coordinates, expected spray gun end posture and spraying start-stop state flag; the current state of the spray gun is initialized; Figure 4

[0106] Starting from the first path point, in each discrete servo control cycle, the current state of the spray gun and the current path point information are combined into a state vector, the policy network receives the input state vector and outputs an action instruction vector; the target speed and posture instructions in the action instruction vector are converted into specific joint motion instructions; the generated specific joint motion instructions are added to a first-in-first-out instruction sequence buffer; after the robot motion controller executes the instructions of the current cycle, the current state of the spray gun is updated;

[0107] The distance between the current spray gun position and the current path point is calculated, if the distance is less than the tolerance and the end point of the trajectory is not reached, the next path point is taken as the current path point until the last path point.

[0108] Specifically, in the process of generating the executable spraying path instruction sequence, the global guide trajectory is refined into a series of path points by an interpolation algorithm. For example, assuming that the global guide trajectory is a path from the start point to the end point, the system will use the interpolation algorithm to refine it into multiple path points, each path point contains three-dimensional position coordinates, expected spray gun end posture and spraying start-stop state flag. If the start point coordinates of the global guide trajectory are (0, 0, 0) and the end point coordinates are (10, 10, 10), it can be refined into 10 path points, each point coordinates are (1, 1, 1), (2, 2, 2),..., (10, 10, 10), and the corresponding spray gun posture and start-stop state are set.

[0109] ​The current state of the spray gun is initialized, and a motion command is generated in each discrete servo control cycle starting from the first path point. For example, assuming the coordinates of the current path point are (3, 3, 3) and the current state of the spray gun is (2, 2, 2), the current state of the spray gun and the current path point information are combined into a state vector, which is input into the policy network. After the policy network receives the input state vector, it outputs a motion command vector, for example, the target speed is (1, 1, 1) and the pose command is (0, 0, 0). The motion command vector is converted into specific joint motion commands, which are added to a first-in-first-out command sequence buffer. After the robot motion controller executes the commands of the current cycle, the current state of the spray gun is updated, and the distance between the current spray gun position and the current path point is calculated. If the distance is less than the preset tolerance and the trajectory endpoint has not been reached, the next path point is taken as the current path point, and the motion command is continuously generated until the last path point.

[0110] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. In addition, the various different embodiments of the embodiments of the present application can also be combined arbitrarily, as long as it does not deviate from the idea of the embodiments of the present application, it should also be regarded as the disclosed content of the embodiments of the present application.

Claims

1. A water-based glue spraying path planning method based on reinforcement learning, characterized in that, The method comprises the following steps: acquiring a three-dimensional model of a workpiece and preset spraying process parameters, a strategy network interacting with a spraying environment simulation, and generating trajectory data containing states, actions and rewards, and a value network calculating advantage estimation values, and calculating the variance of the advantage estimation values; when a monitored spraying quality index does not reach a preset performance benchmark, obtaining a trajectory that does not meet the quality from the trajectory, constructing a training data set according to the variance of the advantage estimation values, and updating the value network using the error of the training data set; and optimizing the parameters of the strategy network based on the advantage estimation values calculated by the updated value network; converting the surface of the workpiece into a discrete graph structure, establishing a landmark set and pre-calculating the cost between points; obtaining a node heat map of the surface of the workpiece using the value network; determining a landmark subset corresponding to the current spraying area from the landmark set according to the geometric properties of the current spraying area and the spraying accuracy requirements, and obtaining the contribution value of the landmarks in the landmark subset in the path search of the current spraying area using the node heat map; when the actual surface morphology of the workpiece deviates from the three-dimensional model by more than a preset deviation, adjusting the contribution value of the landmarks related to the target spraying area; and calculating a global guide trajectory on the graph structure using the landmark subset and the contribution value, and generating an executable spraying path instruction sequence according to the strategy network and the global guide trajectory.

2. The reinforcement learning-based waterborne adhesive spray path planning method according to claim 1, characterized in that, The method comprises the following steps: calculating the variance of the advantage estimation values of each state-action pair in the time segment in the trajectory; adding trajectory data with a variance greater than a variance threshold or trajectory data ranked in the top N% of variances to the training data set.

3. The reinforcement learning-based waterborne adhesive spray path planning method of claim 1, wherein, The method comprises the following steps: extracting the three-dimensional spatial position coordinates, surface normal vector and preset spraying process parameters of each node to obtain a state vector; inputting each state vector into the trained value network to obtain an evaluation value; and normalizing the evaluation values of all nodes to obtain a normalized node heat map.

4. The reinforcement learning-based waterborne adhesive spray path planning method according to claim 1, wherein, The method comprises the following steps: obtaining all landmarks whose geometric positions are located inside the current spraying area and landmarks whose distances from the current spraying area are less than a predetermined threshold from the pre-set global landmark set, and then obtaining the current landmark subset; For each landmark in the subset of landmarks , determine the set of neighborhood nodes N with the closest Euclidean distance to it; calculating an average of the evaluation values of the nodes in the neighborhood node set, and taking the average as the landmark contribution value in the path search process .

5. The reinforcement learning-based waterborne adhesive spray path planning method of claim 1, wherein, The method comprises the following steps: obtaining an upper threshold value for positive deviations and a lower threshold value for negative deviations ; for each landmark within the target spray area calculating the average value Avg of the normal deviation amounts of all actual surface points of the influence area with radius R centered on the landmark If Avg is greater than then the contribution value of the landmark is reduced according to the formula where is a contribution value decay coefficient; If Avg is less than then the contribution value of the landmark is reduced according to the formula where is a contribution value decay coefficient.

6. The reinforcement learning-based aqueous adhesive spray path planning method of claim 1, wherein, The method comprises the following steps: According to a discretized graph structure G=(V,E) of the surface of the workpiece, wherein V is a node set representing each micro region on the surface, and E is an edge set representing edges between nodes that can be directly connected for spray operation; each edge e(u,v) connects node u and node v and has a basic path cost value The basic path cost value is determined according to a three-dimensional Euclidean distance or an estimated spray time between nodes u and v. determining the start node and the end node of the current spraying task in the graph structure G; For each edge e(u,v) in the graph G, obtain all landmarks whose distance to the edge e(u,v) is less than the influence radius R Compute the adjustment factor for the edge according to where is the contribution value of the neighboring landmark is the weight of the landmark influence on the edge e(u,v);​ computing a transit cost for each edge in a graph structure G wherein, is a base path cost value for edge e(u,v) using a path search method to search for a path with the minimum cumulative cost from the start node to the end node as the global guide trajectory.

7. The reinforcement learning-based waterborne adhesive spray path planning method according to claim 1, wherein, The method comprises the following steps: refining the global guide trajectory into a series of path points through an interpolation algorithm, each path point containing three-dimensional position coordinates, an expected spraying gun end pose and a spraying start-stop state flag; and initializing the current state of the spraying gun. From the first path point, in each discrete servo control cycle, the current state of the spray gun and the current path point information are combined into a state vector, the strategy network receives the input state vector and outputs an action instruction vector; the target speed and attitude instructions in the action instruction vector are converted into specific joint motion instructions; the generated specific joint motion instructions are added to a first-in-first-out instruction sequence buffer; after the robot motion controller executes the instructions of the current cycle, the current state of the spray gun is updated; The distance between the current spray gun position and the current path point is calculated, and if the distance is less than the tolerance and the trajectory endpoint has not been reached, the next path point is taken as the current path point until the last path point.

8. A reinforcement learning based waterborne adhesive spray path planning system, characterized by, It comprises: A data generation unit is configured to obtain a three-dimensional model of a workpiece and preset spraying process parameters, a strategy network interacts with a spraying environment simulation, and trajectory data including states, actions and rewards are generated, and an advantage estimation value is calculated by a value network, and a variance of the advantage estimation value is calculated; An iterative optimization unit is configured to obtain a trajectory that does not meet a quality standard from the trajectory when a monitored spraying quality indicator does not meet a preset performance benchmark, construct a training data set according to the variance of the advantage estimation value, update the value network using the error of the training data set, and optimize the parameters of the strategy network based on the advantage estimation value calculated by the updated value network. A graph construction unit is configured to convert the surface of the workpiece into a discrete graph structure, establish a landmark set and pre-calculate the cost between points, obtain a node heat map of the surface of the workpiece using the value network, determine a landmark subset corresponding to the current spraying area from the landmark set according to the geometric properties of the current spraying area and the spraying accuracy requirements, and obtain a contribution value of the landmarks in the landmark subset in the path search of the current spraying area using the node heat map. A path generation unit is configured to adjust the contribution value of the landmarks related to the target spraying area when the actual surface morphology of the workpiece deviates from the three-dimensional model by more than a preset deviation, calculate a global guide trajectory on the graph structure using the landmark subset and the contribution value, and generate an executable spraying path instruction sequence according to the strategy network and the global guide trajectory.

9. The reinforcement learning-based aqueous adhesive spray path planning system of claim 8, wherein, The construction of the training data set according to the variance of the advantage estimation value comprises: Calculating the variance of the advantage estimation value of each state-action pair in a time slice in the trajectory; Adding trajectory data with a variance greater than a variance threshold or trajectory data with a top N variance to the training data set.

10. The reinforcement learning-based aqueous adhesive spray path planning system of claim 8, wherein, The node heat map of the surface of the workpiece obtained using the value network comprises: Extracting the three-dimensional spatial position coordinates, surface normal vector and preset spraying process parameters of each node to obtain a state vector; Inputting each state vector into the trained value network to obtain an evaluation value, and normalizing the evaluation values of all nodes to obtain a normalized node heat map.

Images