A Reinforcement Learning-Based Method and System for Track Slab Drilling Path Planning and Optimization

Abstract

This invention provides a method and system for track slab drilling path planning and optimization based on reinforcement learning, belonging to the field of reinforcement learning technology. The method includes acquiring track slab boundary contours, surface constraint regions, and hole distribution data to construct a hole node graph; using a pre-trained reinforcement learning model to generate a drilling path sequence that satisfies the constraints; and controlling the drilling equipment to execute operations in priority order. This invention can avoid restricted areas, follow boundary constraints, improve drilling efficiency, reduce energy consumption, and extend equipment life.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a method and system for planning and optimizing drilling paths for track slabs based on reinforcement learning. Background Technology

[0002] Track slab drilling is a crucial process in railway track construction, and its precise positioning and efficient execution have a significant impact on the safety and stability of the track. Traditionally, track slab drilling relies primarily on manual positioning and operation. However, with the rapid development of railway construction and advancements in intelligent manufacturing technology, the demand for automated and intelligent track slab drilling is growing rapidly.

[0003] Drilling path planning for track slabs is crucial to processing efficiency and quality. Currently, several path planning methods exist in the industry, including the shortest path method based on geometric algorithms, traversal optimization methods based on heuristic algorithms, and parametric planning methods using traditional machine learning. While these methods can achieve basic path planning in simple environments, they still present numerous challenges in complex track slab processing scenarios. Summary of the Invention

[0004] This invention provides a method and system for track slab drilling path planning and optimization based on reinforcement learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a method for planning and optimizing drilling paths in track slabs based on reinforcement learning, comprising:

[0006] The boundary contour, surface constraint area, and hole distribution data of the track plate to be processed are obtained. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole.

[0007] The boundary contour is represented as the boundary of the feasible movement area, and the surface constraint area is represented as the prohibited passage area; the existence of a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area is calculated, the accessibility relationship between each hole node is determined, and a hole node map containing the accessibility relationship is obtained;

[0008] Based on a pre-trained reinforcement learning model, starting from the initial node with the highest processing priority attribute in the hole node graph, and sequentially selecting the hole node with the largest value estimation result according to the reachability association, a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints is generated.

[0009] Based on the drilling path sequence, the drilling equipment is controlled to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area, and the drilling operation is performed in the order of the processing priority attribute.

[0010] The method also includes training the reinforcement learning model:

[0011] The state space of the reinforcement learning model consists of the current hole position, the set of currently processed hole nodes, and the set of remaining unprocessed hole nodes.

[0012] The action space is a set of possible transitions from the current hole node to the remaining unprocessed hole nodes according to the reachability association; the reward function is a function composed of path length cost, the degree of satisfaction of the processing priority attribute, and constraint state, the constraint state including whether the current path violates the boundary contour and the surface constraint region;

[0013] The reinforcement learning model is trained, and during the training process, the negative value of the path movement distance is used as the basic reward component, the hole node is selected as the positive incentive component according to the processing priority attribute order, and the deviation from the boundary contour or entry into the surface constraint region is used as the negative penalty component.

[0014] The reinforcement learning model learns a decision strategy that minimizes the path length while satisfying the reachability association and processing priority attribute constraints by driving the parameter update of the reinforcement learning model through a reward signal consisting of the basic reward component, the positive incentive component, and the negative penalty component.

[0015] Calculate whether there exists a connection path between any two aperture nodes in the aperture distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association between each aperture node, and obtain an aperture node graph containing the reachability association, including:

[0016] Construct an initial node set containing all hole nodes, and select the first hole node and the second hole node from the initial node set as the node pair to be verified;

[0017] A connecting line segment is generated based on the spatial coordinates of the first hole node and the second hole node. The connecting line segment represents the straight path of the drilling equipment moving from the first hole node to the second hole node.

[0018] When the connecting line segment is completely within the boundary of the feasible movement area and the connecting line segment does not intersect with the prohibited passage area, a directed edge is established in the hole node graph from the first hole node to the second hole node. The directed edge represents the reachability association between the first hole node and the second hole node.

[0019] Traverse all node pairs in the initial node set to complete the construction of all reachability relationships in the hole location node graph.

[0020] Starting from the initial node with the highest processing priority attribute in the hole location node graph, and sequentially selecting the hole location nodes with the largest value estimation results according to the reachability association, a drilling path sequence that traverses the hole location node graph and satisfies the boundary contour and the surface constraint region constraints is generated, including:

[0021] Set the starting node with the highest processing priority attribute as the current hole position node, and initialize the set of visited nodes to an empty set;

[0022] Based on the reachability association, a set of candidate aperture nodes that can be directly reached from the current aperture node is determined, wherein each candidate aperture node in the candidate aperture node set is not in the visited node set;

[0023] The location information of the current hole node, the identification information of the visited node set, and the location information of the candidate hole node set are input into the trained reinforcement learning model to obtain the value estimation results corresponding to each candidate hole node;

[0024] The candidate borehole node with the highest value estimation result is selected as the next borehole node. It is verified whether the connection path from the current borehole node to the next borehole node crosses the surface constraint region or exceeds the boundary contour. If the verification is successful, the next borehole node is added to the drilling path sequence and the current borehole node is updated as the next borehole node. At the same time, the identifier of the next borehole node is added to the visited node set.

[0025] Repeat the above steps until the visited node set contains all borehole nodes, thus completing the generation of the borehole path sequence.

[0026] The value estimation results for each candidate aperture node include:

[0027] The spatial coordinates of the current hole node are concatenated with the spatial coordinates of a preset number of hole nodes in the visited node set in the order of visit to form a local path trajectory feature;

[0028] The spatial coordinates of each candidate hole node and the local path trajectory features are input into the trained reinforcement learning model. The similarity score between each candidate hole node and the local path trajectory features in terms of spatial distribution pattern is calculated, and the similarity score is mapped into a contrast embedding vector.

[0029] The order reward component is calculated based on the relationship between the processing priority attribute of each candidate hole node and the processing priority attribute of the visited hole nodes in the visited node set; the distance reward component is calculated based on the Euclidean distance between the candidate hole node and the current hole node.

[0030] The contrast embedding vector, the order reward component, and the distance reward component are weighted and summed, and the weighted sum is used as the value estimation result for each candidate hole node.

[0031] Based on the drilling path sequence, the drilling equipment is controlled to move sequentially to each hole node along a path within the feasible movement area boundary, avoiding the prohibited passage area, and the drilling operation is performed according to the order of the processing priority attribute, including:

[0032] Based on the spatial coordinates of the current target hole node and the next target hole node, and combined with the boundary contour information of the prohibited passage area, a geometric collision detection method is used to determine whether the straight line segment connecting the two hole nodes intersects with the prohibited passage area. When the determination result is that an intersection has occurred, a detour path is planned along the boundary of the feasible movement area. All path points of the detour path are located within the boundary of the feasible movement area and do not overlap with the prohibited passage area.

[0033] The spatial coordinates of each path point of the bypass path are converted into motion control commands for the drilling equipment. The motion control commands include the target position coordinates and movement speed parameters corresponding to each path point. The movement speed parameters are calculated based on the distance between adjacent path points and the dynamic constraints of the drilling equipment.

[0034] The motion control command is sent to the drilling equipment to control the drilling equipment to move sequentially to each path point of the detour path. A drilling operation command is generated according to the processing priority attribute and sent to the drilling equipment to control the drilling equipment to perform the drilling operation.

[0035] A second aspect of the present invention provides a reinforcement learning-based track slab drilling path planning and optimization system, comprising:

[0036] The first unit is used to acquire the boundary contour, surface constraint area, and hole distribution data of the track plate to be processed. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole.

[0037] The second unit is used to represent the boundary contour as the boundary of the feasible movement area and the surface constraint area as the prohibited passage area; calculate whether there is a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association relationship between each hole node, and obtain a hole node map containing the reachability association relationship;

[0038] The third unit is used to generate a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints, starting from the starting node with the highest processing priority attribute in the hole node graph based on a pre-trained reinforcement learning model;

[0039] The fourth unit is used to control the drilling equipment to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area based on the drilling path sequence, and to perform drilling operations in the order of the processing priority attribute.

[0040] A third aspect of the present invention,

[0041] An electronic device is provided, comprising:

[0042] processor;

[0043] Memory used to store processor-executable instructions;

[0044] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0045] Fourth aspect of the embodiments of the present invention,

[0046] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0047] The beneficial effects of this application are as follows:

[0048] By acquiring the boundary contour, surface constraint area, and hole distribution data of the track plate, and transforming them into a representation of feasible moving areas and prohibited areas, accurate modeling of complex processing environments is achieved, improving the accuracy and adaptability of path planning.

[0049] By calculating the reachability relationships between aperture nodes and constructing an aperture node graph, the problem of effectively handling complex constraints in traditional path planning is solved, ensuring that the generated path meets spatial constraint requirements.

[0050] Path planning based on a pre-trained reinforcement learning model can dynamically select the optimal node according to the value estimate. Compared with traditional deterministic algorithms, this method has stronger adaptability and optimization capabilities, can effectively avoid prohibited areas, and ensure that all holes are traversed. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the reinforcement learning-based track slab drilling path planning and optimization method according to an embodiment of the present invention.

[0052] Figure 2 A flowchart illustrating the process of performing a drilling operation. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] refer to Figure 1 and Figure 2 The present invention provides a reinforcement learning-based method for track slab drilling path planning and optimization, comprising:

[0056] The boundary contour, surface constraint area, and hole distribution data of the track plate to be processed are obtained. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole.

[0057] The boundary contour is represented as the boundary of the feasible movement area, and the surface constraint area is represented as the prohibited passage area; the existence of a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area is calculated, the accessibility relationship between each hole node is determined, and a hole node map containing the accessibility relationship is obtained;

[0058] Based on a pre-trained reinforcement learning model, starting from the initial node with the highest processing priority attribute in the hole node graph, and sequentially selecting the hole node with the largest value estimation result according to the reachability association, a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints is generated.

[0059] Based on the drilling path sequence, the drilling equipment is controlled to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area, and the drilling operation is performed in the order of the processing priority attribute.

[0060] In one alternative implementation, the method further includes training the reinforcement learning model:

[0061] The state space of the reinforcement learning model consists of the current hole position, the set of currently processed hole nodes, and the set of remaining unprocessed hole nodes.

[0062] The action space is a set of possible transitions from the current hole node to the remaining unprocessed hole nodes according to the reachability association; the reward function is a function composed of path length cost, the degree of satisfaction of the processing priority attribute, and constraint state, the constraint state including whether the current path violates the boundary contour and the surface constraint region;

[0063] The reinforcement learning model is trained, and during the training process, the negative value of the path movement distance is used as the basic reward component, the hole node is selected as the positive incentive component according to the processing priority attribute order, and the deviation from the boundary contour or entry into the surface constraint region is used as the negative penalty component.

[0064] The reinforcement learning model learns a decision strategy that minimizes the path length while satisfying the reachability association and processing priority attribute constraints by driving the parameter update of the reinforcement learning model through a reward signal consisting of the basic reward component, the positive incentive component, and the negative penalty component.

[0065] First, the state space of the reinforcement learning model is constructed. The state space comprises three key components: the current position coordinates of the hole node, represented by two-dimensional or three-dimensional coordinates (x, y) or (x, y, z); the set of currently processed hole nodes, recording information on all completed hole nodes; and the set of remaining unprocessed hole nodes, representing all holes yet to be processed. This state representation allows the reinforcement learning model to fully understand the current processing progress and the possible subsequent processing targets.

[0066] Secondly, the action space of the reinforcement learning model is defined. The action space is constructed based on reachability relationships, representing which remaining unprocessed hole nodes can be reached from the current hole node. Reachability relationships are typically determined by process constraints, the motion characteristics of the machining equipment, and the geometric features of the part. For example, for a given hole node with position (10,20), possible actions include moving to remaining unprocessed hole nodes at positions (15,25), (12,18), etc., which meet the reachability requirements.

[0067] Then, a reward function is designed, which includes three key components: path length cost, the degree of satisfaction of processing priority attributes, and constraint status. The path length cost reflects the Euclidean distance between adjacent holes or the machine movement cost during actual processing; the degree of satisfaction of processing priority attributes represents the degree of compliance with sequential processing of holes of different priorities; and the constraint status reflects whether the path violates the boundary contour and surface constraint area restrictions.

[0068] Next, the reinforcement learning model is trained. The negative value of the path movement distance is used as the base reward component; for example, if the distance from hole A to hole B is 5 units, a base reward of -5 is given. When selecting hole nodes according to processing priority attributes, a positive incentive component is given; for example, following the rule of "processing high-precision holes first" results in a reward of +3. When the path exceeds the boundary contour or enters the surface constraint region, a negative penalty component is applied; for example, violating constraints results in a penalty of -10.

[0069] During training iterations, the reinforcement learning model continuously adjusts its network parameters based on the aforementioned reward signals. Specifically, it employs a Deep Q-Network (DQN) or a policy gradient method, accumulating training samples through an experience replay mechanism and updating model parameters using gradient descent. For example, for a training sample with the current state of hole (5,10), choosing to move to hole (8,12) yields a comprehensive reward of -1.6, including path cost -3.6, priority compliance +2, and no constraint violation. The model calculates the loss and updates its parameters accordingly, gradually optimizing its decision-making strategy.

[0070] During training, to ensure the model fully explores possible path solutions, an ε-greedy strategy is adopted to balance exploration and utilization. This involves randomly selecting an action with probability ε and choosing the action with the highest current evaluation probability of 1-ε. As training progresses, the ε value gradually decreases from the initial 0.9 to 0.1, prompting the model to shift from exploration to utilizing learned knowledge.

[0071] Furthermore, to enhance training effectiveness, an experience-first replay strategy is implemented, prioritizing samples with larger absolute reward values ​​or larger temporal difference errors for learning, thus accelerating the model's mastery of key decision points. Simultaneously, a target network mechanism is introduced to stabilize the training process through delayed updates, reducing fluctuations in Q-value estimation.

[0072] After training, the reinforcement learning model, given an initial set of holes and a set of remaining holes, can generate a hole machining path that satisfies accessibility and processing priority constraints while minimizing path length, based on the learned policy. This path considers both process constraints and optimizes processing efficiency.

[0073] In practical applications, the trained model can be deployed to the path planning system of a CNC machining center. Combined with part design data and process parameters, it can automatically generate hole machining paths, significantly improving part manufacturing efficiency and quality consistency.

[0074] In one optional implementation, the existence of a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area is calculated, the reachability association between each hole node is determined, and a hole node map containing the reachability association is obtained, including:

[0075] Construct an initial node set containing all hole nodes, and select the first hole node and the second hole node from the initial node set as the node pair to be verified;

[0076] A connecting line segment is generated based on the spatial coordinates of the first hole node and the second hole node. The connecting line segment represents the straight path of the drilling equipment moving from the first hole node to the second hole node.

[0077] When the connecting line segment is completely within the boundary of the feasible movement area and the connecting line segment does not intersect with the prohibited passage area, a directed edge is established in the hole node graph from the first hole node to the second hole node. The directed edge represents the reachability association between the first hole node and the second hole node.

[0078] Traverse all node pairs in the initial node set to complete the construction of all reachability relationships in the hole location node graph.

[0079] In actual drilling operations, to ensure that the drilling equipment can access all boreholes in a reasonable order, it is necessary to first determine the reachability relationships between the drilling equipment and each borehole node. Based on the given borehole distribution data, an initial node set containing all borehole nodes is constructed. This set can be represented as V={v1,v2,...,vn}, where vi represents the i-th borehole node and n represents the number of borehole nodes. Each borehole node contains its spatial coordinate information (x,y,z). For example, in a planar drilling task, there are borehole nodes v1=(10,15,0), v2=(25,30,0), etc.

[0080] The first and second hole nodes are selected from the initial node set as the node pair to be verified. This process uses a traversal approach, sequentially taking any two nodes from the node set for verification. For example, first, v1 and v2 are taken as the node pair to be verified, then v1 and v3 are taken, and so on.

[0081] For each node to be verified, a connecting line segment is generated based on the spatial coordinates of the first and second borehole nodes. This connecting line segment can be represented by the parametric equation: P(t) = P1 + t·(P2 - P1), where P1 and P2 are the spatial coordinates of the first and second borehole nodes, respectively, and t ∈ [0,1]. This connecting line segment represents the path taken by the drilling equipment to move directly from the first borehole node to the second borehole node.

[0082] It is necessary to determine whether the connecting line segment meets two conditions: it must be completely located within the boundary of the feasible movement area and it must not intersect with any prohibited passage area. For the first condition, check whether multiple sampling points on the connecting line segment are all located within the feasible movement area. A uniform sampling method can be used, taking k points on the connecting line segment for detection. For the second condition, calculate the intersection points of the connecting line segment with each prohibited passage area. If no intersection points exist, it indicates that the connecting line segment does not intersect with any prohibited passage area.

[0083] When the connecting line segment satisfies the above two conditions, a directed edge is established in the hole node graph, pointing from the first hole node to the second hole node. This directed edge can be represented as e=(v1,v2), representing the reachability relationship between the first hole node and the second hole node. Simultaneously, the weight of this directed edge is recorded; for example, the length of the connecting line segment can be used as the weight to represent the cost of the drilling equipment moving between the two nodes.

[0084] In practical applications, feasible movement areas are typically represented by polygons, while prohibited areas may consist of multiple irregularly shaped regions. To determine whether a connecting line segment lies within a feasible movement area, a ray casting method can be used to check if a sampling point on the line segment is located inside the polygon. Conversely, to determine whether a connecting line segment intersects with a prohibited area, a line segment-polygon intersection detection algorithm can be employed.

[0085] For example, in a specific scenario, the feasible movement area is a rectangular region [(0,0),(100,0),(100,100),(0,100)], and the prohibited area is a circular region with a center point of (50,50) and a radius of 10. For hole nodes v1=(10,20) and v2=(80,70), it is necessary to determine whether the connecting line segment crosses the prohibited area. By calculating the intersection point of the connecting line segment and the circle, it can be determined that the connecting line segment does indeed cross the prohibited area, therefore, no directed edge is established from v1 to v2.

[0086] Following the above method, all node pairs in the initial node set are traversed, the reachability relationship between each pair of nodes is verified in turn, and the corresponding directed edges are established when the conditions are met, thus completing the construction of all reachability relationships in the hole location node graph.

[0087] The constructed borehole node graph can be represented as G=(V,E), where V is the set of borehole nodes and E is the set of directed edges. This graph structure fully represents the reachability relationships between all borehole nodes, providing a foundation for subsequent drilling path planning. In practical applications, based on this graph structure, graph theory algorithms (such as shortest path algorithms and traveling salesman problem algorithms) can be further used to plan the movement path of drilling equipment, thereby optimizing the drilling operation process and improving production efficiency.

[0088] In one optional implementation, starting from the starting node with the highest processing priority attribute in the hole location node graph, the hole location nodes with the largest value estimation results are selected sequentially according to the reachability association relationship to generate a drilling path sequence that traverses the hole location node graph and satisfies the boundary contour and the surface constraint region constraints, including:

[0089] Set the starting node with the highest processing priority attribute as the current hole position node, and initialize the set of visited nodes to an empty set;

[0090] Based on the reachability association, a set of candidate aperture nodes that can be directly reached from the current aperture node is determined, wherein each candidate aperture node in the candidate aperture node set is not in the visited node set;

[0091] The location information of the current hole node, the identification information of the visited node set, and the location information of the candidate hole node set are input into the trained reinforcement learning model to obtain the value estimation results corresponding to each candidate hole node;

[0092] The candidate borehole node with the highest value estimation result is selected as the next borehole node. It is verified whether the connection path from the current borehole node to the next borehole node crosses the surface constraint region or exceeds the boundary contour. If the verification is successful, the next borehole node is added to the drilling path sequence and the current borehole node is updated as the next borehole node. At the same time, the identifier of the next borehole node is added to the visited node set.

[0093] Repeat the above steps until the visited node set contains all borehole nodes, thus completing the generation of the borehole path sequence.

[0094] In this embodiment, the three-dimensional model data of the workpiece is first acquired. This model contains multiple holes that need to be machined. Based on process requirements and machining characteristics, a machining priority attribute is assigned to each hole node, and accessibility relationships between hole nodes are established based on machining feasibility, forming a complete hole node diagram. Simultaneously, the boundary contour and surface constraint regions of the workpiece are determined; these constraints will be used to verify the legality of the drilling path.

[0095] When generating the drilling path sequence, reinforcement learning is used to optimize the path to achieve an efficient processing procedure. The specific implementation steps are as follows:

[0096] Set the hole node with the highest processing priority attribute as the current hole node. For example, in structural component machining, critical connection holes on main load-bearing members typically have the highest processing priority. Initialize the visited node set to an empty set to record nodes already added to the drilling path.

[0097] Based on pre-established reachability relationships, a set of candidate hole nodes directly reachable from the current hole node is determined. Reachability relationships are typically established based on factors such as Euclidean distance between nodes, machining requirements, and tool accessibility. In practical applications, a maximum reachability distance threshold is set, such as 150 mm; nodes exceeding this distance are considered inaccessible. Furthermore, nodes from the already visited node set are excluded to ensure that none of the nodes in the candidate set have been visited, thus avoiding path duplication.

[0098] The location information of the current hole node, the identifiers of the visited node set, and the location information of the candidate hole node set are input into the trained reinforcement learning model. This model, built on a Deep Q-Network (DQN) or policy gradient method, is trained on a large amount of historical processing data and can evaluate the value of each candidate node. The model input includes the 3D coordinates (x, y, z) of the current node, the identifier list of visited nodes, and the 3D coordinates of each candidate node. The model output is the value estimate for each candidate hole node, typically ranging from [0, 1], reflecting the rationality of selecting that node as the next processing location.

[0099] Based on the evaluation results of the reinforcement learning model, the candidate hole node with the highest value estimate is selected as the next hole node. For example, in a certain iteration, the value estimates of candidate nodes A, B, and C are 0.85, 0.72, and 0.91, respectively, so node C is selected as the next hole node. Path verification is then performed to check whether the connection path from the current hole node to the next hole node crosses the surface constraint region or exceeds the boundary contour.

[0100] The verification process employs a discrete sampling method, uniformly sampling multiple points (e.g., one sampling point per millimeter) along the straight path between two nodes. Each sampling point is checked to see if it lies within the surface constraint area or exceeds the boundary contour. If all sampling points pass verification, the path verification is successful; otherwise, verification fails, and the candidate node with the second highest value estimate needs to be re-selected for verification until a valid path is found or all candidate nodes have been traversed.

[0101] Once verification is successful, the next borehole node is added to the drilling path sequence, the current borehole node is updated to the next borehole node, and the identifier of the next borehole node is added to the set of visited nodes. For example, if node C passes verification, it is added to the path sequence, the current node is updated to C, and the identifier of C is added to the set of visited nodes.

[0102] Repeat the above process of identifying candidate nodes, evaluating their value, selecting the next node, verifying the path, and updating the status until the set of visited nodes contains all hole nodes, indicating that all holes have been processed. At this point, the drilling path sequence is generated.

[0103] In practical applications, this method can be used for automated drilling of complex parts. Taking a structural component as an example, this component has 120 connecting holes of different specifications distributed on multiple different planes. The drilling path sequence generated by this method reduces the drill bit movement distance by about 25%, improves processing efficiency by about 20%, and effectively avoids all surface constraint areas such as reinforcing ribs and edge areas.

[0104] It is worth noting that the selection and training of the reinforcement learning model are crucial in the implementation process. The model training employs a reward mechanism, providing positive rewards for choices that reduce the total path length and avoid constrained regions, while penalizing choices that increase path complexity. The model training data comes from historical processed cases, typically requiring tens of thousands of iterations to achieve the desired results. The trained model can effectively predict the best next step based on the current state, significantly improving path planning efficiency.

[0105] Furthermore, the value estimation results can be adjusted according to specific process requirements during application. For example, for holes with particularly high precision requirements, a weighting factor can be added to the value estimation to increase their likelihood of being prioritized. Similarly, for groups of holes that require adjacent processing in a specific process flow, their continuous processing can be achieved by adjusting the weights.

[0106] The drilling path sequence generated by the above method not only meets the process constraints but also optimizes the processing path, providing an efficient and reliable path planning solution for automated drilling.

[0107] In one optional implementation, obtaining the value estimation results corresponding to each candidate aperture node includes:

[0108] The spatial coordinates of the current hole node are concatenated with the spatial coordinates of a preset number of hole nodes in the visited node set in the order of visit to form a local path trajectory feature;

[0109] The spatial coordinates of each candidate hole node and the local path trajectory features are input into the trained reinforcement learning model. The similarity score between each candidate hole node and the local path trajectory features in terms of spatial distribution pattern is calculated, and the similarity score is mapped into a contrast embedding vector.

[0110] The order reward component is calculated based on the relationship between the processing priority attribute of each candidate hole node and the processing priority attribute of the visited hole nodes in the visited node set; the distance reward component is calculated based on the Euclidean distance between the candidate hole node and the current hole node.

[0111] The contrast embedding vector, the order reward component, and the distance reward component are weighted and summed, and the weighted sum is used as the value estimation result for each candidate hole node.

[0112] When obtaining the value estimation results corresponding to each candidate hole node, the spatial coordinates of the current hole node are first concatenated with the spatial coordinates of a preset number of hole nodes in the visited node set according to the visiting order to form a local path trajectory feature. In practical applications, assuming the spatial coordinates of the current hole node are (x_c, y_c, z_c), and the spatial coordinates of the last n hole nodes in the visited node set are (x_1, y_1, z_1), (x_2, y_2, z_2), ..., (x_n, y_n, z_n), the local path trajectory feature can be represented as a 3(n+1)-dimensional vector: [(x_c, y_c, z_c), (x_1, y_1, z_1), ..., (x_n, y_n, z_n)]. The preset number n can be set according to the actual application requirements, generally between 4 and 8, which can capture sufficient historical information without causing excessive computational complexity.

[0113] The spatial coordinates of each candidate hole node and the local path trajectory features are input into the trained reinforcement learning model. A similarity score is calculated between each candidate hole node and the local path trajectory features in terms of spatial distribution patterns, and this similarity score is mapped into a contrastive embedding vector. Specifically, for each candidate hole node, its spatial coordinates are (x_i, y_i, z_i), which are concatenated with the local path trajectory features and then input into the reinforcement learning model. This reinforcement learning model employs a dual-channel structure: one channel processes candidate hole node information, and the other channel processes local path trajectory features. After feature extraction using a multilayer perceptron, an attention mechanism is used to calculate the similarity score between the two. The similarity score reflects the degree of matching between the spatial distribution patterns of the candidate hole and the historical access path. Subsequently, a non-linear mapping function converts the similarity score into a contrastive embedding vector. This embedding vector is typically set to 16 or 32 dimensions to capture richer semantic information.

[0114] The sequential reward component is calculated based on the relationship between the processing priority attributes of each candidate hole node and the processing priority attributes of the visited hole nodes in the visited node set. The distance reward component is calculated based on the Euclidean distance between the candidate hole node and the current hole node. Specifically, the calculation of the sequential reward component considers the continuity and consistency of processing priorities. Assuming the priority of the candidate hole node is p_c, and the priorities of the last m nodes in the visited node set are p_1, p_2, ..., p_m, the sequential reward component can be calculated as follows: if p_c forms a monotonically increasing or monotonically decreasing sequence with p_1, p_2, ..., p_m, a positive reward is given; if the monotonicity of the existing sequence is broken, a negative penalty is given. The distance reward component is calculated by plotting the Euclidean distance d between the candidate hole node and the current hole node, and using a decreasing function to map the distance to a reward value, such as exp(-d / d_0), where d_0 is a normalization parameter, usually set to twice the average distance between holes.

[0115] The contrast embedding vector, the order reward component, and the distance reward component are weighted and summed, and the result of the weighted sum is used as the value estimation result for each candidate hole node. In this process, the contrast embedding vector is first mapped to a scalar value v_embed through a fully connected layer, and then weighted and summed with the order reward component v_order and the distance reward component v_dist. Here, w_1, w_2, and w_3 are weight coefficients, which can be set empirically or automatically adjusted through reinforcement learning. The values ​​of the weight coefficients reflect the importance of different factors in path planning. Typically, w_1 is set to 0.5, and w_2 and w_3 are each set to 0.25, but appropriate adjustments can be made for different processing scenarios.

[0116] In practical applications, when faced with multiple candidate borehole locations, this value estimation method effectively balances the similarity of spatial distribution patterns, the rationality of processing order, and the economy of travel distance. For example, when two candidate borehole locations are close to each other, the system will prioritize the borehole location that is more similar to the historical path pattern and meets the processing priority requirements, thus forming a processing path that better meets the global optimization objective. Simultaneously, the reinforcement learning model will continuously optimize with accumulated experience, enabling the system to adapt to different types of drilling layouts and processing needs, thereby improving the overall path planning efficiency.

[0117] A key advantage of this value estimation method is that it comprehensively considers historical path information, processing constraints, and distance factors, avoiding the problem of traditional greedy algorithms easily getting trapped in local optima. Through comparative learning, the system can learn effective spatial distribution patterns from historically successful paths and apply these patterns to new path planning, achieving knowledge transfer and experience reuse, further improving the intelligence and adaptability of path planning.

[0118] In one optional implementation, controlling the drilling equipment to move sequentially to each hole node along a path within the feasible movement area boundary and avoiding the prohibited passage area based on the drilling path sequence, and performing drilling operations according to the order of the processing priority attribute, includes:

[0119] Based on the spatial coordinates of the current target hole node and the next target hole node, and combined with the boundary contour information of the prohibited passage area, a geometric collision detection method is used to determine whether the straight line segment connecting the two hole nodes intersects with the prohibited passage area. When the determination result is that an intersection has occurred, a detour path is planned along the boundary of the feasible movement area. All path points of the detour path are located within the boundary of the feasible movement area and do not overlap with the prohibited passage area.

[0120] The spatial coordinates of each path point of the bypass path are converted into motion control commands for the drilling equipment. The motion control commands include the target position coordinates and movement speed parameters corresponding to each path point. The movement speed parameters are calculated based on the distance between adjacent path points and the dynamic constraints of the drilling equipment.

[0121] The motion control command is sent to the drilling equipment to control the drilling equipment to move sequentially to each path point of the detour path. A drilling operation command is generated according to the processing priority attribute and sent to the drilling equipment to control the drilling equipment to perform the drilling operation.

[0122] In the control system of drilling equipment, intelligent path planning and processing control can be achieved based on hole location node information, feasible movement area, and prohibited passage area data. The implementation method of controlling the drilling equipment to move sequentially to each hole location node along a path within the feasible movement area boundary while avoiding prohibited passage areas, and executing drilling operations according to the processing priority attribute, is as follows:

[0123] The drilling equipment starts from its current position and needs to move to the next target hole node. At this point, it's necessary to determine whether the straight path between the current position and the target position crosses a prohibited area. Specifically, based on the spatial coordinates (x1, y1, z1) of the current target hole node and the spatial coordinates (x2, y2, z2) of the next target hole node, a parametric equation for the straight line segment connecting these two points is generated. Simultaneously, combining the boundary contour information of the prohibited area (usually stored as polygons or geometric shapes), a geometric collision detection method is used for judgment.

[0124] Collision detection can be performed using the ray method, treating straight line segments as rays and detecting whether they intersect the boundary of a restricted area. Specifically, for each restricted area (assumed to be a polygon), check whether the straight line segment intersects each edge of the polygon. If an intersection point exists, it indicates that the straight line path intersects the restricted area, and an alternative route needs to be planned.

[0125] When the assessment indicates an intersection has occurred, a detour route needs to be planned within the feasible movement area boundary. The detour route planning employs an improved method. The algorithm discretizes the feasible movement region into a grid, with each grid point representing a possible path point. First, cells representing prohibited areas are marked on the grid. Then, it utilizes... The algorithm searches for the shortest path from the starting point to the target point, avoiding all cells marked as forbidden during the search.

[0126] The steps for detour path planning include: first, constructing a grid map, with the grid size determined based on the size of the drilling equipment and the required processing precision; second, mapping prohibited areas onto the grid and marking them as impassable; then, setting the starting point as the current borehole node and the ending point as the next target borehole node; finally, executing the detour path planning. The search algorithm generates a series of waypoints, all of which are located within the boundaries of the feasible movement area and do not overlap with the prohibited passage area.

[0127] To optimize path smoothness, The initial path generated by the algorithm is smoothed. B-spline interpolation is used to fit discrete path points, generating a smooth and continuous motion trajectory. This reduces the acceleration and deceleration frequency of the drilling equipment during operation, improving motion efficiency.

[0128] After obtaining the detour path, the path points need to be converted into motion control commands that can be executed by the drilling equipment. First, the spatial coordinates of each path point are extracted to form a series of path point sets. For each path point, a corresponding target position coordinate command is generated.

[0129] The calculation of the moving speed parameter needs to consider the distance between adjacent path points and the dynamic constraints of the drilling equipment. Specifically, for adjacent path points i and i+1, the distance d_i between them is calculated as √[(x_i+1-x_i)]. 2 +(y_i+1-y_i) 2 +(z_i+1-z_i) 2 Based on the distance and the device's maximum acceleration a_max and maximum speed v_max, calculate the appropriate moving speed v_i.

[0130] If the distance is short, the speed calculation formula is v_i=min(√( (v_max); If the distance is long, a trapezoidal velocity curve can be used, including acceleration, constant speed, and deceleration sections. At corners, the speed is adjusted according to the turning angle θ; the larger the angle, the lower the speed. Typically, a setting can be used. To ensure a smooth cornering experience.

[0131] The format of motion control commands is {(x1,y1,z1,v1),(x2,y2,z2,v2),...,(xn,yn,zn,vn)}, where (xi,yi,zi) represent the target position coordinates, and vi represents the velocity parameter for moving to that position. These commands are sent to the control system of the drilling equipment via a communication interface (such as Ethernet, serial port, or fieldbus).

[0132] After receiving motion control commands, the drilling equipment starts the motor drive system, controlling each axis motion mechanism to move sequentially to each path point at a specified speed. The equipment's motion controller performs interpolation calculations based on the received commands to generate a smooth motion trajectory, ensuring that it passes through the designated position at an appropriate speed.

[0133] Once the drilling equipment reaches the target hole location, it generates a drilling operation command based on the processing priority attribute of that hole. The drilling operation command includes parameters such as drill bit selection, drilling depth, rotational speed, and feed rate. These parameters are preset according to the material properties of the workpiece and the hole requirements. Holes with higher priority will be drilled first.

[0134] After the drilling operation is completed, the drilling equipment moves to the next target hole and repeats the above process of path planning, motion control, and drilling until all holes are machined. In this way, the drilling equipment can intelligently avoid obstacles in complex environments and efficiently complete drilling tasks.

[0135] The present invention provides a reinforcement learning-based track slab drilling path planning and optimization system, comprising:

[0136] The first unit is used to acquire the boundary contour, surface constraint area, and hole distribution data of the track plate to be processed. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole.

[0137] The second unit is used to represent the boundary contour as the boundary of the feasible movement area and the surface constraint area as the prohibited passage area; calculate whether there is a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association relationship between each hole node, and obtain a hole node map containing the reachability association relationship;

[0138] The third unit is used to generate a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints, starting from the starting node with the highest processing priority attribute in the hole node graph based on a pre-trained reinforcement learning model;

[0139] The fourth unit is used to control the drilling equipment to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area based on the drilling path sequence, and to perform drilling operations in the order of the processing priority attribute.

[0140] A third aspect of the present invention,

[0141] An electronic device is provided, comprising:

[0142] processor;

[0143] Memory used to store processor-executable instructions;

[0144] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0145] Fourth aspect of the embodiments of the present invention,

[0146] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0147] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for track slab drilling path planning and optimization based on reinforcement learning, characterized in that, include: Obtain the boundary contour, surface constraint area, and hole distribution data of the track plate to be processed. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole. The boundary contour is represented as the boundary of the feasible movement region, and the surface constraint region is represented as the prohibited passage region; Calculate whether there exists a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association between each hole node, and obtain a hole node map containing the reachability association; Based on a pre-trained reinforcement learning model, starting from the initial node with the highest processing priority attribute in the hole node graph, candidate hole nodes with the largest value estimation results are selected sequentially according to the reachability association relationship, generating a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints; Based on the drilling path sequence, the drilling equipment is controlled to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area, and the drilling operation is performed in the order of the processing priority attribute. The method further includes: Obtain the value estimation results corresponding to each candidate pore location node, including: Set the starting node with the highest processing priority attribute as the current hole position node, and initialize the set of visited nodes to an empty set; Based on the reachability association, a set of candidate aperture nodes that can be directly reached from the current aperture node is determined, wherein each candidate aperture node in the candidate aperture node set is not in the visited node set; The spatial coordinates of the current hole node are concatenated with the spatial coordinates of a preset number of hole nodes in the visited node set in the order of visit to form a local path trajectory feature; The spatial coordinates of each candidate hole node and the local path trajectory features are input into the trained reinforcement learning model. The similarity score between each candidate hole node and the local path trajectory features in terms of spatial distribution pattern is calculated, and the similarity score is mapped into a contrast embedding vector. The order reward component is calculated based on the relationship between the processing priority attribute of each candidate hole node and the processing priority attribute of the visited hole nodes in the visited node set; the distance reward component is calculated based on the Euclidean distance between the candidate hole node and the current hole node. The contrast embedding vector, the order reward component, and the distance reward component are weighted and summed, and the weighted sum is used as the value estimation result for each candidate hole node.

2. The method according to claim 1, characterized in that, The method also includes training the reinforcement learning model: The state space of the reinforcement learning model consists of the current hole position, the set of currently processed hole nodes, and the set of remaining unprocessed hole nodes. The action space is a set of possible transitions from the current hole node to the remaining unprocessed hole nodes based on the reachability association. The reward function is a function composed of path length cost, the degree of satisfaction of the processing priority attribute, and constraint state, wherein the constraint state includes whether the current path violates the boundary contour and the surface constraint region; The reinforcement learning model is trained, and during the training process, the negative value of the path movement distance is used as the basic reward component, the hole node is selected as the positive incentive component according to the processing priority attribute order, and the deviation from the boundary contour or the entry into the surface constraint region is used as the negative penalty component. The reinforcement learning model learns a decision strategy that minimizes the path length while satisfying the reachability association and processing priority attribute constraints by driving the parameter update of the reinforcement learning model through a reward signal consisting of the basic reward component, the positive incentive component, and the negative penalty component.

3. The method according to claim 1, characterized in that, Calculate whether there exists a connection path between any two aperture nodes in the aperture distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association between each aperture node, and obtain an aperture node graph containing the reachability association, including: Construct an initial node set containing all hole nodes, and select the first hole node and the second hole node from the initial node set as the node pair to be verified; A connecting line segment is generated based on the spatial coordinates of the first hole node and the second hole node. The connecting line segment represents the straight path of the drilling equipment moving from the first hole node to the second hole node. When the connecting line segment is completely within the boundary of the feasible movement area and there is no intersection between the connecting line segment and the prohibited passage area, a directed edge is established in the hole node graph from the first hole node to the second hole node. The directed edge represents the reachability relationship between the first hole node and the second hole node. Traverse all node pairs in the initial node set to complete the construction of all reachability relationships in the hole node graph.

4. The method according to claim 1, characterized in that, Starting from the starting node with the highest processing priority attribute in the hole location node graph, candidate hole location nodes with the largest value estimation results are selected sequentially according to the reachability association relationship to generate a drilling path sequence that traverses the hole location node graph and satisfies the boundary contour and the surface constraint region constraints, including: The candidate borehole node with the highest value estimation result is selected as the next borehole node. It is verified whether the connection path from the current borehole node to the next borehole node crosses the surface constraint region or exceeds the boundary contour. If the verification is successful, the next borehole node is added to the drilling path sequence and the current borehole node is updated as the next borehole node. At the same time, the identifier of the next borehole node is added to the visited node set. Repeat the above steps until the visited node set contains all borehole nodes, thus completing the generation of the borehole path sequence.

5. The method according to claim 1, characterized in that, Based on the drilling path sequence, the drilling equipment is controlled to move sequentially to each hole node along a path within the feasible movement area boundary, avoiding the prohibited passage area, and the drilling operation is performed according to the order of the processing priority attribute, including: Based on the spatial coordinates of the current target hole node and the next target hole node, and combined with the boundary contour information of the prohibited passage area, a geometric collision detection method is used to determine whether the straight line segment connecting the two hole nodes intersects with the prohibited passage area. When the determination result is that an intersection has occurred, a detour path is planned along the boundary of the feasible movement area. All path points of the detour path are located within the boundary of the feasible movement area and do not overlap with the prohibited passage area. The spatial coordinates of each path point of the detour path are converted into motion control commands for the drilling equipment. The motion control commands include the target position coordinates and movement speed parameters corresponding to each path point. The movement speed parameters are calculated based on the distance between adjacent path points and the dynamic constraints of the drilling equipment. The motion control command is sent to the drilling equipment to control the drilling equipment to move sequentially to each path point of the detour path. A drilling operation command is generated according to the processing priority attribute and sent to the drilling equipment to control the drilling equipment to perform the drilling operation.

6. A reinforcement learning-based track slab drilling path planning and optimization system, used to implement the method as described in any one of claims 1-5, characterized in that, include: The first unit is used to acquire the boundary contour, surface constraint area, and hole distribution data of the track plate to be processed. The hole distribution data includes the spatial coordinates and processing priority attributes of each hole. The second unit is used to represent the boundary contour as the boundary of the feasible movement region and the surface constraint region as the prohibited passage region; Calculate whether there exists a connection path between any two hole nodes in the hole distribution data that does not cross the prohibited passage area and does not exceed the boundary of the feasible movement area, determine the reachability association between each hole node, and obtain a hole node map containing the reachability association; The third unit is used to generate a drilling path sequence that traverses the hole node graph and satisfies the boundary contour and the surface constraint region constraints, starting from the starting node with the highest processing priority attribute in the hole node graph based on a pre-trained reinforcement learning model; The fourth unit is used to control the drilling equipment to move sequentially to each hole node along the path within the boundary of the feasible movement area and avoiding the prohibited passage area based on the drilling path sequence, and to perform drilling operations in the order of the processing priority attribute.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images