A method for optimizing control of a robot arm based on reinforcement learning and related devices

Abstract

The application relates to the technical field of mechanical arm control, and provides a mechanical arm control optimization method based on reinforcement learning and related equipment, which comprises the following steps: constructing an optimization objective function and constraint conditions of a target surface polishing process; defining a state space of the target surface polishing process according to basic data and flaw-related data of all mechanical arms, and obtaining a process execution sequence of each mechanical arm by solving the state space based on reinforcement learning; mapping the process execution sequences of all the mechanical arms to obtain a graph structure, and iteratively destroying and repairing nodes of the graph structure based on the optimization objective function and the constraint conditions and the features of the nodes in the graph structure to obtain a final graph structure; determining the final process execution sequence of each mechanical arm from the final graph structure, and controlling the mechanical arms according to the final process execution sequence of each mechanical arm. The method can improve the precision of mechanical arm control in the face of complex processes.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and in particular to a robotic arm control optimization method and related equipment based on reinforcement learning. Background Technology

[0002] High-performance titanium alloy parts play a crucial role in high-end manufacturing fields such as aerospace, precision machinery, and biomedicine. However, during the initial manufacturing and processing of these parts, various types of micro-defects (such as microcracks, localized oxidation points, and pits) often appear on their surfaces. If these micro-defects are not removed in a timely and precise manner, they will lead to stress concentration and potential early failure, thereby reducing the service life and reliability of the parts. To meet stringent quality requirements, achieving high-precision automated grinding of various defects on the surface of titanium rods has become a key technological challenge for improving product quality and reliability.

[0003] Traditional surface polishing relies heavily on human experience and manual operation, resulting in low efficiency and difficulty in ensuring consistent quality. With the rapid development of intelligent manufacturing and robotics, automated surface polishing using industrial robotic arms has become an industry trend. However, achieving intelligent handling of various types of defects in actual production still faces the following key challenges: Differentiated process requirements for various types of defects: Different types of defects require different polishing parameters and strategies, making it difficult to meet multiple processing requirements with a single solution. This significantly increases the complexity of task allocation and process control.

[0004] Nonlinear timing relationship between dual-wheel grinding and forced waiting time: To prevent overheating from degrading material properties, each defect must be polished in two rounds, with a mandatory waiting time inserted between the two rounds. This mandatory waiting time transforms the scheduling problem from a simple sequential execution into a non-linear, segmented temporal structure, posing a significant challenge to decision optimization.

[0005] Spatiotemporal coupling of multi-robotic arm collaboration and path optimization: To improve productivity, multiple robotic arms need to work in parallel. However, how to simultaneously optimize the task allocation and path planning of multiple robotic arms in both time and space dimensions to reduce idle and ineffective movements, avoid resource conflicts, and ultimately reduce the overall completion time is a complex optimization problem with high dimensionality and strong coupling.

[0006] To address the aforementioned challenges, existing research primarily employs operations research, heuristics, or metaheuristic methods. However, when problems involve multiple process types, forced waiting, multi-resource parallelism, and path optimization, these methods face bottlenecks in solution quality and scalability. In recent years, deep reinforcement learning (DRL) and graph neural networks (GNN) have increasingly demonstrated their potential in solving complex industrial optimization problems. However, research on integrating multi-agent reinforcement learning (MARL) with GNN-driven large neighborhood search and systematically applying it to industrial optimization problems involving multiple complex elements such as forced waiting in dual-wheel processes, multi-robotic arm collaboration, and path planning remains insufficient, resulting in poor robotic arm control accuracy when facing complex processes. Summary of the Invention

[0007] This application provides a reinforcement learning-based robotic arm control optimization method and related equipment, which can solve the problem of poor control accuracy of robotic arms when facing complex processes.

[0008] In a first aspect, embodiments of this application provide a robotic arm control optimization method based on reinforcement learning, the robotic arm control optimization method comprising: Acquire basic data of multiple robotic arms in the target surface grinding process, as well as defect-related data of the target surface grinding process; the target surface grinding process includes multiple grinding processes to be ground, and the multiple grinding processes include the first grinding process and the second grinding process for multiple defect points on the surface to be ground, and the basic data includes the initial position and moving speed of the robotic arms. Construct the objective function and constraints for the target surface grinding process; the objective function describes the total completion time of the target surface grinding process. Based on the basic data and defect-related data of all robotic arms, the state space of the target surface polishing process is defined, and the initial process execution sequence of each robotic arm is obtained by using reinforcement learning based on the state space; the initial process execution sequence includes at least one polishing process executed by the robotic arm in chronological order. The initial process execution sequence of all robotic arms is mapped to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, the nodes of the graph structure are iteratively destroyed and repaired to obtain the final graph structure. In the graph structure, multiple nodes correspond one-to-one with multiple grinding processes, and the edges between nodes are the connection relationship or spatial adjacency relationship between the corresponding grinding processes. The final process execution sequence of each robotic arm is determined from the final diagram structure, and the robotic arm is controlled according to the final process execution sequence of each robotic arm.

[0009] Secondly, embodiments of this application provide a reinforcement learning-based robotic arm control optimization device, comprising: The acquisition module is used to acquire basic data of multiple robotic arms in the target surface grinding process, as well as defect-related data of the target surface grinding process; the target surface grinding process includes multiple grinding processes to be ground, and the multiple grinding processes include the first grinding process and the second grinding process for multiple defect points on the surface to be ground; the basic data includes the initial position and moving speed of the robotic arms. The construction module is used to construct the optimization objective function and constraints for the target surface grinding process; the optimization objective function describes the total completion time of the target surface grinding process. A definition module is used to define the state space of the target surface polishing process based on the basic data and defect-related data of all robotic arms, and to solve the initial process execution sequence of each robotic arm using reinforcement learning based on the state space; the initial process execution sequence includes at least one polishing process executed by the robotic arm in chronological order. The mapping module is used to map the initial process execution sequence of all robotic arms to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, the module iteratively destroys and repairs the nodes in the graph structure to obtain the final graph structure. In the graph structure, multiple nodes correspond one-to-one with multiple grinding processes, and the edges between nodes represent the connection relationship or spatial adjacency relationship between the corresponding grinding processes. The control module is used to determine the final process execution sequence of each robotic arm from the final diagram structure, and to control the robotic arm according to the final process execution sequence of each robotic arm.

[0010] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned reinforcement learning-based robotic arm control optimization method.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based robotic arm control optimization method.

[0012] The above-mentioned solution in this application has the following beneficial effects: In the embodiments of this application, basic data of multiple robotic arms in the target surface polishing process and defect-related data of the target surface polishing process are obtained. Then, the optimization objective function and constraints of the target surface polishing process are constructed. Based on the basic data and defect-related data of all robotic arms, the state space of the target surface polishing process is defined. Based on the state space, the initial process execution sequence of each robotic arm is obtained by using reinforcement learning. Then, the initial process execution sequences of all robotic arms are mapped to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, the graph structure is iteratively broken and repaired to obtain the final graph structure. Finally, the final process execution sequence of each robotic arm is determined from the final graph structure, and the robotic arm is controlled according to the final process execution sequence of each robotic arm. Among them, an optimization objective function for the surface grinding process is constructed to achieve a quantitative representation of the grinding time of complex processes. By mapping the process execution sequence of the robotic arm to a graph structure, the process of the robotic arm under complex processes can be visualized. By destroying and repairing nodes in the graph structure, the correlation and connection rationality between the robotic arm processes can be effectively explored. With the grinding process time as the optimization objective, the allocation of grinding processes of the robotic arm is optimized to ensure that the control accuracy of the robotic arm under complex processes is effectively improved while reducing the process time.

[0013] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating a reinforcement learning-based robotic arm control optimization method provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a reinforcement learning-based robotic arm control optimization device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0020] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] To address the problem of poor precision in robotic arm control when dealing with complex processes, this application provides a reinforcement learning-based robotic arm control optimization method. This method acquires basic data of multiple robotic arms in a target surface grinding process, as well as defect-related data for the process. It then constructs an optimization objective function and constraints for the target surface grinding process. Based on the basic and defect-related data of all robotic arms, it defines the state space of the target surface grinding process. Using reinforcement learning, it obtains the initial process execution sequence for each robotic arm. The initial process execution sequences of all robotic arms are then mapped to obtain a graph structure. Based on the optimization objective function, constraints, and the characteristics of nodes in the graph structure, iterative node destruction and repair are performed to obtain the final graph structure. Finally, the final process execution sequence for each robotic arm is determined from the final graph structure, and the robotic arm is controlled according to this sequence. Among them, an optimization objective function for the surface grinding process is constructed to achieve a quantitative representation of the grinding time of complex processes. By mapping the process execution sequence of the robotic arm to a graph structure, the process of the robotic arm under complex processes can be visualized. By destroying and repairing nodes in the graph structure, the correlation and connection rationality between the robotic arm processes can be effectively explored. With the grinding process time as the optimization objective, the allocation of grinding processes of the robotic arm is optimized to ensure that the control accuracy of the robotic arm under complex processes is effectively improved while reducing the process time.

[0023] The following is an illustrative example of the reinforcement learning-based robotic arm control optimization method provided in this application.

[0024] like Figure 1 As shown, the reinforcement learning-based robotic arm control optimization method provided in this application includes the following steps: Step 11: Obtain basic data of multiple robotic arms in the target surface polishing process, as well as defect-related data of the target surface polishing process.

[0025] The target surface polishing process is the polishing process for titanium alloy bars that require surface imperfection removal. This target surface polishing process includes multiple polishing steps on the surface to be polished (i.e., the surface of the titanium alloy bar). These multiple polishing steps include a first polishing step and a second polishing step for multiple imperfections on the surface to be polished (such as microcracks, localized oxidation points, pits, etc. on the titanium alloy bar). That is, each imperfection requires two polishing steps. If the number of imperfections to be polished is X, then the number of polishing steps is 2X. The first and second polishing steps for each imperfection are two independent polishing steps, with the first polishing step performed before the second polishing step.

[0026] The aforementioned basic data includes the robotic arm's initial position and its movement speed towards the target. The aforementioned defect-related data includes the location of the defect and the progress of the polishing process.

[0027] In some embodiments of this application, basic data of the robotic arm can be obtained using devices such as sensors, and defect-related data can be obtained by accessing process control systems, etc.

[0028] Step 12: Construct the objective function and constraints for the target surface grinding process.

[0029] The above-mentioned objective function is used to describe the total time required to complete the surface polishing process.

[0030] Specifically, the objective function to be optimized is: , The total time is the sum of the grinding time of all grinding processes (i.e., the time from the start of the grinding process to the end of the grinding process) and the forced waiting time (i.e. the waiting time to prevent the material from overheating and the robotic arm from moving).

[0031] The constraints are: ; ; ; ; ; ; in, Indicates defects The start time of the second polishing process, Indicates defects The start time of the first polishing process. Indicates defects The polishing time for the first polishing step. Indicates defects The forced waiting time for the second polishing process, Represents the set of defect indexes. Represents a set of defect type indexes. This indicates a set of polishing processes. Represents the set of robotic arm indices. Indicates the polishing process Is it by robotic arm? implement, , This indicates the polishing process. By robotic arm implement, This indicates the polishing process. Not by robotic arm implement, Indicates robotic arm Perform the polishing process Should the polishing process be performed next? , Indicates robotic arm Perform the polishing process Then the polishing process is carried out. , Indicates robotic arm Perform the polishing process The polishing process is not performed afterward. , Indicates robotic arm Perform the polishing process Should the polishing process be performed next? , Indicates robotic arm Perform the polishing process Then the polishing process is carried out. , Indicates robotic arm Perform the polishing process The polishing process is not performed afterward. , Indicates the polishing process The start time, Indicates the polishing process The start time, For defects The The polishing time in the polishing process. Indicates defects With defects The distance between them Indicates the speed at which the robotic arm moves. This indicates the total time required to complete the surface polishing process. Represents the constraint constant.

[0032] For example, actual polishing and waiting time versus defect radius related: ; in, This indicates the basic polishing time for the first polishing step. This indicates a mandatory waiting time. This indicates the basic polishing time for the second polishing process.

[0033] It should be noted that the above constraints, in sequence, include: ensuring that the second round of polishing can only begin after the first round has ended and the mandatory waiting time has been met; each process must be executed by one and only one robotic arm; for each robotic arm, the processes executed should form a chain sequence; and in the constraints on time connection and path relationship, a large number C is used to ensure that the constraints only apply to... (i.e., robotic arm) Complete the process The process will then proceed immediately. This will take effect when the second round of sanding for each defect is completed (i.e., the start time of the second round of sanding). Plus polishing time The time to completion must not exceed the maximum completion time. .

[0034] when When the constraint is simplified to: ; Indicate process The start time must be later than the process time. The completion time plus the movement time.

[0035] when At that time, due to It is a sufficiently large positive number (usually much larger than all possible positive numbers). and (difference), the constraint becomes: ; This condition is met naturally, which is equivalent to the constraint being "closed".

[0036] It should be noted that the distance between the above-mentioned defects is: ; ; in, Indicates defects The x-coordinate (i.e., the coordinate value along the axis of the titanium rod). Indicates defects x-coordinate Indicates defects The ordinate, Indicates defects The ordinate, Indicates the cross-sectional radius of the titanium alloy bar. Indicates defects The position of the circumferential angle (i.e., the angle around the circumference of the titanium rod).

[0037] Grinding time calibration: based on the material removal rate formula ,in For grinding depth, For feed rate, This refers to the width of the grinding head. Based on existing research and grinding tests on TC4 titanium alloy, the material removal rate can reach 5000 mm³ / min, combined with the defect area. The grinding time per unit area can be calculated by converting the depth.

[0038] Calibration of forced waiting time: To prevent overheating during grinding from degrading the performance of titanium alloy materials, a cooling waiting period must be inserted between two grinding cycles. According to the theory of heat conduction, the time required for the surface temperature to drop to a safe threshold (e.g., 60°C) can be determined through finite element simulation.

[0039] Step 13: Based on the basic data and defect-related data of all robotic arms, define the state space of the target surface polishing process, and use reinforcement learning to solve for the initial process execution sequence of each robotic arm based on the state space.

[0040] The aforementioned initial process execution sequence includes at least one grinding process executed by the robotic arm in chronological order.

[0041] Specifically, the state space is as follows: ; in, Indicates defects The The completion progress of the polishing process. Indicates the remaining amount of forced waiting time. Indicates the remaining time for the currently executing process. Indicates the number of incomplete processes. For the global time step, Indicates the position information of the robotic arm: ; ; ; ; ; ; in, Indicates defects The The start time of the polishing process. Indicates defects The The polishing time in the polishing process. Indicates defects The first polishing process has been completed. Indicates defects The second polishing process has not yet begun.

[0042] For example, after defining the state space, a multi-agent reinforcement learning (MARL) algorithm can be used to solve for the sequence of actions to be performed by each robotic arm based on the state space. Specifically: Define the action space: The action space defines the actions that agent k (i.e., the robotic arm) can take at each time step, and includes two actions: 1. Select the next process to be executed. This process must meet the constraints corresponding to its type, and if it is the second round of polishing, then its corresponding forced waiting time has ended; 2. Wait in place: If there are no processes that meet the conditions to execute at present.

[0043] Actions can be represented as ,in It includes an optional set of operations (operations that satisfy type and waiting constraints) and in-place waiting actions.

[0044] Design a reward function: The reward function guides the agent's behavior towards the optimization objective (such as minimizing the maximum completion time Makespan), coordinating the reduction of Makespan with idle movement time (i.e., idle and ineffective moves). The reward design includes: (1) Positive reward: If the action taken reduces the overall estimated completion time If there is a decline, a positive reward will be given.

[0045] (2) Punishment mechanism: If the robotic arm is idle for too long or moves ineffectively, an appropriate punishment will be given.

[0046] (3) Cumulative rewards: Rewards can be accumulated over multiple time steps to guide the agent to optimize the global Makespan.

[0047] The instant reward function can be defined as follows: ; in Z represents the weight parameters in the reward function, which can be fine-tuned experimentally; Z is the estimated total completion time after the action is performed, and k is the robotic arm's... This refers to the idle time of robotic arm k in the current step. Let $k$ be the movement cost of the robotic arm $k$ in the current step.

[0048] The training process for MARL is as follows: (1) Algorithm selection: Reinforcement learning algorithms suitable for collaborative multi-agent tasks are selected, with typical examples including: Monotonic Value Function Factorization (QMIX): This method combines the local Q-values ​​of each agent into a global Q-value through a monotonic hybrid network, ensuring policy consistency and making it suitable for highly collaborative scenarios.

[0049] Multi-Agent Deep Deterministic Policy Gradient (MADDPG): This framework employs centralized training and distributed execution, where each agent has its own independent policy network and uses policy information from other agents during training.

[0050] (2) Interaction and Environment Simulation: Construct a simulation environment to simulate the physical and technological processes of surface polishing of a titanium rod: Each robotic arm functions as an independent intelligent agent; The environment updates its state based on the current state and the agent's actions (such as process completion, position movement, waiting timer, etc.). The environment returns a reward signal and the next state.

[0051] Interaction flow (per round): Initialize the environment (defects, robotic arm status); Each agent selects an action based on its current state; Perform actions and update the environment; Collect experience data In the environment, first observe the current state. The agent selects actions The environment performs actions and calculates rewards. The environment shifts to a new state , store experience .

[0052] Repeat until all steps are completed, and record the end time of each round. .

[0053] (3) Experience playback and network updates: Use an experience replay buffer to store historical interaction data; During each update, a batch of experiences is randomly sampled, and the target Q value is calculated: .in As a discount factor, The global Q-value predicted for the target network.

[0054] Network parameters are updated by minimizing the temporal difference error. θ : ,in Main network parameters, For learning rate, For gradient operators, This is the current network prediction value.

[0055] Training iterations continue until the policy stabilizes or a preset stopping condition is met.

[0056] (4) Conditions for stopping training: Reach the preset maximum number of training rounds; The strategy performance converges (e.g., the average completion time fluctuation in the most recent N rounds is less than the threshold). The quality of solutions on the validation set no longer improves significantly.

[0057] After training, for a given problem instance (i.e., a specific set of defect distributions and parameters), the trained policy network is run to generate a complete initial solution.

[0058] Reasoning process: Input the initial state of the problem instance into the policy network; Each agent selects an action based on the action probability distribution output by the network (or directly selects the action with the largest Q value). Execute step by step until all procedures are completed; Record the following throughout the process: which robotic arm performed which procedure, the sequence of procedures, the start time of each procedure, and the specific output of the initial solution.

[0059] Process allocation matrix Dimension (Number of processes) Number of robotic arms): Element definition: ; Physical significance: Clearly define the robotic arm responsible for each process.

[0060] Process sequence matrix Dimension: (Number of processes) Number of processes Number of robotic arms): Element definition: ; Physical meaning: Describes the sequence of operations performed on each robotic arm, forming a process chain.

[0061] Process start time Dimensions: (One time value for each process); Recording method: During the simulation execution process, when the process... Record its timestamp at the actual start time; Constraint satisfaction: Automatically satisfy constraints such as forced wait and move time; Initial maximum completion time This refers to the maximum time required to complete the second round of polishing for all defects.

[0062] Finally, the above process allocation matrix, process sequence matrix, and process start time are analyzed to extract the grinding process assigned to each robotic arm. These grinding processes are then arranged according to their start times to obtain the process execution sequence of the robotic arm.

[0063] Step 14: Map the initial process execution sequence of all robotic arms to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, iteratively destroy and repair the nodes of the graph structure to obtain the final graph structure.

[0064] In the graph structure described above, multiple nodes correspond one-to-one with multiple polishing processes. The edges between nodes represent the connection or spatial adjacency relationship between the corresponding polishing processes. The edges of the graph structure can be directed or undirected.

[0065] In some embodiments of this application, the steps of mapping the execution sequences of all robotic arms to obtain a graph structure, and iteratively destroying and repairing nodes in the graph structure based on the optimization objective function and constraints, as well as the characteristics of nodes in the graph structure, to obtain the final graph structure include: The first step is to map the process execution sequences of all robotic arms to obtain a graph structure.

[0066] Specifically, a node is generated for each polishing process; if two polishing processes belong to the same process execution sequence and are adjacent in the process execution sequence, then the two polishing processes are considered to have a connection relationship; if the distance between two defect points corresponding to two polishing processes is less than a distance threshold, then the two polishing processes are considered to have a spatial adjacency relationship; a directed edge is generated between the nodes corresponding to every two polishing processes with a connection relationship, and an undirected edge is generated between the nodes corresponding to every two polishing processes with a spatial adjacency relationship, resulting in a graph structure.

[0067] If two polishing processes do not belong to the same process execution sequence or are not adjacent in the process execution sequence, it is considered that there is no connection between the two polishing processes; if the distance between two defect points corresponding to two polishing processes is greater than or equal to the distance threshold, it is considered that there is no spatial adjacency between the two polishing processes.

[0068] For example, the attributes of the polishing process corresponding to a node are used as the node features, and the attributes such as the distance between the two polishing processes corresponding to an edge are used as the edge features, such as node features. for: ; in, Code the process type. First round of grinding. The code is 0, and the second round of polishing is performed. The code is 1. For defects The u The actual processing time required for wheel grinding. For defects The horizontal coordinate, For defects Unfold the vertical coordinates on the plane. , For defects The u The start time of wheel grinding in the initial solution. For defects radius, The robotic arm numbered k (by...) performs this process. Sure), The required mandatory waiting time is 0 for the first round of processes and 0 for the second round of processes. , To complete the status flag, in the initial solution, all nodes have this flag set to 0 (incomplete).

[0069] Edge features for: ; in, Edge type encoding. 1 represents a directed edge, 2 represents an undirected edge. Two defects associated with the edge and Surface distance between them : Only valid for sequential edges. Represents the time interval from the end of the source operation to the start of the target operation, calculated using the following formula: , : Binary symbol. If the processes at both ends of an edge are performed by the same robotic arm (according to...) If the condition is true, the value is 1; otherwise, it is 0. Binary identifier. If the processes at both ends of an edge belong to the same defect (i.e. and If the result is 1, then the result is 0; otherwise, it is 1. This indicates a special relationship between two processes with the same defect.

[0070] The second step is to use a graph neural network to extract features from the graph structure and obtain the node embedding vector for each node.

[0071] For example, the adjacency matrix, node feature matrix, and edge feature matrix of the above graph structure are input into a graph neural network for computation, outputting a node embedding vector for each node. Specifically, an L-layer graph neural network is used. Let the first layer be... layer( ) nodes The representation of is ,in These are the features of the input nodes.

[0072] Each layer of a Graph Neural Network (GNN) performs two core operations: message passing and feature aggregation. A widely used example is the Graph Convolutional Network (GCN), whose single-layer update formula can be specified as: (1) Message Generation: For each edge This generates a message that is sent from neighbor node u to target node v. In standard GCN, a message is the representation of the node one level above the neighbor node. If edge features are considered, the message can be enhanced as follows: ; in, It is a learnable function (such as a multilayer perceptron). It is the state of the sending node u. The status of the receiving node (optional, used to adjust messages). The edge connecting the two nodes Its characteristics.

[0073] Here the message function Typically implemented as a multi-layer perceptron (MLP), it concatenates or combines the above inputs in some way and maps them to a new message vector: ; in, This represents a vector concatenation operation. These are learnable parameters unique to this layer. In this way, the information transmitted by edges that are physically far apart and edges with large time intervals can be treated differently by the network.

[0074] (2) Message aggregation: Node v from all its neighboring nodes Messages are collected and aggregated into a single aggregated message vector (a single summary vector) using a permutation-invariant aggregation function (i.e., the aggregation result is the same regardless of the order of neighboring nodes, because the graph structure has no order) (such as summation, averaging, or taking the maximum value). .

[0075] ; in, This refers to the message vector sent from neighbor node u to target node v, as mentioned above. This refers to aggregate functions, commonly used functions include: Summation (Sum): ; Calculate the average (Mean): ; Get the maximum value (Max): ; Here, 'max' means taking the maximum value of each element.

[0076] Physical meaning: This step "condenses" the local neighborhood information of node v into a vector of fixed size. In the context of summation aggregation, for example, node v experiences the sum of the influence of all its neighbors.

[0077] (3) Node Update: node Based on its previous statement and aggregated neighbor messages Update its representation: ; in It is usually also a learnable function, such as in an MLP: ; Or, a more common simplified form is: ; in: and It is a learnable weight matrix. It is a non-linear activation function, such as ReLU, which introduces non-linear expressive power into the network. In this way, the new state of a node retains its historical information (through...). It also incorporates contextual information from its neighbors (through...). ).

[0078] (4) Simplified form of standard GCN layer (ignoring edge features) is usually written as: ; The three steps of messaging, aggregation, and updating were combined and normalized. Among them, It is a non-linear activation function (such as ReLU); It is the learnable weight matrix of this layer; These are the degrees (or normalization coefficients) of node v and node u, respectively. This coefficient is used to prevent nodes with a large number of neighbors (high-degree nodes) from dominating the entire network during the aggregation process. It stabilizes the training process.

[0079] (5) Handling of directed and undirected edges: This process diagram contains both directed temporal edges and undirected spatial edges. During message passing: For time-dependent edges, messages typically flow only from the predecessor node to the successor node, following the causal logic of process execution.

[0080] For spatially adjacent edges, messages flow bidirectionally between the two node pairs. The network weights can be shared or learned selectively for both edge types to capture the different effects of different types of relationships.

[0081] After message passing through layer L, the node representation of the final layer is obtained. As its node embedding vector .

[0082] The third step involves identifying multiple potential optimization nodes from all nodes based on their node embedding vectors, and then removing all potential optimization nodes from the graph structure to obtain a corrupted graph structure.

[0083] Specifically, based on the node embedding vector of each node, the node removal score is calculated; all removal scores are sorted from largest to smallest, and the nodes corresponding to the top preset number of removal scores in the sorting results are all regarded as potential optimization nodes.

[0084] Through the formula: ; Removal score of computing nodes .

[0085] in, The node embedding vector represents the node. , , These are all weight vectors, which can be trained using a small neural network or an online learning mechanism. The goal is to make removing and repairing high-scoring nodes more likely to reduce the total completion time. This represents the path redundancy factor for the grinding process corresponding to the node. This indicates the idle time ratio of the grinding process corresponding to the node: ; ; in, This indicates the path distance for the grinding process corresponding to the node. This represents the ideal direct distance after removing the node. This represents the sum of the idle time of the previous grinding process and the idle time of the next grinding process in the process execution sequence. This indicates the end time of the grinding process corresponding to the node.

[0086] For example, when removing a node from a graph structure, the edges of that node are also removed. For instance, suppose the original process chain of a robotic arm is: A → B → C, where process B is the node that needs to be removed; Before removal: Strict order A → B → C; Removal operation: Remove B from the sequence, and remove edges A→B and B→C; Partial solution state: The sequence becomes A and C, but there is no direct order constraint between A and C. A "vacancy" is formed between them, where B or other processes are to be inserted.

[0087] The fourth step is to take the position of each optimization potential node in the damaged graph structure as the insertion position, select the corresponding insertion node from all optimization potential nodes for each insertion position, and insert all insertion nodes into the damaged graph structure to obtain the repair graph structure.

[0088] Specifically, a cost function is defined, along with multiple weight parameter combinations for that cost function. Then, based on the selection scores of all weight parameter combinations, a target weight parameter combination is determined from all these combinations. Based on the target weight parameter combination and the cost function, the insertion cost value for each potential node when inserted into each desired insertion position is calculated. For each potential node, the node with the smallest insertion cost value is selected as the insertion node for that desired position. All inserted nodes are then inserted into the damaged graph structure to obtain the repaired graph structure.

[0089] It should be noted that the cost function is: ; in, Indicates the inserted cost value. , , Represents the weight parameters. Indicates the insertion of the grinding process. The subsequent time delay increment (due to the delay of the process, all subsequent processes of this process may be further postponed, and it needs to be propagated backward along the sequence of robot arm k to calculate the final completion time increment) This is the maximum increase in the completion time of the robotic arm caused by the insertion operation. Indicates the insertion of the grinding process. The path length is increased afterward. Indicates the polishing process Similarity to the embedding process of grinding before and after the insertion position: ; ; in, These are the surface distances from defect p to v, v to s, and p to s, respectively. This formula represents the path... Replace the original direct movement The increased distance, An aggregate representation indicating the embedding of preceding and following processes at the insertion position. , , These are the node embedding vectors for the preceding and following nodes at that position, respectively. If p or s is empty, it is replaced with a zero vector or a special "start / end" embedding.

[0090] The above combination of weight parameters constitutes the weight parameters in the cost function. , , The possible combinations of values, such as: , , All three weight parameters are preset positive parameters.

[0091] The initial value of the selection score for the weight parameter combination is a preset value. When determining the target weight parameter combination from all weight parameter combinations, the selection weight of the weight parameter combination can be calculated based on the selection score (e.g., the selection score can be divided by the number of times the weight parameter combination has been used). Then, all selection weights are smoothed to obtain the selection probability. The target weight parameter combination is then determined from all weight parameter combinations based on the selection probability.

[0092] It should be noted that the steps described above for calculating the insertion cost of each potential node when inserted into each position based on the target weight parameter combination and cost function are as follows: The target weight parameters are substituted into the cost function, and the insertion cost value of the optimized potential node when it is inserted into each insertion position is calculated using this formula. Finally, the optimized potential node is assigned to the insertion position with the smallest insertion cost value, and is used as the insertion node for that insertion position.

[0093] For example, when inserting a node into the desired position, a corresponding edge is generated, such as: The node to be inserted Insert into its optimal position .

[0094] set up and If the node It exists and exists in the original order. If so, then cancel that edge.

[0095] Recalculate the robotic arm based on the new sequence after insertion. upper node The start time of all subsequent processes starting from the corresponding polishing process (local time rearrangement).

[0096] The fifth step involves iteratively updating the graph structure based on the optimization objective function and constraints to obtain the final graph structure.

[0097] Specifically, the repair graph structure is constrained and validated according to the constraints. After the validation is passed, the total completion time corresponding to the repair graph structure is calculated by optimizing the objective function. The selection score of the target weight parameter combination corresponding to the repair graph structure is updated using the total completion time. The repair graph structure is iteratively updated based on the updated selection score to obtain the final graph structure.

[0098] For example, the system determines whether the repair graph structure meets the constraints. If it does, the verification passes; otherwise, the verification fails, indicating that the robot arm's process execution sequence reflected in the current graph structure is unreasonable. The process of repairing and destroying the robot arm needs to be adjusted (e.g., by selecting different weight parameter combinations) and then recalculated. After the verification passes, the robot arm's process execution sequence reflected in the graph structure is substituted into the optimization objective function to calculate the total completion time.

[0099] The total completion time corresponding to the current repair graph structure Total completion time of the diagram structure before damage and repair Optimal total completion time The comparison is used to update the selection score of the weight parameter combination, such as: If the result is obtained after repair using a combination of weight parameters Better than the current best known total completion time The combination of weighted parameters will then receive a reward score. (e.g., +10).

[0100] like Superior But not better The combination of weighted parameters will then receive a reward score. (e.g., +5).

[0101] like Inferior Then the combination of weight parameters will result in a penalty score. (e.g., -2).

[0102] The reward or penalty score obtained through the above method is added to the selection score of the weighted parameter combination to obtain the updated selection score. The initial value of the selection score is a preset value.

[0103] Every time Rounds (e.g., 10 rounds), weights for the combination of weight parameters. Perform normalization update: ; in For learning rate, For weight parameter combination j Updated selection score, For weight parameter combination j The number of times the value is used is selected to avoid a denominator of zero. Then, all weights are smoothed. ; To ensure the probability distribution of choices in roulette, where For weight parameter combination i The weight.

[0104] It should be noted that the specific steps for iteratively updating the repair graph structure based on the updated selection score to obtain the final graph structure are as follows: Determine whether the iteration termination condition is met.

[0105] If the conditions are not met, the repaired graph structure is used as the graph structure, the updated selection score is used as the selection score, and the process returns to the step of using a graph neural network to extract features from the graph structure and obtain the node embedding vector of each node.

[0106] If the conditions are met, the repaired graph structure will be used as the final graph structure.

[0107] For example, the above iteration termination conditions can be: the number of iterations reaches the maximum number of iterations (the initial value of the number of iterations is 0, and the number of iterations is incremented by one for each time the graph structure is repaired), the difference between the total completion time of the current repaired graph structure and the total completion time of the previous graph structure is less than a preset difference, the solution operation time reaches the budget, etc.

[0108] In some embodiments of this application, the repaired graph structure is used only when the total completion time of the repaired graph structure is less than the total completion time of the graph structure before repair; otherwise, the graph structure before repair is still used for iterative calculations.

[0109] Step 15: Determine the final process execution sequence of each robotic arm from the final diagram structure, and control the robotic arm according to the final process execution sequence of each robotic arm.

[0110] The aforementioned final process execution sequence includes at least one polishing process to be performed by a robotic arm, and the polishing processes are arranged in chronological order. For example, the robotic arm is to perform the first polishing process for defect point A and the second polishing process for defect point B, and the start time of the first polishing process for defect point A is earlier than the start time of the second polishing process for defect point B. Then the final process execution sequence of the robotic arm is: the first polishing process for defect point A → the second polishing process for defect point B.

[0111] For example, the nodes corresponding to each robotic arm are extracted from the final diagram structure, and the grinding processes corresponding to the extracted nodes are arranged in order of start time to obtain the final process execution sequence. Then, the final process execution sequence is input into the control system of the robotic arm to control the robotic arm to execute the final process execution sequence to grind the titanium alloy bar.

[0112] In some embodiments of this application, the final process execution sequence and start time are used as the basis. For each robotic arm Generate its unique sequence of operations: (1) Sequence extraction: For each robotic arm It can parse a complete chain from the starting process (without a predecessor) to the ending process (without a successor) from its final process execution sequence.

[0113] The sequence is represented as an ordered list: ,in Corresponding process , This is the total number of processes handled by the robotic arm.

[0114] (2) Time calculation: For each process (correspond ): Start Time : Obtained directly from the final process execution sequence, processing time End time Shift time (from preceding process p to v): (For the first operation in the sequence, the movement time is calculated from the initial position of the robotic arm or is 0), robotic arm completion time: That is, the end time of its last process.

[0115] (3) Calculation of global indicators: Maximum completion time (Makespan): Total processing time Total travel time Robotic arm utilization rate .

[0116] Then, a continuous movement path is generated for each robotic arm on the unfolded plane of the titanium rod, which is the direct input for motion control: (1) Coordinate transformation: For process Corresponding defects The coordinates of its processing point are: Horizontal: Vertical: .

[0117] (2) Path point sequence generation: For the process sequence of robotic arm k Its path point sequence is as follows: ; in These are the initial coordinates of the robotic arm k. The line segments between adjacent path points represent the straight paths (on the unfolded plane) that the robotic arm needs to move.

[0118] Path-based time addition: Associating each path point with an "arrival time" and "departure time" (i.e., processing start and end times) forms a spatiotemporal trajectory, and controlling the robotic arm to run according to this spatiotemporal trajectory.

[0119] It is worth mentioning that an optimization objective function for the surface grinding process is constructed to achieve a quantitative representation of the grinding time of complex processes. By mapping the process execution sequence of the robotic arm to a graph structure, the processes of the robotic arm under complex processes can be visualized. By destroying and repairing nodes in the graph structure, the correlation and connection rationality between the robotic arm processes can be effectively explored. With the grinding process time as the optimization objective, the allocation of grinding processes of the robotic arm is optimized to ensure that the control accuracy of the robotic arm under complex processes is effectively improved while reducing the process time.

[0120] The following is an exemplary description of the reinforcement learning-based robotic arm control optimization device provided in this application.

[0121] like Figure 2 As shown, this application embodiment provides a reinforcement learning-based robotic arm control optimization device 200, which includes: The acquisition module 201 is used to acquire basic data of multiple robotic arms in the target surface grinding process, as well as defect-related data of the target surface grinding process; the target surface grinding process includes multiple grinding processes to be ground, and the multiple grinding processes include a first grinding process and a second grinding process for multiple defect points on the surface to be ground, and the basic data includes the initial position and moving speed of the robotic arms. Module 202 is used to construct the optimization objective function and constraints for the target surface grinding process; the optimization objective function describes the total completion time of the target surface grinding process. The definition module 203 is used to define the state space of the target surface polishing process based on the basic data and defect-related data of all robotic arms, and to solve the initial process execution sequence of each robotic arm using reinforcement learning based on the state space; the initial process execution sequence includes at least one polishing process executed by the robotic arm in time sequence. The mapping module 204 is used to map the initial process execution sequence of all robotic arms to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, the module iteratively destroys and repairs the nodes in the graph structure to obtain the final graph structure. In the graph structure, multiple nodes correspond one-to-one with multiple grinding processes, and the edges between nodes represent the connection relationship or spatial adjacency relationship between the corresponding grinding processes. The control module 205 is used to determine the final process execution sequence of each robotic arm from the final diagram structure, and to control the robotic arm according to the final process execution sequence of each robotic arm.

[0122] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0124] like Figure 3 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 3 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0125] Specifically, when the processor D100 executes the computer program D102, it acquires the basic data of multiple robotic arms in the target surface polishing process, as well as the defect-related data of the target surface polishing process. Then, it constructs the optimization objective function and constraints of the target surface polishing process. Based on the basic data and defect-related data of all robotic arms, it defines the state space of the target surface polishing process. Based on the state space, it uses reinforcement learning to solve for the process execution sequence of each robotic arm. Then, it maps the process execution sequences of all robotic arms to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, it iteratively destroys and repairs the nodes of the graph structure to obtain the final graph structure. Finally, it determines the final process execution sequence of each robotic arm from the final graph structure and controls the robotic arm according to the final process execution sequence of each robotic arm. Among them, an optimization objective function for the surface grinding process is constructed to achieve a quantitative representation of the grinding time of complex processes. By mapping the process execution sequence of the robotic arm to a graph structure, the process of the robotic arm under complex processes can be visualized. By destroying and repairing nodes in the graph structure, the correlation and connection rationality between the robotic arm processes can be effectively explored. With the grinding process time as the optimization objective, the allocation of grinding processes of the robotic arm is optimized to ensure that the control accuracy of the robotic arm under complex processes is effectively improved while reducing the process time.

[0126] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0127] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0128] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0129] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the reinforcement learning-based robotic arm control optimization method device / terminal device, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A method for optimizing the control of a robotic arm based on reinforcement learning, characterized in that, include: Acquire basic data of multiple robotic arms in the target surface polishing process, as well as defect-related data of the target surface polishing process; the target surface polishing process includes multiple polishing processes of the surface to be polished, the multiple polishing processes include a first polishing process and a second polishing process of multiple defect points on the surface to be polished, and the basic data includes the initial position and moving speed of the robotic arms. Construct the optimization objective function and constraints for the target surface polishing process; the optimization objective function is used to describe the total completion time of the target surface polishing process; Based on the basic data of all robotic arms and the defect-related data, the state space of the target surface polishing process is defined, and based on the state space, the initial process execution sequence of each robotic arm is obtained by using reinforcement learning; the initial process execution sequence includes at least one polishing process executed by the robotic arm in chronological order. The initial process execution sequence of all robotic arms is mapped to obtain a graph structure. Based on the optimization objective function and constraints, as well as the characteristics of the nodes in the graph structure, the graph structure is iteratively broken and repaired to obtain the final graph structure. In the graph structure, multiple nodes correspond one-to-one with multiple grinding processes, and the edges between nodes represent the connection relationship or spatial adjacency relationship between the corresponding grinding processes. The final process execution sequence of each robotic arm is determined from the final diagram structure, and the robotic arm is controlled according to the final process execution sequence of each robotic arm.

2. The robotic arm control optimization method according to claim 1, characterized in that, The constraints are as follows: ； ； ； ； ； ； in, Indicates defects The start time of the second polishing process, Indicates defects The start time of the first polishing process. Indicates defects The polishing time for the first polishing step. Indicates defects The forced waiting time for the second polishing process, Represents the set of defect indexes. Represents a set of defect type indexes. This indicates a set of polishing processes. Represents the set of robotic arm indices. Indicates the polishing process Is it by robotic arm? implement, , This indicates the polishing process. By robotic arm implement, This indicates the polishing process. Not by robotic arm implement, Indicates robotic arm Perform the polishing process Should the polishing process be performed next? , Indicates robotic arm Perform the polishing process Then the polishing process is carried out. , Indicates robotic arm Perform the polishing process The polishing process is not performed afterward. , Indicates robotic arm Perform the polishing process Should the polishing process be performed next? , Indicates robotic arm Perform the polishing process Then the polishing process is carried out. , Indicates robotic arm Perform the polishing process The polishing process is not performed afterward. , Indicates the polishing process The start time, Indicates the polishing process The start time, For defects The The polishing time in the polishing process. Indicates defects With defects The distance between them Indicates the speed at which the robotic arm moves. This indicates the total time required to complete the surface polishing process. Represents the constraint constant.

3. The robotic arm control optimization method according to claim 2, characterized in that, The edges of the graph structure can be directed or undirected. The mapping of the process execution sequences of all robotic arms to obtain a graph structure includes: Generate a node for each polishing process; If two polishing processes belong to the same process execution sequence and are adjacent in the process execution sequence, then it is considered that there is a connection between the two polishing processes; If the distance between two defect points corresponding to two polishing processes is less than the distance threshold, then the two polishing processes are considered to have a spatial adjacency relationship. A directed edge is generated between every two nodes corresponding to grinding processes that are connected, and an undirected edge is generated between every two nodes corresponding to grinding processes that are spatially adjacent, thus obtaining a graph structure.

4. The robotic arm control optimization method according to claim 3, characterized in that, The process of iteratively destroying and repairing nodes in the graph structure based on the optimization objective function, constraints, and characteristics of nodes in the graph structure to obtain the final graph structure includes: The graph structure is used to extract features to obtain the node embedding vector of each node; Based on the node embedding vectors of all nodes, multiple potential optimization nodes are identified from all nodes, and all potential optimization nodes are removed from the graph structure to obtain a destructive graph structure. The position of each optimization potential node in the destruction graph structure is taken as the insertion position. For each insertion position, the corresponding insertion node is selected from all optimization potential nodes. All insertion nodes are inserted into the destruction graph structure to obtain the repair graph structure. Based on the optimization objective function and constraints, the repair graph structure is iteratively updated to obtain the final graph structure.

5. The robotic arm control optimization method according to claim 4, characterized in that, The node embedding vectors based on all nodes identify multiple nodes with optimization potential from all nodes, including: Calculate the node removal score based on the node embedding vector of each node; Sort all removal scores from largest to smallest, and select the nodes corresponding to the top preset number of removal scores in the sorting results as nodes with optimization potential.

6. The robotic arm control optimization method according to claim 5, characterized in that, The step of calculating the node removal score based on the node embedding vector of each node includes: Through the formula: ； Removal score of computing nodes ; in, The node embedding vector represents the node. , , Both are weight vectors. This represents the path redundancy factor for the grinding process corresponding to the node. This indicates the idle time ratio of the grinding process corresponding to the node: ； ； in, This indicates the path distance for the grinding process corresponding to the node. This represents the ideal direct distance after removing the node. This represents the sum of the idle time of the previous grinding process and the idle time of the next grinding process in the process execution sequence corresponding to the node. This indicates the end time of the polishing process corresponding to the node.

7. The robotic arm control optimization method according to claim 4, characterized in that, The step of selecting the corresponding insertion node from all potential optimization nodes for each insertion position includes: Define a cost function and define multiple weight parameter combinations for the cost function; Based on the selection score of all weight parameter combinations, the target weight parameter combination is determined from all weight parameter combinations, and based on the target weight parameter combination and the cost function, the insertion cost value of each optimization potential node when inserted into each insertion position is calculated. For each of the optimization potential nodes, the optimization potential node is used as the insertion node corresponding to the insertion position with the smallest insertion cost value of the optimization potential node.

8. The robotic arm control optimization method according to claim 7, characterized in that, Based on the optimization objective function and constraints, the repair graph structure is iteratively updated to obtain the final graph structure, including: The repair graph structure is constrained and verified according to the constraints. After the verification is passed, the total completion time corresponding to the repair graph structure is calculated through the optimization objective function. The selection score of the target weight parameter combination corresponding to the repair graph structure is updated using the total completion time. The repair graph structure is iteratively updated based on the updated selection score to obtain the final graph structure.

9. The robotic arm control optimization method according to claim 7, characterized in that, The cost function is: ； in, Indicates the inserted cost value. , , Represents the weight parameters. Indicates the insertion of the grinding process. The subsequent time delay increment, Indicates the insertion of the grinding process. The path length is increased afterward. Indicates the polishing process Similarity to the embedding process of grinding before and after the insertion position.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the reinforcement learning-based robotic arm control optimization method as described in any one of claims 1 to 9.

Images