Multi-bolt assembly control method and device based on knowledge-enhanced hierarchical reinforcement learning

Abstract

This invention discloses a multi-bolt assembly control method and apparatus based on knowledge-enhanced hierarchical reinforcement learning, comprising: acquiring current state data during the multi-bolt assembly process of a robot; inputting the current state data into a hierarchical policy model, and outputting a composite action containing discrete selection actions and continuous control actions to control the robot to perform assembly operations; generating a first prior action matching the discrete selection actions and a second prior action matching the continuous control actions based on the current state data during the training phase; and adaptively switching between a knowledge-dominated mode and an exploration-dominated mode based on the loss statistics during the training process to update the parameters of the hierarchical policy model. This invention effectively compresses the exploration space in the early stage of training and adaptively releases constraints in the later stage, improving assembly efficiency and robustness.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control and industrial automation assembly technology, specifically to a multi-bolt assembly control method and device based on knowledge-enhanced hierarchical reinforcement learning. Background Technology

[0002] In the robotic multi-bolt flange assembly task, due to the elastic interaction between the flange and multiple bolts, tightening one bolt changes the stress state of the already tightened bolts, causing a redistribution of preload. This process is essentially a long-term complex action decision problem involving two types of decision variables: discrete bolt sequence selection and continuous contact force adjustment, exhibiting sparse reward feedback. Existing reinforcement learning methods often face bottlenecks in handling such problems, such as a large blind exploration space and difficulty in convergence during the early training phase. While introducing fixed expert rules can accelerate early training, overly strong rule constraints can compress the agent's space for exploring better strategies in the later stages of training, leading to convergence to a suboptimal solution. Therefore, there is an urgent need for a control method that can both utilize prior knowledge to improve early training efficiency and adaptively release constraints to ensure the ability to optimize strategies in the later stages. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-bolt assembly control method and device based on knowledge-enhanced hierarchical reinforcement learning. This invention effectively compresses the exploration space in the early stages of training and adaptively releases constraints in the later stages, improving assembly efficiency and robustness.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning, comprising: Step 1: Obtain the current status data during the robot's multi-bolt assembly process; Step 2: Input the current state data into the hierarchical strategy model and output composite actions to control the robot to perform assembly operations; the composite actions include discrete selection actions and continuous control actions, and the hierarchical strategy model includes a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. Step 3: During the training phase of the hierarchical strategy model, a first prior action matching the discrete selection action and a second prior action matching the continuous control action are generated based on the current state data. The first prior action is generated by constructing a pairing mapping function based on the principles of diagonal alternating tightening and planetary rotation tightening. Combining the bolt states selected in the current step and the previous step, selected states are excluded from the candidate bolt set, and an action is randomly selected as the first prior action. The second prior action is generated by using triangular and trapezoidal membership functions to perform fuzzy modeling of the axial contact force and axial tightening torque, respectively, to obtain the membership degree of the corresponding evaluation level. The membership degree is truncated to a preset interval to construct a basic probability assignment function, and multi-channel evidence is normalized and fused using evidence combination rules. The reference force value corresponding to the evaluation level with the largest fused basic probability assignment value is selected as the second prior action. Step 4: Based on the loss statistics during training, adaptively switch between knowledge-driven mode and exploration-driven mode to update the parameters of the hierarchical policy model; wherein, the adaptive switching includes: Calculate the first loss mean within the historical window, the second loss mean within the current window, and the loss standard deviation within the current window; When the ratio of the second loss mean to the first loss mean is less than a first preset threshold, and the loss standard deviation is less than a second preset threshold, the training mode is switched from knowledge-driven mode to exploration-driven mode.

[0005] The aforementioned multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning, under the knowledge-dominated mode, uses a supervised learning loss function to update the first policy model and a mean squared error loss function to update the second policy model. In the exploration-dominated mode, the first policy model is updated using the reinforcement learning temporal difference loss function, and the second policy model is updated using a loss function that integrates reinforcement learning objectives and knowledge-guided soft constraints. The knowledge-guided soft constraint is determined by multiplying the difference between the continuous control action output by the second strategy model and the second prior action by the knowledge constraint weight coefficient.

[0006] The aforementioned multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning includes real-time tightening torque, real-time contact force, and discrete state codes characterizing the current tightening stage and assembly progress of each bolt position. The discrete selection action corresponds to the discrete selection of the target fastening bolt, and the continuous control action corresponds to the continuous desired force value output by the impedance controller.

[0007] The aforementioned multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning employs an actor-critic structure in its second strategy model, wherein: In the actor-critic structure, the actor network takes the concatenated vector of discrete choice actions output by the first strategy model and the current state data as input, and outputs a continuous expectation force value. The critic network in the actor-critic structure adopts a double-Q network structure, taking the current state data and compound actions as inputs, and outputting a state-action value estimate, which is used to update the policy gradient of the actor network.

[0008] The aforementioned multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning, during the training phase, uses a soft update method to synchronize the target network parameters corresponding to the first policy model, the actor network, and the critic network. The soft update method uses exponential linear interpolation to make the target network parameters approximate the current network parameters.

[0009] A multi-bolt assembly control device based on knowledge-enhanced hierarchical reinforcement learning, the device comprising: The status acquisition module is used to acquire the current status data of the robot during the multi-bolt assembly process; The action output module is used to input the current state data into the hierarchical strategy model and output a composite action to control the robot to perform assembly operations; the composite action includes discrete selection actions and continuous control actions, and the hierarchical strategy model includes a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. The prior generation module is used to generate a first prior action that matches the discrete selection action and a second prior action that matches the continuous control action based on the current state data during the training phase of the hierarchical policy model. An adaptive training module is used to adaptively switch between a knowledge-driven mode and an exploration-driven mode based on the loss statistics during the training process in order to update the parameters of the hierarchical strategy model.

[0010] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the aforementioned multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning.

[0011] Compared with existing technologies, this invention decouples the long-term complex action space into two levels: discrete selection and continuous control. During the training phase, prior knowledge matching the decision meaning is injected into each level, effectively compressing the blind exploration space in the early stages of training and improving the quality of the starting point for policy learning. Simultaneously, by monitoring loss statistics during training, it adaptively switches between knowledge-driven and exploration-driven modes. This allows the policy to fully rely on prior knowledge for rapid convergence in the early stages of training, while automatically releasing rule constraints in the later stages. This avoids the limitations of fixed prior knowledge hindering the policy's evolution towards the global optimum, thus improving assembly success rate and environmental robustness while ensuring training efficiency. Attached Figure Description

[0012] Figure 1 is a block diagram of the overall architecture of the multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning in an embodiment of the present invention; Figure 2 is a schematic diagram of the triangular membership function used for axial contact force in an embodiment of the present invention; Figure 3 is a schematic diagram of the trapezoidal membership function used for the axial fastening torque in an embodiment of the present invention; Figure 4 is a schematic diagram of the physical experimental platform for multi-bolt assembly of robots according to an embodiment of the present invention.

[0013] Among them, 1-UR5 robotic arm, 2-six-dimensional force / torque sensor, 3-nut, 4-flange, 5-bolt to be tightened. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0016] Example 1: This example provides a multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning. This method solves the exploration difficulties and suboptimal convergence problems in long-term complex action decision-making through hierarchical decoupling and adaptive knowledge guidance mechanisms. Specifically, the method includes the following steps.

[0017] Step 1: Acquire current state data during the robot's multi-bolt assembly process. In this step, the current state data serves as the basis for the agent to perceive the assembly environment and its own state. It typically includes real-time tightening torque and contact force at each bolt position, as well as discrete state codes representing the current tightening stage and assembly progress. For example, in a flange assembly scenario, an axial contact force and tightening torque are collected using a six-dimensional force / torque sensor at the end effector, and combined with encoder information to generate discrete state codes, thereby constructing a state vector that reflects the physical process of assembly. This acquisition of multimodal state data provides the necessary information foundation for subsequent hierarchical decision-making, enabling the strategy model to simultaneously perceive continuous mechanical interaction states and discrete assembly progress.

[0018] Step 2 involves inputting the current state data into the hierarchical strategy model, which outputs composite actions to control the robot in performing assembly operations. These composite actions include discrete selection actions and continuous control actions. The hierarchical strategy model comprises a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. The core purpose of this hierarchical architecture is to address the curse of dimensionality caused by long-term temporal coupling in multi-bolt assembly tasks. Figure 1 As shown, the first strategy model is responsible for high-level decision-making, i.e., selecting the next target bolt to be tightened from the candidate bolt set, corresponding to discrete selection actions; the second strategy model is responsible for low-level control, i.e., outputting the desired force value of the impedance controller based on the currently selected bolt and the environmental state, corresponding to continuous control actions. The composite action space can be formally defined as: ; In the formula: For composite action space, Select sub-action space for discrete bolts (corresponding to discrete selection of target bolts in a four-bolt structure). This is the continuous force control sub-action space (corresponding to the continuous desired force value output by the impedance force controller).

[0019] At every moment The intelligent agent executes compound actions ,in Generated by the first strategy model, Based on the second strategy model Compared with the current state It was produced jointly. It should be understood that, although... Figure 1While a specific hierarchical partitioning method is illustrated, in other embodiments, any implementation that satisfies the function of decoupling the composite action space into two subspaces—discrete and continuous—and making decisions separately, can be considered an equivalent implementation of the present invention. Through this hierarchical decoupling, the originally large joint action space is decomposed into two relatively independent sub-problems, significantly reducing the difficulty of policy learning and enabling the robot to achieve precise sequence selection and force control adjustment in complex, elastically coupled environments.

[0020] Step 3: During the training phase of the hierarchical strategy model, a first prior action matching the discrete choice action and a second prior action matching the continuous control action are generated based on the current state data. In this step, because discrete decision-making and continuous control have drastically different physical characteristics and expert experience forms, two different types of prior actions need to be generated for targeted guidance. The first prior action is typically generated based on sequence rules in the assembly process (such as diagonal or planetary fastening principles) to constrain the search space of discrete choices; the second prior action is typically generated based on fuzzy reasoning of the mechanical state or multi-source evidence fusion to provide a reference benchmark for continuous force control. By distinguishing between the two types of prior actions, this invention ensures a high degree of matching between the injected knowledge and the decision characteristics of the corresponding level, avoiding the limitation that a single form of knowledge cannot simultaneously adapt to discrete and continuous decision-making, thereby improving the effectiveness of knowledge guidance.

[0021] Step 4: Based on the loss statistics during training, the system adaptively switches between knowledge-driven and exploration-driven modes to update the parameters of the hierarchical policy model. In the early stages of training, the policy model is in knowledge-driven mode, primarily relying on prior actions for supervised learning or strong constraint optimization to quickly obtain an initial policy with basic assembly capabilities. As training progresses, when the loss statistics indicate that the policy has fully absorbed prior knowledge and tends to stabilize, the system automatically switches to exploration-driven mode. At this point, knowledge constraints are weakened or transformed, and reinforcement learning objectives are used to drive further optimization. This adaptive switching mechanism based on loss statistics essentially dynamically balances the relationship between "utilizing known knowledge" and "exploring unknown optima." Compared to manually preset fixed switching step sizes, this embodiment can automatically determine the switching timing based on the actual training state, avoiding both premature constraint release leading to policy collapse and delayed constraint release causing the policy to fall into local optima, thereby maximizing final assembly performance while ensuring training efficiency. It should be understood that the specific calculation method of the loss statistics and the setting of the switching threshold can be adjusted according to the actual task requirements. As long as the function of adaptively adjusting the knowledge constraint strength based on training feedback is satisfied, it falls within the protection scope of this invention.

[0022] Furthermore, steps 1 to 4 above are executed in an alternating iterative manner during the training phase: the first strategy model and the second strategy model are trained alternately. That is, in each training round, the parameters of the first strategy model are first updated based on the current state data, and then the parameters of the second strategy model are updated based on the output of the first strategy model and the current state data. This alternating iteration continues until the training termination condition is met. The training termination condition includes: the contact force and tightening torque of each bolt reach the target preload range and meet the preload distribution uniformity constraint, or the preset maximum number of training steps is reached. After training is completed, the converged first strategy model and second strategy model are jointly deployed to the robot control system, thereby realizing autonomous motion and force control planning for the multi-bolt flange assembly task.

[0023] Through steps 1 to 4 above, this embodiment constructs a complete knowledge-enhanced hierarchical reinforcement learning control framework. This framework reduces the difficulty of long-term decision-making through hierarchical decoupling, improves the accuracy of knowledge injection through differentiated prior generation, and ensures the full-cycle optimization capability of the policy through adaptive mode switching, laying a solid structural foundation for the implementation of subsequent specific algorithms.

[0024] As a preferred embodiment, this embodiment further defines the specific generation mechanism of differentiated prior knowledge. Specifically, it generates a first prior action matching discrete selection actions and a second prior action matching continuous control actions based on current state data, including: generating the first prior action based on historical action states and preset sequence mapping rules; and generating the second prior action based on the fuzzy membership degree of multi-source physical state quantities and multi-source evidence fusion rules. This differentiated generation concept stems from the fact that discrete decision-making and continuous control differ significantly in their physical nature; the former focuses on temporal logic and geometric constraints, while the latter focuses on mechanical equilibrium and uncertainty handling. Therefore, different knowledge representation forms are needed for targeted guidance.

[0025] For generating the first prior action, a pairing mapping function is constructed based on the principles of diagonal alternating tightening and planetary rotation tightening. Combining the bolt states selected in the current step and the previous step, the selected states are excluded from the candidate bolt set, and an action is randomly selected as the first prior action. Specifically, in multi-bolt flange assemblies, to prevent warping deformation of the flange due to uneven stress, a diagonal or planetary tightening sequence is typically followed in engineering. This embodiment encodes this expert experience by constructing a pairing mapping function. For example, for a four-bolt flange structure, the mapping function P maps bolt numbers {0,1,2,3} to {1,0,3,2}, meaning bolts 0 and 1 are diagonally / planetarily paired, and bolts 2 and 3 are paired. When generating the prior action, the system first obtains the bolt states selected in the current step and the bolt states selected in the previous step, and then strictly excludes these two states from the candidate bolt set. The physical significance of this exclusion mechanism is to avoid continuously tightening the same bolt or adjacent bolts, thereby reducing the local off-center loading effect and improving the uniformity of preload distribution. After eliminating invalid options, the system randomly selects an action from the remaining valid candidate set as the first prior action. The process of generating the first prior action can be formally expressed as: ; In the formula: The rule actions (i.e., target fastening bolt numbers) generated for the high-level rule controller. This is a diagonal-planet pairing mapping function (implementing the diagonal or planetary pairing conversion of bolt numbers). This represents an operation that randomly selects an action from the candidate set. The bolt status selected in the current step. For the bolt type selected in the previous step, This indicates a set exclusion operation, used to avoid repeatedly tightening the same bolt to reduce local off-center load and improve the uniformity of preload distribution.

[0026] For the generation of the second prior action, fuzzy modeling of axial contact force and axial fastening torque is performed using triangular and trapezoidal membership functions, respectively, to obtain the membership degree of the corresponding evaluation level. Specifically, due to the continuity and uncertainty of the mechanical state during assembly, directly using precise numerical values ​​for rule matching often fails to adapt to noise interference. Therefore, this embodiment introduces fuzzy logic to semantically process physical quantities. For example... Figure 2 As shown, for the axial contact force Fz, a triangular membership function is used to divide it into three evaluation levels: excellent, good, and poor. The vertex parameter b of the triangle represents the optimal or central value for that level, while the base parameters a and c represent the effective coverage area of ​​that level. Figure 3As shown, for the axial tightening torque Tz, considering that it should remain constant within a specific range, a trapezoidal membership function is used for modeling. The plateau region [b,c] represents the ideal torque range, and the ramp regions [a,b] and [c,d] represent the transition regions. Through this fuzzy modeling, continuous mechanical signals are transformed into membership vectors with semantic information, laying the foundation for subsequent evidence fusion. The formal definitions of the triangular membership function and the trapezoidal membership function are as follows: ; In the formula: For input state variables (such as axial contact force or fastening torque). The threshold for the membership segmentation corresponding to the input state is denoted as , where The function is in the interval Linearly increasing within the interval The membership degree decreases linearly within the inner domain, and is 0 in all other cases.

[0027] ; In the formula: This is the input state quantity (i.e., axial clamping torque). , , , The threshold for the membership degree segmentation corresponding to the input state.

[0028] Furthermore, the membership degrees are truncated to a preset interval to construct a basic probability allocation function, and the multi-channel evidence is normalized and fused using evidence combination rules. Specifically, to ensure the stability of numerical calculations and avoid inference failures in extreme cases, this embodiment truncates the membership values ​​(i.e., evidence quality) of all channels to a preset interval, such as [0.025, 0.95]. Setting a lower bound of 0.025 is to prevent a certain mode from being assigned zero evidence due to sensor noise or transient anomalies, causing the information of that channel to be completely lost during the fusion process; setting an upper bound of 0.95 is to avoid suppressing the contribution of another channel due to the high certainty of a single channel, ensuring that multi-source information can complement each other. After constructing the basic probability allocation function, the Dempster-Shafer evidence combination rule is used to normalize and fuse the evidence of the contact force and torque channels. Compared with simple weighted average or product rules, DS evidence theory can effectively handle conflicts and uncertainties between evidence. When the readings of two sensors are inconsistent, the confidence level can be reasonably allocated through the normalization coefficient K, rather than simply canceling each other out or blindly accepting one side. The formal definitions of the evidence combination rule and the normalization coefficient are as follows: ; In the formula: These are the base probability assignment values ​​after fusion. The basic probability assignment function for each channel (such as contact force and clamping torque). Indicates the rules for combining evidence, The normalization coefficient is... To identify frame power sets subset of for and The intersection of.

[0029] ; In the formula, and These are the basic probability assignment values ​​for the corresponding propositions.

[0030] Finally, the reference force value corresponding to the evaluation level with the largest basic probability allocation value after fusion is selected as the second prior action. Specifically, after completing the fusion of multi-source evidence, the system obtains a comprehensive basic probability allocation vector, which represents the comprehensive confidence level of the current assembly state belonging to each evaluation level. The system determines the most credible evaluation level (such as "Excellent", "Good", or "Poor") by taking the maximum probability index operation, and outputs the corresponding reference force value according to the preset mapping relationship. Specifically, the mapping relationship between the evaluation level and the reference force value can be represented as the reference force value vector Vr=[1.2α, 1.0α, 0.5α], where α is the preload reference coefficient, and the three components correspond to the three evaluation levels of Excellent, Good, and Poor, respectively. For example, if the fusion result determines the current state as "Excellent", a higher preload reference coefficient (such as 1.2α) is output; if it is determined as "Good", a standard coefficient (such as 1.0α) is output; and if it is determined as "Poor", a lower correction coefficient (such as 0.5α) is output. This reference force value is the second prior action, which provides a dynamic target based on physical rules and real-time state for the low-level continuous policy, ensuring that the reinforcement learning agent always has a "safe anchor point" that conforms to the principles of mechanics during the exploration process. The process of determining the above-mentioned second prior action can be formally expressed as: ; In the formula: The rule actions (i.e., continuous expectation force reference values) output by the lower-level rule controller. The output expected rule reference force value vector corresponding to the evaluation level. This is the comprehensive basic probability allocation value obtained after normalizing and fusing multi-channel evidence using the Dempster-Shafer evidence theory combination rule. This indicates the operation of taking the index with the highest probability.

[0031] Preferably, this embodiment further specifies the specific implementation mechanism for adaptive switching between knowledge-driven and exploration-driven modes based on loss statistics during training, as well as the construction method of differentiated loss functions for the two modes. This mechanism aims to solve the problem that fixed switching step sizes in existing technologies are difficult to adapt to different training dynamics, and automatically balances the guiding role of prior knowledge and the exploratory ability of reinforcement learning by monitoring the optimization state of the policy model in real time.

[0032] Specifically, the adaptive switching process based on loss statistics during training includes: calculating the first mean loss within the historical window, the second mean loss within the current window, and the standard deviation of the loss within the current window; when the ratio of the second mean loss to the first mean loss is less than a first preset threshold, and the standard deviation of the loss is less than the second preset threshold, the training mode is switched from knowledge-driven mode to exploration-driven mode. In this process, the width of the historical window and the current window can be set according to the convergence characteristics of the actual task. For example, the width of the historical window can be set to 1000 gradient update steps, and the width of the current window can be set to 200 gradient update steps. The first mean loss represents the average optimization level of the policy model over a past period, while the second mean loss reflects the recent optimization level. The ratio of the two essentially represents the rate of decrease of the loss function, i.e., the speed at which the policy model learns and absorbs prior knowledge. When this ratio is less than the first preset threshold (e.g., 0.8), it indicates that the loss within the current window has significantly decreased compared to the historical window, and the policy model has basically mastered the behavioral patterns implied by the prior knowledge. However, switching based solely on the mean ratio may be affected by training noise, leading to erroneous switching during loss oscillations. Therefore, this embodiment introduces the loss standard deviation as a stability criterion. The loss standard deviation reflects the degree of fluctuation of the loss value within the current window. When the loss standard deviation is less than a second preset threshold (e.g., 0.05), it indicates that the optimization process of the policy model has become stable and is no longer in a stage of violent oscillation. Only when both the conditions of "rapid convergence" and "stable optimization" are met simultaneously does the system determine that the policy model has the ability to break away from strong knowledge guidance, thereby triggering a switch from a knowledge-dominated mode to an exploration-dominated mode. Regarding the determination of the first and second preset thresholds, the distribution characteristics of loss changes can be statistically analyzed through multiple pre-experiments, and outlier boundaries that can cover the vast majority of normal convergence trajectories can be selected as thresholds to ensure the robustness of the switching mechanism. It should be understood that although this embodiment uses a combination of mean ratio and standard deviation as the criteria, in other embodiments, the loss change rate, gradient norm, or other statistics that can represent the convergence state can also be used to construct the switching conditions, as long as they can achieve the function of automatic adjustment mode based on training feedback. In the above adaptive switching process, the formal definitions of historical window mean, current window mean, current window loss standard deviation, and switching conditions are as follows: ; In the formula: For the first The average Actor loss of the historical window during subgradient updates For the first The average Actor loss of the current window during subgradient update For the width of the history window, The width of the current window. For the first Actor loss during subgradient update.

[0033] ; In the formula: For the first The standard deviation of the loss for the current window during subgradient update. The width of the current window. For the first Actor loss during subgradient update This represents the average loss within the current window.

[0034] ; In the formula: For the first Indicators for determining adaptive switching conditions during subgradient updates The average Actor loss for the current window. The average Actor loss for the historical window. This is the first preset threshold.

[0035] After determining the training mode, differentiated loss functions are used to update the policy models at different levels to adapt to the optimization objectives at each stage. In the knowledge-dominated mode, a supervised learning loss function is used to update the first policy model, and a mean squared error loss function is used to update the second policy model. Specifically, in this mode, the first policy model uses the first prior action (such as discrete bolt selection based on diagonal or planetary rules) as the supervision label and is quickly pre-trained using cross-entropy loss or similar classification loss, enabling the high-level policy to quickly learn the fastening sequence that conforms to the process specifications; the second policy model uses the second prior action (such as continuous expectation force generated based on fuzzy inference) as the regression target and approximates the output of the rule controller using mean squared error loss. This purely supervised training method can utilize high-quality prior knowledge to provide a good starting point for the policy model in the early stages of training, effectively avoiding the problem of blind exploration in reinforcement learning under sparse reward environments. The supervised learning loss function of the first policy model and the mean squared error loss function of the second policy model in the knowledge-dominated mode can be formalized as follows: ; In the formula: The cross-entropy loss function for the first strategy model is... For the parameters of the first strategy model, Indicates the experience replay buffer medium state Expectations The first policy model in the state Next action The value estimate, It is a set of rules generated based on the principles of diagonal alternating fastening and planetary rotational fastening.

[0036] ; In the formula: For the Actor loss function in the knowledge-dominated model, For actor network parameters, Indicates based on dataset medium state Expectation of the distribution For actors' networks in the current state The compound action output below, Reference actions for rules generated by the rule controller at the same time.

[0037] In the exploration-driven mode, the first policy model is updated using a reinforcement learning temporal difference loss function, and the second policy model is updated using a loss function that integrates reinforcement learning objectives and knowledge-guided soft constraints. The knowledge-guided soft constraints are determined based on the difference between the continuous control actions output by the second policy model and the second prior action. Specifically, when switching to exploration-driven mode, the first policy model is updated using a temporal difference loss based on the Bellman equation (such as Q-learning loss) to maximize long-term cumulative rewards and further explore potentially better tight-fitting orders based on the learned rules. For the second policy model, its loss function consists of two parts: one is the standard reinforcement learning loss (such as the policy gradient loss in the Actor-Critic framework), used to drive the policy towards higher rewards; the other is the knowledge-guided soft constraint term, typically represented as the mean squared error or L1 distance between the policy output action and the second prior action, multiplied by the knowledge constraint weight coefficient λ. The knowledge constraint weight coefficient λ is used to balance the relative importance between the reinforcement learning objective and the knowledge guidance term. It can be set to a larger value in the early stages of training to enhance the rule-guided effect, and can be gradually decayed as training progresses so that the policy relies more on the reinforcement learning objective for autonomous exploration. This soft constraint term serves to retain prior knowledge as a "safety anchor" even during the exploration phase, preventing the policy from taking extreme actions that violate physical laws or may lead to assembly failure. As exploration deepens, the weight coefficient can be gradually decayed to allow the policy to gradually break free from its dependence on rules, eventually converging to the global optimum. Compared to a hard switch that completely removes rule constraints, this soft constraint mechanism provides a smoother transition while ensuring safety, significantly improving training stability. The reinforcement learning temporal difference loss function of the first policy model and the fusion loss function of the second policy model in the exploration-dominated mode can be formalized as follows: ; In the formula: For reinforcement learning loss function of high-level policy networks; Indicates the experience replay buffer The expected values ​​of the sampled state, action, reward, and next state; For the current state-action value estimate of the high-level policy network output; The target value for time-series difference is calculated using the following formula: ,in For instant rewards, As a discount factor, This is the end marker; These are the parameters for the high-level policy network.

[0038] ; In the formula: Let be the reinforcement learning loss function of the Actor network. The state-action value estimate for the commentator network output, the first term represents maximizing the expected return; The first term represents the knowledge constraint weighting coefficient, and the second term represents the mean square error between the policy output and the rule action, which is used to provide soft constraints during the exploration process.

[0039] By combining the aforementioned adaptive switching mechanism with the differentiated loss function, this embodiment enables the policy model to fully utilize expert knowledge to compress the search space in the early stages of training, automatically identify the convergence state and smoothly transition to the exploration phase in the middle stages of training, and continuously optimize until convergence under safety constraints in the later stages of training. The entire process does not require manual pre-setting of fixed switching time steps, nor does it require repeated tuning of hyperparameters for specific tasks, significantly reducing the application threshold of the method and improving its generalization ability and robustness in different assembly scenarios.

[0040] Preferably, this embodiment further defines the specific composition of the state space and the composite action space, as well as the network structure and parameter update method of the second strategy model. Specifically, the current state data includes the real-time tightening torque and real-time contact force of each bolt position, as well as discrete state codes representing the current tightening stage and assembly progress. In the multi-bolt assembly task of flanges, a single mechanical signal is often insufficient to fully characterize the assembly state. The real-time tightening torque and real-time contact force directly reflect the physical interaction strength between the bolts and the flange, and are the key basis for judging whether the preload is up to standard; while the discrete state code provides the necessary temporal context information, enabling the strategy model to perceive which stage of the assembly process it is currently in (e.g., initial alignment, pre-tightening, or final tightening), and which bolts have been tightened. This state representation, which integrates continuous mechanical quantities and discrete semantic quantities, provides a complete decision-making basis for the hierarchical strategy model. Correspondingly, the composite action space is decoupled into two subspaces: discrete selection actions correspond to the discrete selection of the target tightening bolts, and continuous control actions correspond to the continuous expected force value output by the impedance controller. This decoupling design allows high-level strategies to focus on the time-series planning problem of "who to secure next," while low-level strategies focus on the fine-grained control problem of "how much force to apply." The two perform their respective functions while working together, effectively reducing the learning difficulty of long-term complex decision-making.

[0041] For the specific implementation of the second policy model, this embodiment adopts an actor-critic structure. The actor network takes the concatenated vector of discrete choice actions output by the first policy model and the current state data as input, and outputs a continuous expected force value. This input concatenation is designed to establish an explicit conditional dependency of continuous force control on discrete choices. Because in actual assembly, even with the same current mechanical reading, the required expected force value may be drastically different for bolts in different locations or at different assembly stages. By injecting discrete choice actions as conditional variables into the actor network's input, the lower-level policy is forced to learn a conditional policy of "how to adjust the force value according to the current state given a specific bolt," thus avoiding ambiguous or erroneous force control outputs from the lower-level policy due to a lack of high-level decision context. The critic network adopts a dual-Q network structure, taking the current state data and composite actions as input, and outputting a state-action value estimate used to update the policy gradient of the actor network. In reinforcement learning training, single-Q networks are prone to overestimation of Q-values ​​due to function approximation errors, leading to policy instability or even divergence. This embodiment introduces a dual-Q structure, where the smaller of the estimates from two independent Q-networks is used when calculating the temporal difference objective. This effectively suppresses overestimation and improves the accuracy of value assessment and the stability of the training process. When calculating the temporal difference objective, the target action is jointly generated by the target network and superimposed with truncated Gaussian noise. The truncation range of this truncated Gaussian noise is typically set to a bounded interval to prevent excessive deviation of the target action from causing training instability. Compared to unbounded Gaussian noise, truncated Gaussian noise ensures that the target action remains within a reasonable range, further improving training stability.

[0042] In this embodiment, the output of the actor network and the double Q loss function of the critic network can be formalized as follows: ; In the formula: For low-level strategies at time The output continuous expected force value, For low-level actor (Actor) networks, For the parameters of the actor network, For high-level strategy at all times Selected discrete bolt numbering action, For a moment The current state vector, This indicates that the high-level decision-making actions are combined with the current state.

[0043] ; In the formula: The loss value of the critic network, For the critic's network parameters, This is the index of the double-Q network (with a value of 1 or 2). Indicates the experience replay buffer Expected sequence of mid-sampled state, action, reward and next state , for the first A network of critics on the current state and compound movements The value estimate, The temporal difference objective value is determined by the state at the next time step, the action generated by the target network, and the immediate reward. and discount factor The calculation yielded the result.

[0044] To further improve training stability, a soft update method is used during the training phase to synchronize the target network parameters corresponding to the first policy model, the actor network, and the critic network. Specifically, the soft update method uses exponential linear interpolation to approximate the target network parameters to the current network parameters. Its mathematical expression is typically: the new target network parameter equals (1 minus the soft update coefficient) multiplied by the old target network parameter plus the soft update coefficient multiplied by the current network parameter. Compared to the hard update method that directly copies the current network parameters, this soft update mechanism based on exponential moving averages makes the changes in the target network smoother, avoiding oscillations in the temporal difference target value caused by drastic fluctuations in the current network parameters, thus significantly improving the convergence performance of deep reinforcement learning algorithms in complex assembly tasks. The soft update coefficient is usually set to a small positive number (e.g., between 0.005 and 0.01) to ensure that the target network can slowly track the learning progress of the current network while maintaining sufficient smoothness to filter high-frequency noise. It should be understood that the specific soft update coefficients can be adjusted according to the dynamic characteristics of the actual task. As long as the function of stabilizing the training process through gradual updates is satisfied, it should be considered an equivalent implementation of this invention. The soft update synchronization process of all target network parameters can be formalized as follows: ; In the formula: For high-level policy network parameters, For low-level actor network parameters, For the critic's network parameters, The soft update coefficient (i.e., the update step size or mixing ratio of the target network parameters) is represented by the minus sign and subscript in the superscript, which indicate the version of the target network parameters of the corresponding layer. This formula achieves slow approximation of the target network parameters to the current network parameters through exponential linear interpolation to stabilize training.

[0045] Example 2: This example provides a multi-bolt assembly control device based on knowledge-enhanced hierarchical reinforcement learning. This device is a concrete manifestation of the aforementioned method embodiments at the functional module level, forming multiple protections through a pure functional module architecture and methodological independence. Specifically, the device includes a state acquisition module, an action output module, a priori generation module, and an adaptive training module.

[0046] The state acquisition module is used to acquire current state data during the robot assembly process. Specifically, this module serves as the perception entry point for the entire control device, responsible for real-time acquisition and preprocessing of multimodal information from the assembly environment. In the signal flow relationship, the current state data output by this module is transmitted in parallel to the action output module and the prior generation module, providing a unified data foundation for subsequent decision-making reasoning and knowledge generation. For example, this module can read six-dimensional force / torque sensor data and joint encoder data by calling the sensor interface, and encapsulate them into a standardized state vector format, thereby shielding the impact of underlying hardware differences on upper-level algorithms.

[0047] The action output module is used to input current state data into the hierarchical strategy model and output composite actions to control the robot to perform assembly operations. The composite actions include discrete selection actions and continuous control actions. The hierarchical strategy model includes a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. Specifically, this module carries the core hierarchical decision-making function. After receiving the state data, its internal first strategy model first outputs discrete selection actions to determine the target bolt. Subsequently, the second strategy model outputs continuous control actions based on the concatenated vector of the discrete selection result and the state data to adjust the desired force value. This modular action output mechanism ensures the decoupled execution of discrete timing planning and continuous force control adjustment, enabling the device to flexibly adapt to different robot body and end effector configurations. It should be understood that although this embodiment describes a specific hierarchical processing sequence, in other implementations, as long as the functional logic of outputting two types of actions separately is satisfied, it can be considered an equivalent substitution.

[0048] The prior generation module is used during the training phase of the hierarchical policy model to generate a first prior action matching discrete selection actions and a second prior action matching continuous control actions based on the current state data. Specifically, this module is dedicated to the knowledge injection stage of the training phase. The generated prior actions do not directly affect the physical robot but are passed to the adaptive training module as supervision signals or soft constraints to guide the parameter update direction of the hierarchical policy model. This module integrates sequence mapping rules and multi-source evidence fusion logic, enabling it to dynamically generate reference actions conforming to process specifications based on the current state. By making prior generation an independent functional module, this invention decouples knowledge representation from policy learning, facilitating flexible replacement or upgrading of the prior knowledge base for different assembly tasks without modifying the core policy network structure.

[0049] The adaptive training module adaptively switches between knowledge-driven and exploration-driven modes based on loss statistics during training to update the parameters of the hierarchical policy model. Specifically, this module acts as the control center of the training process, receiving feedback from loss statistics during policy optimization and dynamically adjusting the training mode accordingly. When the loss decrease rate and stability meet preset conditions, the module automatically triggers a switch command from knowledge-driven to exploration-driven mode, thereby changing the gradient update strategy of the hierarchical policy model. This closed-loop feedback mechanism enables the device to autonomously balance knowledge utilization and policy exploration without human intervention, significantly improving training efficiency and final performance. In terms of signal flow, the output of this module directly affects the optimizer configuration of the hierarchical policy model, forming a complete data closed loop of "state awareness - decision output - prior guidance - adaptive optimization".

[0050] In practical deployment, the aforementioned functional modules can be stored as software programs in a computer-readable storage medium and loaded and executed by a processor. This device can be integrated into the robot's local controller for low-latency real-time control, or deployed on a host computer or cloud server to support large-scale parallel training and offline optimization. Through this purely functional modular architecture design, this embodiment not only maintains strict consistency with the scope of protection of the method's proprietary rights but also avoids the narrowing of the scope of protection caused by introducing specific hardware connections, providing clear functional boundaries for subsequent rights protection and infringement determination.

[0051] Example 3: This example provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method. The processor can be a chip with data processing capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC). It is responsible for executing the program instructions stored in the memory to complete core computational tasks such as state data acquisition, hierarchical policy model inference, prior action generation, and adaptive training mode switching. The memory is used to persistently or temporarily store the computer program code, model parameters, historical loss statistics, and state data during the assembly process required to implement the above method. It should be understood that this invention does not limit the specific hardware form of the computer device. It can be an industrial personal computer (IPC) or embedded robot controller deployed on the production line to achieve low-latency real-time assembly control; it can also be a server cluster located in the cloud or data center for offline training and parameter optimization of the hierarchical policy model using large-scale computing power; or it can be an edge computing module integrated inside the robot body to balance real-time performance and computing power requirements.

[0052] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method. For example, the storage medium can be a tangible magnetic storage device (such as a hard disk, floppy disk, magnetic tape), an optical storage device (such as a CD-ROM, DVD), or a semiconductor storage device (such as a solid-state drive (SSD), flash memory (Flash), EEPROM, ROM, RAM, etc.). By encoding and storing the program instructions for implementing the aforementioned assembly control method in such a medium, the technical solution of the present invention can be distributed and delivered as an independent software product. In practical applications, users can install the medium storing the program into existing robot control systems or host computers, thereby upgrading general-purpose equipment into dedicated assembly control equipment with knowledge-enhanced hierarchical reinforcement learning capabilities.

[0053] Example 4: To verify the effectiveness and superiority of the multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning proposed in this invention, this example built a flange multi-bolt assembly test platform in a virtual simulation environment and conducted multiple sets of comparative experiments. For example... Figure 4As shown, the simulation environment is built on the Isaac Sim platform and mainly includes a UR5 robotic arm 1, a six-dimensional force / torque sensor 2, a nut 3, a flange 4, and bolts 5 to be tightened. The UR5 robotic arm 1 serves as the execution body, with its base fixed behind the experimental platform. Its end is connected to the six-dimensional force / torque sensor 2 via a flange, used to collect the axial contact force Fz and tightening torque Tz during the assembly process in real time. The flange 4 is placed in the center of the platform and has four bolt holes arranged in a circle. The bolts 5 and the nut 3 cooperate to complete the assembly task. Considering that the preload is difficult to measure directly in the simulation environment, this embodiment uses the axial contact force Fz and tightening torque Tz as indirect evaluation indicators, with reasonable value ranges set as Fz∈[0.1, 0.7]N and Tz∈[0.01, 0.03]N·m, respectively. This simulation environment can simulate the elastic coupling effect and mechanical interaction characteristics in the multi-bolt assembly process with high fidelity, providing a reliable test benchmark for algorithm verification.

[0054] In the test of this embodiment, the key hyperparameters mentioned in the previous embodiments are assigned the following specific values: the historical window width Wh is set to 1000 gradient update steps, the current window width Wc is set to 200 gradient update steps, the first preset threshold ρ used to calculate the loss mean and standard deviation is set to 0.8, and the second preset threshold β is set to 0.05.

[0055] First, to verify the effectiveness of the differentiated knowledge injection mechanism, this embodiment compares the complete method proposed in this invention with three baseline methods: a baseline reinforcement learning method that does not employ a hierarchical structure and does not introduce rules, a method that directly adds rule information to a non-hierarchical baseline framework, and a method that retains the hierarchical structure but does not introduce a rule controller. After 100 independent deployment tests, the average assembly success rate is shown in Table 1 below: Table 1

[0056] Experimental results show that the complete method proposed in this invention significantly outperforms other comparative methods in terms of assembly success rate, achieving 100%. In contrast, while methods that retain only the hierarchical structure without introducing rule controllers can achieve high sequential accuracy in some stages, the lack of prior guidance from continuous force control makes it difficult for the agent to learn a stable resistance adjustment strategy under sparse rewards, resulting in a final success rate of only 49.1%. The baseline method without a hierarchical structure and its simple rule-enhanced form, due to the failure to effectively decouple discrete and continuous action spaces and the lack of targeted knowledge injection, struggles to converge stably throughout the training process, resulting in an extremely low success rate. This result strongly demonstrates that this invention, by decoupling long-sequential complex actions into two levels—discrete selection and continuous control—and injecting sequence mapping rules and multi-source evidence fusion rules respectively, can effectively compress the blind exploration space in the early stages of training, enabling the policy to quickly learn assembly behavior that conforms to physical laws, thereby significantly improving the assembly success rate and training efficiency.

[0057] Furthermore, to verify the independent contribution of the adaptive training mode switching mechanism based on loss statistics, this embodiment sets up comparative training groups with different fixed knowledge guidance durations and compares them with the adaptive update strategy group proposed in this invention. The test results are shown in Table 2 below: Table 2

[0058] The data in the table shows that if the knowledge guidance time is too short or the rule constraints are removed from the early stages of training, the strategy will enter the free exploration stage before fully absorbing prior knowledge. This makes it prone to losing its way due to an excessively large action space, resulting in a success rate of only 6.47%. Conversely, if the rule guidance time is too long, it will excessively compress the later exploration space, limiting the strategy's evolution towards the global optimum, causing the success rate to stagnate at 61.5%. Although the optimal fixed switching parameter set obtained through multiple pre-experiments can also achieve a 100% success rate, this relies heavily on manual parameter tuning experience and trial-and-error costs. The adaptive update strategy proposed in this invention, by monitoring the loss reduction rate and stability in real time, can automatically identify the strategy's convergence state and smoothly transition to the exploration-dominated mode without requiring manual pre-setting of fixed switching time steps, ultimately achieving a 100% success rate comparable to the optimal fixed parameter set. This result fully demonstrates that the adaptive switching mechanism not only ensures the strategy's full-cycle optimization capability but also significantly reduces the dependence on manual parameter tuning experience, improving the method's generalization ability and engineering practicality in different assembly scenarios.

[0059] In summary, through the simulation tests and data analysis, the knowledge-enhanced hierarchical reinforcement learning method and system proposed in this invention demonstrate high efficiency, environmental adaptability, and robustness in handling long-sequence, discrete-continuous composite action coupling robot autonomous assembly tasks such as multi-bolt flange assembly. In particular, the introduction of an adaptive update strategy enables the system to automatically balance the relationship between knowledge utilization and strategy exploration while ensuring training efficiency, providing a practical solution for the intelligent control of complex industrial assembly tasks.

[0060] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning, characterized in that, include: Step 1: Obtain the current status data during the robot's multi-bolt assembly process; Step 2: Input the current state data into the hierarchical strategy model and output composite actions to control the robot to perform assembly operations; the composite actions include discrete selection actions and continuous control actions, and the hierarchical strategy model includes a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. Step 3: During the training phase of the hierarchical strategy model, a first prior action matching the discrete selection action and a second prior action matching the continuous control action are generated based on the current state data. The first prior action is generated by constructing a pairing mapping function based on the principles of diagonal alternating tightening and planetary rotation tightening. Combining the bolt states selected in the current step and the previous step, selected states are excluded from the candidate bolt set, and an action is randomly selected as the first prior action. The second prior action is generated by using triangular and trapezoidal membership functions to perform fuzzy modeling of the axial contact force and axial tightening torque, respectively, to obtain the membership degree of the corresponding evaluation level. The membership degree is truncated to a preset interval to construct a basic probability assignment function, and multi-channel evidence is normalized and fused using evidence combination rules. The reference force value corresponding to the evaluation level with the largest fused basic probability assignment value is selected as the second prior action. Step 4: Based on the loss statistics during training, adaptively switch between knowledge-driven mode and exploration-driven mode to update the parameters of the hierarchical policy model; wherein, the adaptive switching includes: Calculate the first loss mean within the historical window, the second loss mean within the current window, and the loss standard deviation within the current window; When the ratio of the second loss mean to the first loss mean is less than a first preset threshold, and the loss standard deviation is less than a second preset threshold, the training mode is switched from knowledge-driven mode to exploration-driven mode.

2. The multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning according to claim 1, characterized in that, In the knowledge-dominated mode, the first policy model is updated using a supervised learning loss function, and the second policy model is updated using a mean squared error loss function. In the exploration-dominated mode, the first policy model is updated using the reinforcement learning temporal difference loss function, and the second policy model is updated using a loss function that integrates reinforcement learning objectives and knowledge-guided soft constraints. The knowledge-guided soft constraint is determined by multiplying the difference between the continuous control action output by the second strategy model and the second prior action by the knowledge constraint weight coefficient.

3. The multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning according to claim 1, characterized in that, The current status data includes the real-time tightening torque, real-time contact force of each bolt position, and discrete status codes characterizing the current tightening stage and assembly progress. The discrete selection action corresponds to the discrete selection of the target fastening bolt, and the continuous control action corresponds to the continuous desired force value output by the impedance controller.

4. The multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning according to claim 3, characterized in that, The second strategy model adopts an actor-critic structure, where: In the actor-critic structure, the actor network takes the concatenated vector of discrete choice actions output by the first strategy model and the current state data as input, and outputs a continuous expectation force value. The critic network in the actor-critic structure adopts a double-Q network structure, taking the current state data and compound actions as inputs, and outputting a state-action value estimate, which is used to update the policy gradient of the actor network.

5. The multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning according to claim 4, characterized in that, During the training phase, the target network parameters corresponding to the first policy model, the actor network, and the critic network are synchronized using a soft update method. The soft update method uses exponential linear interpolation to make the target network parameters approximate the current network parameters.

6. A multi-bolt assembly control device based on knowledge-enhanced hierarchical reinforcement learning, characterized in that, The device includes: The status acquisition module is used to acquire the current status data of the robot during the multi-bolt assembly process; The action output module is used to input the current state data into the hierarchical strategy model and output a composite action to control the robot to perform assembly operations; the composite action includes discrete selection actions and continuous control actions, and the hierarchical strategy model includes a first strategy model that outputs discrete selection actions and a second strategy model that outputs continuous control actions. The prior generation module is used to generate a first prior action that matches the discrete selection action and a second prior action that matches the continuous control action based on the current state data during the training phase of the hierarchical policy model. An adaptive training module is used to adaptively switch between a knowledge-driven mode and an exploration-driven mode based on the loss statistics during the training process in order to update the parameters of the hierarchical strategy model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the multi-bolt assembly control method based on knowledge-enhanced hierarchical reinforcement learning as described in any one of claims 1 to 5.

Images