Mechanical arm precise shaft hole assembling method based on lunar eclipse heuristic scene observation and simulation

Abstract

The invention discloses a mechanical arm precise shaft hole assembling method based on lunar eclipse heuristic scene observation and simulation. A visual sensor installed at the tail end of a mechanical arm is adopted for collecting image data of a to-be-assembled hole piece; performing target detection and positioning on a to-be-assembled hole piece in the image data through a deep learning-based YOLOv8 model, performing image acquisition on the to-be-assembled hole piece again by the visual sensor, and inputting a lunar eclipse assembly scene into a YOLOv8-seg network model for detection to obtain instance segmentation output information of the lunar eclipse assembly scene, the instance segmentation output information is input into a simulation learning strategy network model, and corresponding mechanical arm execution actions are output according to different lunar eclipse assembly scenes; and operating the mechanical arm to execute the action, returning to the process of obtaining the lunar eclipse assembly scene, inputting the obtained lunar eclipse assembly scene into the YOLOv8-seg network model to be continuously executed, finally obtaining the mechanical arm execution action, and repeatedly executing until the lunar eclipse assembly scene cannot be obtained, so that the alignment process is completed.

Description

Technical Field

[0001] The invention relates to the technical field of robot assembly, and in particular to a method for assembling a precise shaft hole of a robot arm based on lunar eclipse inspired scenario observation and simulation. Background Art

[0002] With the rapid development of advanced technologies such as artificial intelligence, robotics, automation and intelligent manufacturing, intelligent assembly has received widespread attention in the field of production and manufacturing. Existing industrial robotic arms perform assembly tasks mostly in structured environments. However, due to the uncertainty of the working environment and the complexity of assembly tasks, they are mainly completed by human operators, which cannot meet the needs of intelligence and are not universal. Therefore, it is very important to design an intelligent industrial robotic arm precision shaft hole assembly method based on the idea of ​​human-like scenario observation. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention discloses a method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation, which specifically comprises the following steps:

[0004] The visual sensor installed at the end of the robotic arm is used to collect image data of the hole parts to be assembled;

[0005] The YOLOv8 model based on deep learning is used to detect and locate the hole parts to be assembled in the image data. The visual sensor uses the depth image to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and outputs the bounding box, confidence, category and three-dimensional coordinates of the center point, and controls the robot arm to move in the three-dimensional space to the vicinity of the center point of the target hole part to complete the rough positioning process and form a lunar eclipse assembly scenario;

[0006] The visual sensor again collects images of the hole parts to be assembled, and inputs the lunar eclipse assembly scenario into the YOLOv8-seg network model for detection, and obtains instance segmentation output information of the lunar eclipse assembly scenario, including instance segmentation mask, output bounding box, confidence and lunar eclipse category;

[0007] The instance segmentation output information is input into the imitation learning strategy network model, and the corresponding robot arm execution actions are output according to different lunar eclipse assembly scenarios;

[0008] Run the robot arm to perform actions, return to the process of obtaining the lunar eclipse assembly scenario, and input the obtained lunar eclipse assembly scenario into the YOLOv8-seg network model to continue execution. Finally, obtain the robot arm to perform actions, and repeat the execution until the lunar eclipse assembly scenario cannot be obtained to complete the alignment process.

[0009] Furthermore, the lunar eclipse assembly scenario is formed in the following manner:

[0010] The RGB image and depth image of the on-site environment of the hole parts to be assembled are collected by an RGB-D camera, where the RGB image is used as the input information of the YOLOv8 model, and the output information of the model is the 2D bounding box of the hole parts to be assembled, the pixel coordinates of the vertices and center points of the bounding box, and the depth image is used to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system;

[0011] Calculate the 3D coordinates of the center point of the hole in the RGB image, and obtain the 3D coordinates of the center point of the hole to be assembled in the RGB-D camera coordinate system after target detection;

[0012] Based on the position control method, the end effector of the robotic arm is navigated to the set position and moved downward along the Z axis for a certain distance. The contact state is determined by the force sensor. If the monitored force exceeds the preset threshold, the lunar eclipse assembly scenario is roughly completed.

[0013] Furthermore, the mask in the lunar eclipse assembly scenario is obtained in the following manner: an RGB-D camera is used to obtain an RGB image in the current state, and the RGB image is used as input information of a YOLOv8-seg network model, and the output information of the YOLOv8-seg network model is a 2D bounding box of the lunar eclipse assembly scenario, a lunar eclipse mask, and a lunar eclipse category;

[0014] The backbone network of the YOLOv8-seg network model extracts the features of the input image and obtains the feature map, and converts the feature map into a mask through the mask terminal:

[0015] M=σ(W·F+b)

[0016] Where M is the predicted mask, σ is the activation function, W and b are the weight and bias respectively, and F is the feature map;

[0017] Control the mask generated by the YOLOv8-seg network model to match the real mask, and use the binary cross entropy loss function as the mask loss function:

[0018]

[0019] Where N is the number of samples in the batch, y i is the true label of the i-th sample, which takes the value of 0 or 1, p i is the probability predicted by the model, that is, the probability of belonging to this category, and its value range is 0≤p i ≤1.

[0020] Furthermore, according to different lunar eclipse assembly scenarios, the corresponding robot arm execution actions are output in the following specific manner:

[0021] Obtain the key information of each assembly action and state in the process of the expert user performing the shaft hole assembly task, form a state and corresponding action pair, and construct an expert dataset D, which contains N demonstration samples. Each sample consists of a state s i and the corresponding action a i composition:

[0022] Perform data cleaning, data alignment and data normalization on the collected assembly data of expert users, and divide them into training set and test set;

[0023] Construct an imitation learning strategy network model, imitate the learning expert assembly behavior strategy based on the behavior cloning model, input the state s into the imitation learning strategy network model, and output the probability distribution of action a:

[0024] π θ (a|s;θ)=P(a|s;θ)

[0025] Among them, θ is the parameter of the model, and the state s is forward propagated in the imitation learning strategy network model. The input data is convolved through the convolution kernel to extract local features. The convolution output is then nonlinearly mapped through the activation layer to enable the model to learn more complex features. The feature map is downsampled through the pooling layer. Finally, the extracted local features are mapped to a high-dimensional space through the fully connected layer, the global information is integrated, and finally the execution action of the robot is output.

[0026] Furthermore, the imitation learning strategy network model compares the action probability distribution generated by forward propagation based on the cross entropy loss function with the true labels of the pre-collected expert user assembly data to measure the difference between the robot arm's predicted action and the true action:

[0027]

[0028] Where: L(θ) represents the degree of match between the probability predicted by the model and the actual observed data given the parameter θ, π θ (a i |s i ; θ) means that under the parameter θ, given state s i When , select action a i probability;

[0029]

[0030] Where η is the learning rate, is the bias-corrected first-order moment estimate (related to the gradient), is the bias-corrected second-order moment estimate, ∈ is a small constant used for numerical stability to prevent the denominator from being zero;

[0031] The gradient of the loss relative to the model parameters is calculated through the back-propagation algorithm, and the parameters of the imitation learning strategy network model are updated according to the optimizer to maximize the predicted probability of the correct action, thereby minimizing the loss function L(θ) and improving the performance and prediction accuracy of the imitation learning network model;

[0032]

[0033] Where α is the learning rate, ▽ θ L(θ) is the gradient of the loss function with respect to the parameter θ;

[0034] Iterative training is performed to gradually optimize the parameters of the imitation learning strategy network model. After each parameter update, it is checked whether the loss function has reached the convergence condition or reached the preset maximum number of iterations.

[0035] Due to the adoption of the above-mentioned technical scheme, the present invention provides a method for precise shaft-hole assembly of a robotic arm based on lunar eclipse-inspired scenario observation and imitation. The method uses a visually guided robotic arm control scheme and a deep learning target recognition and positioning method to achieve rapid approximation connection between shaft holes, reduce the calibration process, and improve the versatility of shaft-hole assembly; proposes lunar eclipse assembly logic, uses the YOLOv8-seg network model to perform pixel-level instance segmentation on the lunar eclipse assembly scenario, and outputs a mask image as the input of the strategy network, which reduces the interference of irrelevant information such as the workbench background environment on the assembly and improves the recognition efficiency and detection accuracy; constructs an imitation learning strategy network model based on imitation learning, simplifies the control strategy through state-action pairs, and realizes expert assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of the method of the present invention;

[0038] Figure 2 This is the shaft hole assembly diagram inspired by the lunar eclipse scenario in the present invention;

[0039] Figure 3 This is a flowchart of the shaft hole assembly approach process based on the lunar eclipse scenario in the present invention.

[0040] Figure 4 This is a flowchart of the imitation learning network model based on the lunar eclipse scenario inspiration in the present invention.

[0041] Figure 5 This is a specific implementation flow chart of the mechanical arm shaft hole assembly operation in the present invention DETAILED DESCRIPTION

[0042] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0043] like Figure 1 A method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation is shown, and specifically comprises the following steps:

[0044] S1: In the initial state, the robot arm is at the zero initial position, the end of the robot arm is parallel to the workbench plane, the shaft parts to be assembled are installed at the end of the robot arm through the adapter flange, the hole parts to be assembled are randomly placed on the workbench plane, and the RGB-D camera is fixed near the end of the robot arm through a non-standard adapter and rotated a certain angle so that the field of view covers the entire work plane.

[0045] S2: The robot arm uses the YOLOv8 target detection model to identify and locate the hole part through the RGB-D camera installed near the end, and moves the end effector to the vicinity of the center of the hole part. The specific process is as follows Figure 3 shown.

[0046] S21: Use an RGB-D camera to obtain RGB images and depth images of the environment. The RGB image is used as the input of the YOLOv8 model to output the 2D bounding box of the hole to be assembled, the pixel coordinates of the vertices and center point of the bounding box.

[0047] S22: The depth image is used to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and calculate the three-dimensional coordinates P of the center point P = (u, v) of the hole in the RGB image h =(X h ,Y h ,Z h ) and output it to the host computer. The specific mapping conversion calculation process is as follows:

[0048] According to the imaging principle of the RGB-D camera, the camera intrinsic parameter matrix defines the projection process from 3D world coordinates to 2D image coordinates:

[0049]

[0050] where f x , f y is the focal length of the camera; c x , c y are the coordinates of the camera’s optical center;

[0051] For a given pixel coordinate point P = (u, v), the depth value can be expressed as Z = d(u, v),

[0052] Use the camera intrinsic parameter matrix to back-project back to the camera coordinate system:

[0053]

[0054] To ensure that the field of view of the RGB-D camera can cover the entire assembly work plane of the robot, the camera is tilted toward the direction of the robot base by θ to provide the robot with a more complete field of view. Considering that the camera is tilted θ in the positive direction of the X-axis, the coordinate system is rotated around the X-axis to obtain the corresponding rotation matrix R x :

[0055]

[0056] Apply the rotation matrix R to the point (X', Y', Z') in the camera coordinate system x After the rotation transformation, the new coordinates (X, Y, Z) are obtained:

[0057]

[0058] S23: The three-dimensional coordinates P of the center point of the hole to be assembled in the RGB-D camera coordinate system after target detection can be calculated by the above formula. h =(X h ,Y h ,Z h ).

[0059] S24: The host computer uses position control to navigate the end effector of the robot arm to (X h ,Y h ) and move down along the Z axis h , use the force sensor to determine the contact state, if the monitored force exceeds the predetermined threshold F Z , then the first stage of rough positioning is considered completed, forming the lunar eclipse assembly scenario, such as Figure 2 and Figure 3 shown.

[0060] S3: After rough positioning, the robot arm uses the YOLOv8-seg instance segmentation model through the RGB-D camera to identify and classify the lunar eclipse assembly scenario.

[0061] S31: Use the RGB-D camera to obtain the RGB image in the current state.

[0062] S32: The RGB image is used as the input of the YOLOv8-seg algorithm, which outputs the 2D bounding box, eclipse mask, and eclipse category of the lunar eclipse assembly scene. The eclipse mask generation is the key to this step.

[0063] Specifically, the backbone network of the YOLOv8 network extracts the features of the input image, obtains the feature map, and then converts the feature map into a mask through the mask terminal:

[0064] M=σ(W·F+b)

[0065] Where M is the predicted mask, σ is the activation function, W and b are the weight and bias respectively, and F is the feature map.

[0066] The activation function uses the sigmoid function to limit the output value to [0,1]:

[0067]

[0068] Among them, M ij Represents the mask value of the pixel in the i-th row and j-th column.

[0069] In order to train the model so that the generated mask matches the real mask as closely as possible, the binary cross entropy loss function is used as the mask loss function:

[0070]

[0071] Where N is the number of samples in the batch, y i is the true label of the i-th sample, which takes the value of 0 or 1, p i is the probability predicted by the model, that is, the probability of belonging to this category, and its value range is 0≤p i ≤1.

[0072] S4: Input the output state of S3 into the imitation learning strategy network to select the corresponding expert assembly action. The steps to build the imitation learning strategy network are:

[0073] S41: Collect expert data: Human experts complete multiple robot arm shaft hole assembly tasks, collect RGB images to obtain key information of each assembly action and state during the expert's shaft hole assembly task, form state and corresponding action pairs, and construct an expert dataset D, which contains N demonstration samples, each sample consists of state s i and the corresponding action a i composition:

[0074]

[0075] Constructing the goal to learn a policy network π θ (a|s;θ), where θ is the parameter of the network, so that for each state s i , the action predicted by the model is a i As close as possible to expert moves.

[0076] S42: Data preprocessing: Perform data cleaning, data alignment, data normalization and other operations on the collected expert data, and divide the data set into a training set and a test set.

[0077] S43: Design an imitation learning strategy network, adopt a behavior cloning model, and imitate the learning expert to assemble the behavior strategy. The strategy network model receives the state s as input and outputs the probability distribution of action a:

[0078] π θ (a|s;θ)=P(a|s;θ)

[0079] Specifically, the state s is forward propagated in the imitation learning strategy network model, and the input data is convolved through the convolution kernel to extract local features; the convolution output is then nonlinearly mapped through the activation layer, so that the model can learn more complex features; the feature map is downsampled through the pooling layer; finally, the extracted local features are mapped to a high-dimensional space through the fully connected layer, the global information is integrated, and finally the execution action of the robot arm is output.

[0080] S44: Define the loss function. In order to effectively predict discrete-value action instructions, the cross entropy loss function is used to compare the action probability distribution generated by forward propagation with the true label of the expert operation data collected in advance, measure the difference between the predicted action of the robot arm and the true action, and calculate the loss.

[0081]

[0082] Where: L(θ) represents the degree of match between the probability predicted by the model and the actual observed data given the parameter θ, π θ (a i |s i ; θ) means that under the parameter θ, given state s i When i probability.

[0083]

[0084] Where η is the learning rate, is the bias-corrected first-order moment estimate (related to the gradient), is the bias-corrected second-order moment estimate (related to the square of the gradient), and ∈ is a small constant used for numerical stability to prevent the denominator from being zero.

[0085] S45: Parameter updating process: In order to minimize the loss function L(θ), the gradient of the loss relative to the model parameters is calculated through the back propagation algorithm, and the parameters of the imitation learning strategy network model are updated according to the optimizer to maximize the predicted probability of the correct action, thereby improving the performance and prediction accuracy of the imitation learning network model.

[0086]

[0087] Among them, α is the learning rate, is the gradient of the loss function with respect to the parameter θ. This can reduce the cross entropy and make the decision made by the policy network closer to the expert's action.

[0088] S46: Loss function convergence check process: Perform iterative training to gradually optimize the parameters of the imitation learning strategy network model. After each parameter update, check whether the loss function has reached the convergence condition or reached the preset maximum number of iterations.

[0089] S5: The host computer uses the position control robot end effector to perform the corresponding expert assembly action, and then repeats S3 to observe the eclipse assembly scenario after the action is completed, and uses this scenario as the state input of the next S4 to select the next action. Executing S4 once can get one assembly action. The shaft-hole assembly task often requires multiple assembly actions. Therefore, the intelligent agent needs to execute S4 multiple times to obtain multiple assembly actions to achieve the alignment work before the shaft is vertically inserted into the hole.

[0090] Specifically, Figure 4 As shown in the figure, when S3 cannot observe the lunar eclipse assembly scenario, the assembly state is approximately aligned, and there is only a small gap between the shaft and the hole, which can be eliminated by other mainstream hole search methods. Then, the end effector of the robot arm is given a vertical downward shaft hole insertion instruction to perform the assembly operation task and complete the shaft hole assembly.

[0091] S6: Use the spiral hole search method to eliminate small deviations, complete the alignment process, and install the shaft vertically downward into the hole. The specific implementation process is as follows: Figure 5 shown.

[0092] In an embodiment of the present invention, a neural network based on deep learning is used to process image data in a complex environment, which can better simulate the shaft hole assembly task in an unstructured environment and improve the assembly efficiency of the robotic arm. An innovative shaft hole assembly method based on lunar eclipse scenario observation imitation learning is proposed. The cylindrical shaft hole assembly scenario is inspired into a lunar eclipse scenario based on human-like thinking, and the assembly logic is understood by using real-time lunar eclipse semantics. The action output by the expert strategy network controls the movement direction and distance of the robot, thus overcoming the problem of the slowness of the traditional hole search method. In summary, this method can make the shaft hole assembly process more flexible, adaptable and robust.

[0093] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for precise shaft-hole assembly of a robotic arm based on lunar eclipse heuristic scenario observation and simulation, characterized in that include: The visual sensor installed at the end of the robotic arm is used to collect image data of the hole parts to be assembled; The YOLOv8 model based on deep learning is used to detect and locate the hole parts to be assembled in the image data. The visual sensor uses the depth image to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and outputs the bounding box, confidence, category and three-dimensional coordinates of the center point, and controls the robot arm to move in the three-dimensional space to the vicinity of the center point of the target hole part to complete the rough positioning process and form a lunar eclipse assembly scenario; The visual sensor again collects images of the hole parts to be assembled, and inputs the lunar eclipse assembly scenario into the YOLOv8-seg network model for detection, and obtains instance segmentation output information of the lunar eclipse assembly scenario, including instance segmentation mask, output bounding box, confidence and lunar eclipse category; The instance segmentation output information is input into the imitation learning strategy network model, and the corresponding robot arm execution actions are output according to different lunar eclipse assembly scenarios; Run the robot arm to perform actions, return to the process of obtaining the lunar eclipse assembly scenario, and input the obtained lunar eclipse assembly scenario into the YOLOv8-seg network model to continue execution. Finally, obtain the robot arm to perform actions, and repeat the execution until the lunar eclipse assembly scenario cannot be obtained to complete the alignment process.

2. The method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation according to claim 1, characterized in that: The lunar eclipse assembly scenario is formed in the following manner: The RGB image and depth image of the on-site environment of the hole parts to be assembled are collected by an RGB-D camera, where the RGB image is used as the input information of the YOLOv8 model, and the output information of the model is the 2D bounding box of the hole parts to be assembled, the pixel coordinates of the vertices and center points of the bounding box, and the depth image is used to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system; Calculate the 3D coordinates of the center point of the hole in the RGB image, and obtain the 3D coordinates of the center point of the hole to be assembled in the RGB-D camera coordinate system after target detection; Based on the position control method, the end effector of the robotic arm is navigated to the set position and moved downward along the Z axis for a certain distance. The contact state is determined by the force sensor. If the monitored force exceeds the preset threshold, the lunar eclipse assembly scenario is roughly completed.

3. The method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation according to claim 2, characterized in that: The mask in the lunar eclipse assembly scenario is obtained by using an RGB-D camera to obtain an RGB image in the current state, and using the RGB image as input information of a YOLOv8-seg network model, wherein the output information of the YOLOv8-seg network model is a 2D bounding box of the lunar eclipse assembly scenario, a lunar eclipse mask, and a lunar eclipse category; The backbone network of the YOLOv8-seg network model extracts the features of the input image and obtains the feature map, and converts the feature map into a mask through the mask terminal: M=σ(W·F+b) Where M is the predicted mask, σ is the activation function, W and b are the weight and bias respectively, and F is the feature map; Control the mask generated by the YOLOv8-seg network model to match the real mask, and use the binary cross entropy loss function as the mask loss function: Where N is the number of samples in the batch, y i is the true label of the i-th sample, which takes the value of 0 or 1, p i is the probability predicted by the model, that is, the probability of belonging to this category, and its value range is 0≤p i ≤1.

4. The method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation according to claim 1, characterized in that: According to different eclipse assembly scenarios, the following methods are used to output the corresponding robot arm execution actions: Obtain the key information of each assembly action and state in the process of the expert user performing the shaft hole assembly task, form a state and corresponding action pair, and construct an expert dataset D, which contains N demonstration samples. Each sample consists of a state s i and the corresponding action a i composition: Perform data cleaning, data alignment and data normalization on the collected assembly data of expert users, and divide them into training set and test set; Construct an imitation learning strategy network model, imitate the learning expert assembly behavior strategy based on the behavior cloning model, input the state s into the imitation learning strategy network model, and output the probability distribution of action a: p θ (a|s;θ)=P(a|s;θ) Among them, θ is the parameter of the model, and the state s is forward propagated in the imitation learning strategy network model. The input data is convolved through the convolution kernel to extract local features. The convolution output is then nonlinearly mapped through the activation layer to enable the model to learn more complex features. The feature map is downsampled through the pooling layer. Finally, the extracted local features are mapped to a high-dimensional space through the fully connected layer, the global information is integrated, and finally the execution action of the robot is output.

5. The method for assembling a precise shaft hole of a robotic arm based on lunar eclipse heuristic scenario observation and simulation according to claim 4, characterized in that: The imitation learning strategy network model compares the action probability distribution generated by forward propagation based on the cross entropy loss function with the true labels of the pre-collected expert user assembly data to measure the difference between the robot arm's predicted action and the real action: Where: L(θ) represents the degree of match between the probability predicted by the model and the actual observed data given the parameter θ, π θ (a i |s i ; θ) means that under the parameter θ, given state s i When i probability; Where η is the learning rate, is the bias-corrected first-order moment estimate (related to the gradient), is the bias-corrected second-order moment estimate, ∈ is a small constant used for numerical stability to prevent the denominator from being zero; The gradient of the loss relative to the model parameters is calculated through the back-propagation algorithm, and the parameters of the imitation learning strategy network model are updated according to the optimizer to maximize the predicted probability of the correct action, thereby minimizing the loss function L(θ) and improving the performance and prediction accuracy of the imitation learning network model; Where α is the learning rate, is the gradient of the loss function with respect to the parameter θ; Iterative training is performed to gradually optimize the parameters of the imitation learning strategy network model. After each parameter update, it is checked whether the loss function has reached the convergence condition or reached the preset maximum number of iterations.

Images