Lunar eclipse inspired scenario observation mimicking based robotic precise shaft-hole assembly method

Abstract

The application discloses a mechanical arm precise shaft hole assembly method based on a lunar eclipse heuristic scene observation simulation, adopts a visual sensor installed at the end of the mechanical arm to collect image data of a to-be-assembled hole part; target detection and positioning are performed on the to-be-assembled hole part in the image data based on a YOLOv8 model of deep learning, the visual sensor collects image data of the to-be-assembled hole part again, inputs a lunar eclipse assembly scene into a YOLOv8-seg network model for detection, obtains instance segmentation output information of the lunar eclipse assembly scene, inputs the instance segmentation output information into a simulation learning strategy network model, outputs corresponding mechanical arm execution actions according to different lunar eclipse assembly scenes, runs the mechanical arm execution actions, returns to the process of acquiring the lunar eclipse assembly scene, inputs the acquired lunar eclipse assembly scene into the YOLOv8-seg network model for continuous execution, finally acquires the mechanical arm execution actions, and repeats the execution until the alignment process is completed after the lunar eclipse assembly scene cannot be acquired.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot assembly, and particularly relates to a mechanical arm precise shaft hole assembly method based on a lunar eclipse heuristic scene observation simulation. BACKGROUND

[0002] With the rapid development of advanced technologies such as artificial intelligence, robotics, automation technology and intelligent manufacturing, intelligent assembly has attracted widespread attention in the field of production and manufacturing. Existing industrial robots perform assembly tasks mostly in a structured environment, but due to the uncertainty of the working environment and the complexity of the assembly task, human operators are mainly responsible for completing the task, which cannot meet the demand for intelligent assembly and is not universal. Therefore, it is very important to design an intelligent industrial robot precise shaft hole assembly method based on human scene observation. SUMMARY

[0003] According to the problems existing in the prior art, the present application discloses a mechanical arm precise shaft hole assembly method based on lunar eclipse heuristic scene observation simulation, which specifically comprises the following steps:

[0004] A visual sensor mounted at the end of the mechanical arm is used to collect image data of the hole part to be assembled;

[0005] A YOLOv8 model based on deep learning is used to detect and locate the target hole part in the image data, the visual sensor uses a depth image to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and outputs the bounding box, confidence, class and center point three-dimensional coordinates, and controls the mechanical arm to move to the target hole part center point in the three-dimensional space to complete the coarse positioning process and form a lunar eclipse assembly scene;

[0006] The visual sensor collects image data of the hole part to be assembled again, and inputs the lunar eclipse assembly scene into the YOLOv8-seg network model for detection to obtain instance segmentation output information of the lunar eclipse assembly scene, including instance segmentation mask, output bounding box, confidence and lunar eclipse class;

[0007] The instance segmentation output information is input into a simulation learning strategy network model, and according to different lunar eclipse assembly scenes, corresponding mechanical arm execution actions are output;

[0008] The mechanical arm executes the action, returns to the lunar eclipse assembly scene acquisition process, and inputs the acquired lunar eclipse assembly scene into the YOLOv8-seg network model for continuous execution, finally acquires the mechanical arm execution action, and repeats the execution until the lunar eclipse assembly scene cannot be acquired to complete the alignment process.

[0009] Further, the lunar eclipse assembly scene is formed in the following way:

[0010] An RGB image and a depth image of the on-site environment of the hole part to be assembled are collected by an RGB-D camera, wherein the RGB image is input information of a YOLOv8 model, and output information of the model is a 2D bounding box of the hole part to be assembled, pixel coordinates of a top point and a center point of the bounding box, and the depth image is used to map the pixel coordinates in the RGB image to three-dimensional coordinates in a camera coordinate system;

[0011] The three-dimensional coordinates of the center point of the hole part in the RGB image are calculated, and the three-dimensional coordinates of the center point of the hole part to be assembled in the camera coordinate system after target detection are obtained;

[0012] The end effector of the mechanical arm is navigated to a set position and moved downward along the Z axis by a certain distance based on a position control mode, and a force sensor is used to judge the contact state, and if the monitored force exceeds a predetermined threshold, the coarse positioning is completed to form a lunar eclipse assembly scenario.

[0013] Further, the mask in the lunar eclipse assembly scenario is obtained in the following manner: an RGB image in the current state is obtained using an RGB-D camera, and the RGB image is used as input information of a YOLOv8-seg network model, and output information of the YOLOv8-seg network model is a 2D bounding box of the lunar eclipse assembly scenario, a lunar eclipse mask, and a lunar eclipse category;

[0014] Wherein the backbone network of the YOLOv8-seg network model extracts features of the input image and obtains a feature map, and the feature map is converted into a mask through a mask head:

[0015] M = σ(W·F + b)

[0016] Wherein, M is the predicted mask, σ is the activation function, W and b are the weight and bias respectively, and F is the feature map;

[0017] The mask generated by the YOLOv8-seg network model is matched with the real mask, and a binary cross-entropy loss function is used as a mask loss function:

[0018]

[0019] Wherein N is the number of samples in a batch, y i is the true label of the i-th sample, taking values 0 or 1, p i is the probability predicted by the model, i.e. the probability of belonging to the category, taking values in the range 0 ≤ p i ≤ 1.

[0020] Further, according to different lunar eclipse assembly scenarios, the corresponding mechanical arm execution actions are output in the following manner:

[0021] Key information of each assembly action and state of the expert user in the shaft hole assembly task process is acquired, a state and a corresponding action pair are formed, an expert dataset D is constructed, and the expert dataset D contains N demonstration samples, each sample is composed of a state s i and a corresponding action a i :

[0022] The collected expert user assembly data is subjected to data cleaning, data alignment and data normalization processing, and is divided into a training set and a test set;

[0023] A strategy network model of imitation learning is constructed, an expert assembly behavior strategy is learned based on a behavior cloning model, a state s is input into the strategy network model of imitation learning, and a probability distribution of an action a is output:

[0024] π θ (a|s;θ)=P(a|s;θ)

[0025] Wherein θ is a parameter of the model, the state s is subjected to forward propagation in the strategy network model of imitation learning, the input data is subjected to convolution operation through a convolution kernel, local features are extracted; the convolution output is subjected to non-linear mapping through an activation layer, so that the model can learn more complex features; the feature map is subjected to down-sampling through a pooling layer; finally, the extracted local features are mapped to a high-dimensional space through a fully connected layer, global information is integrated, and finally the execution action of the mechanical arm is output.

[0026] Further, the strategy network model of imitation learning compares the action probability distribution generated by forward propagation based on a cross-entropy loss function with the true label of the pre-collected expert user assembly data, measures the difference between the predicted action of the mechanical arm and the true action, and outputs the difference between the predicted action of the mechanical arm and the true action:

[0027]

[0028] Wherein: L(θ) represents the matching degree between the probability predicted by the model and the actual observation data under the given parameter θ, π θ (a i |s i ;θ) represents the probability of selecting an action a i when the state s i is given under the parameter θ.

[0029]

[0030] Wherein, η is a learning rate, is a first moment estimate of bias correction (related to gradient), is a second moment estimate of bias correction, ∈ is a small constant for numerical stability, and the denominator is prevented from being zero;

[0031] The gradient of the loss with respect to the model parameters is calculated by the back propagation algorithm, and the parameters of the imitation learning strategy network model are updated according to the optimizer to maximize the prediction probability of the correct action, thereby minimizing the loss function L(θ) and improving the performance and prediction accuracy of the imitation learning network model;

[0032]

[0033] Wherein, α is a learning rate, and ▽ θ L(θ) is the gradient of the loss function with respect to the parameter θ.

[0034] Iterative training is performed to gradually optimize the parameters of the imitation learning strategy network model, and after each parameter update, it is checked whether the loss function has reached the convergence condition or has reached the preset maximum number of iterations.

[0035] Since the above technical solutions are adopted, the method provided by the application is a mechanical arm precision shaft hole assembly method based on eclipse heuristic scenario observation imitation. The method uses a visual guidance mechanical arm control scheme, realizes the rapid approach connection between shaft holes based on a deep learning target recognition and positioning method, reduces the calibration process, and improves the universality of shaft hole assembly. The eclipse assembly logic is proposed, the YOLOv8-seg network model is used for pixel-level instance segmentation of the eclipse assembly scene, and a mask image is output as the input of the strategy network, which reduces the interference of irrelevant information such as the background environment of the workbench on the assembly, and improves the recognition efficiency and detection accuracy. The imitation learning strategy network model based on imitation learning is constructed, the control strategy is simplified through state-action pairs, and expert assembly is realized. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0037] Figure 1 The flowchart of the method of the present application;

[0038] Figure 2 The shaft hole assembly diagram based on the eclipse scenario inspiration in the present application;

[0039] Figure 3 The shaft hole assembly approach process flowchart based on the eclipse scenario inspiration in the present application

[0040] Figure 4 The imitation learning network model flowchart based on the eclipse scenario inspiration in the present application

[0041] Figure 5 Specific implementation flowchart of mechanical arm shaft hole assembly operation in the application DETAILED DESCRIPTION

[0042] In order to make the technical solutions and advantages of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application:

[0043] As Figure 1 shown, a mechanical arm precision shaft hole assembly method based on a lunar eclipse heuristic scenario observation simulation, specifically includes the following steps:

[0044] S1: In the initial state, the mechanical arm is located at the zero initial position, the mechanical arm end is parallel to the workbench plane, the shaft piece to be assembled is installed at the mechanical arm end position through the adapter flange, the hole piece to be assembled is randomly placed on the workbench plane, the RGB-D camera is fixed near the mechanical arm end through a non-standard adapter and rotated by a certain angle, so that the field of view range covers the entire work plane.

[0045] S2: The mechanical arm uses the YOLOv8 target detection model to identify and locate the hole piece through the RGB-D camera installed near the end, and moves the end effector to the center of the hole piece. The specific process is as shown in Figure 3 .

[0046] S21: The RGB-D camera is used to obtain the RGB image and depth image of the environment, the RGB image is used as the input of the YOLOv8 model, and the 2D bounding box, bounding box vertex and center point pixel coordinates of the hole piece to be assembled are output.

[0047] S22: The depth image is used to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and the three-dimensional coordinates P h =(X h ,Y h ,Z h ) of the hole center point P=(u,v) in the RGB image are calculated and output to the host computer. The specific mapping and conversion calculation process is as follows:

[0048] According to the imaging principle of the RGB-D camera, the camera intrinsic matrix defines the projection process from the 3D world coordinates to the 2D image coordinates:

[0049]

[0050] Where f x , f y is the focal length of the camera; c x , c y is the coordinate of the camera optical center;

[0051] For a given pixel coordinate point P = (u, v), the depth value can be represented as Z = d(u, v),

[0052] Back-project to the camera coordinate system using the camera intrinsic matrix:

[0053]

[0054] To ensure that the field of view of the RGB-D camera can cover the entire assembly work plane of the robot, the camera is tilted by θ towards the direction of the robot base, providing a more comprehensive field of view for the robot. Considering that the camera is tilted by θ around the positive direction of the X-axis, the coordinate system is rotated around the X-axis to obtain the corresponding rotation matrix R x :

[0055]

[0056] Apply the rotation matrix R x to the point (X', Y', Z') in the camera coordinate system to obtain the new coordinates (X, Y, Z):

[0057]

[0058] S23: Through the above formula, the three-dimensional coordinates P h = (X h , Y h , Z h ) of the center point of the target detected hole part in the RGB-D camera coordinate system can be calculated.

[0059] S24: The host computer uses position control to navigate the robot end effector to (X h , Y h ) and move Z h down along the Z-axis, and uses the force sensor to determine the contact state. If the monitored force exceeds the predetermined threshold F Z , the first stage of coarse positioning is considered complete, forming a lunar eclipse assembly scenario, as shown in Figure 2 and Figure 3 .

[0060] S3: After coarse positioning, the robot uses the YOLOv8-seg instance segmentation model to identify and classify the lunar eclipse assembly scenario through the RGB-D camera.

[0061] S31: Use the RGB-D camera to obtain the RGB image under the current state.

[0062] S32: The RGB image is input to the YOLOv8-seg algorithm, which outputs the 2D bounding box, lunar eclipse mask, and lunar eclipse category of the lunar eclipse assembly scenario. The generation of the lunar eclipse mask is the key to this step.

[0063] Specifically, the backbone network of the YOLOv8 network extracts the features of the input image, and after obtaining the feature map, the feature map is converted into a mask through the mask head:

[0064] M = σ(W·F + b)

[0065] where M is the predicted mask, σ is the activation function, W and b are the weights and biases respectively, and F is the feature map.

[0066] The activation function is selected as the sigmoid function, which limits the output value to [0, 1]:

[0067]

[0068] where M ij represents the mask value of the i-th row and j-th column pixel.

[0069] In order to train the model so that the generated mask matches the real mask as much as possible, the binary cross-entropy loss function is used as the mask loss function:

[0070]

[0071] where N is the number of samples in the batch, y i is the true label of the i-th sample, taking values 0 or 1, p i is the probability predicted by the model, i.e. the probability of belonging to that class, taking values in the range 0 ≤ p i ≤ 1.

[0072] S4: input the S3 output state into the imitation learning policy network to select the corresponding expert assembly action. The steps to build the imitation learning policy network are:

[0073] S41: collect expert data: human experts operate to complete multiple robot shaft hole assembly tasks, collect RGB images to obtain key information of each assembly action and state in the process of expert performing shaft hole assembly task, form state and corresponding action pair, build expert dataset D, containing N demonstration samples, each sample is composed of state s i and corresponding action a i :

[0074]

[0075] Build a target learning policy network π θ (a|s; θ), where θ is the parameter of the network, so that for each state s i , the action a i predicted by the model is as close as possible to the expert action.

[0076] S42: Data preprocessing. The collected expert data is subjected to data cleaning, data alignment, data normalization and other operations, and the data set is divided into training set and test set.

[0077] S43: Designing a strategy network for imitation learning, using a behavior cloning model to learn the expert assembly behavior strategy. The strategy network model receives state s as input and outputs the probability distribution of action a:

[0078] π θ (a|s;θ)=P(a|s;θ)

[0079] Specifically, the state s is forward propagated in the imitation learning strategy network model, the input data is convolved by the convolution kernel to extract local features, and the convolution output is nonlinearly mapped by the activation layer to enable the model to learn more complex features. The feature map is down-sampled by the pooling layer; finally, the local features extracted by the fully connected layer are mapped to a high-dimensional space, integrating global information, and finally outputting the execution action of the robot arm.

[0080] S44: Define the loss function. In order to effectively predict the discrete value of the action instruction, the cross-entropy loss function is used, the action probability distribution generated by the forward propagation is compared with the true label of the pre-collected expert operation data, the difference between the predicted action of the robot arm and the true action is measured, and the loss is calculated.

[0081]

[0082] Where: L(θ) represents the matching degree between the probability predicted by the model and the actual observation data under the given parameter θ, π θ (a i |s i ;θ) represents the probability of selecting action a i when the state s i is given under the parameter θ.

[0083]

[0084] Where, η is the learning rate, is the first moment estimate of bias correction (related to the gradient), is the second moment estimate of bias correction (related to the square of the gradient), ∈ is a small constant for numerical stability to prevent the denominator from being zero.

[0085] S45: Parameter update process: In order to minimize the loss function L(θ), the gradient of the loss with respect to the model parameters is calculated by the backpropagation algorithm, and the imitation learning strategy network model parameters are updated according to the optimizer to maximize the prediction probability of the correct action, thereby improving the performance and prediction accuracy of the imitation learning network model.

[0086]

[0087] wherein, alpha is a learning rate, is the gradient of the loss function with respect to the parameter theta. This can make the cross-entropy decrease, and the decision made by the policy network is closer to the action of the expert.

[0088] S46: Loss function convergence checking process: iteratively optimize the imitation learning policy network model parameters, and check whether the loss function has reached the convergence condition or reached the preset maximum iteration number after each parameter update.

[0089] S5: The host computer uses the position control robot end effector to perform the corresponding expert assembly action, and then repeatedly performs S3 to observe the lunar eclipse assembly scene after the action is completed, and takes the scene as the state input of the next S4 to select the next action. One execution of S4 can obtain one assembly action, and the shaft hole assembly task often requires multiple assembly actions, so the intelligent agent needs to execute S4 multiple times to obtain multiple assembly actions to realize the alignment work before the shaft is vertically assembled into the hole.

[0090] Specifically, as shown in Figure 4 When S3 cannot observe the lunar eclipse assembly scene, the assembly state has been approximately aligned, and only a small gap exists between the shaft and the hole. The gap can be eliminated by other mainstream hole searching methods. Then, the robot end effector is given a vertical downward shaft hole insertion instruction to perform the assembly operation task and complete the shaft hole assembly.

[0091] S6: Eliminate the small deviation by the spiral line hole searching method to complete the alignment process and vertically assemble the shaft into the hole. The specific implementation process is shown in Figure 5 .

[0092] In the embodiments of the present application, a neural network based on deep learning is used to process image data in a complex environment, which can better simulate the shaft hole assembly task in an unstructured environment and improve the assembly efficiency of the robot. The innovative shaft hole assembly method of lunar eclipse scene observation imitation learning is proposed, the cylindrical shaft hole assembly scene is inspired by the lunar eclipse scene according to the human-like idea, the real-time lunar eclipse semantic understanding assembly logic is used, the action control robot moving direction and distance are output by the expert strategy network, and the slow problem of the traditional hole searching method is overcome. In summary, the method can make the shaft hole assembly process more flexible, adaptive and robust.

[0093] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for precision shaft and hole assembly of a robotic arm based on lunar eclipse-inspired scenario observation and simulation, characterized in that... include: Image data of the parts to be assembled is acquired using a vision sensor installed at the end of the robotic arm; The YOLOv8 model based on deep learning is used to detect and locate the holes to be assembled in the image data. The vision sensor uses depth image to map the pixel coordinates in the RGB image to the three-dimensional coordinates in the camera coordinate system, and outputs the bounding box, confidence, category and center point three-dimensional coordinates. The robot arm is controlled to move in three-dimensional space to the vicinity of the center point of the target hole to complete the coarse positioning process and form the lunar eclipse assembly scenario. The visual sensor acquires images of the parts to be assembled again and inputs the lunar eclipse assembly scenario into the YOLOv8-seg network model for detection, obtaining instance segmentation output information of the lunar eclipse assembly scenario, including instance segmentation mask, output bounding box, confidence score and lunar eclipse category. The instance segmentation output information is input into the imitation learning policy network model, and the corresponding robotic arm action is output according to different lunar eclipse assembly scenarios. The robotic arm performs actions, returns to the process of acquiring the lunar eclipse assembly scene, and inputs the acquired lunar eclipse assembly scene into the YOLOv8-seg network model to continue execution. Finally, the robotic arm performs actions, and the process is repeated until the lunar eclipse assembly scene can no longer be acquired, after which the alignment process is completed.

2. The method for precision shaft hole assembly of a robotic arm based on lunar eclipse heuristic scenario observation simulation as described in claim 1, characterized in that: The lunar eclipse assembly scenario is formed in the following manner: RGB and depth images of the environment of the hole to be assembled are acquired by an RGB-D camera. The RGB images are used as input information for the YOLOv8 model, and the output information of the model is the 2D bounding box of the hole to be assembled, the pixel coordinates of the bounding box vertices and the center point. The depth image is used to map the pixel coordinates in the RGB images to the three-dimensional coordinates in the camera coordinate system. Calculate the three-dimensional coordinates of the center point of the hole in the RGB image, and at the same time obtain the three-dimensional coordinates of the center point of the hole to be assembled in the RGB-D camera coordinate system after target detection; The robotic arm end effector is guided to a set position and moved a certain distance downward along the Z-axis using a position control method. A force sensor is used to determine the contact state. If the monitored force exceeds a predetermined threshold, the lunar eclipse assembly scenario is roughly completed.

3. The method for precision shaft hole assembly of a robotic arm based on lunar eclipse heuristic scenario observation simulation as described in claim 2, characterized in that: The mask in the lunar eclipse assembly scenario is obtained in the following way: an RGB image in the current state is acquired using an RGB-D camera, and the RGB image is used as the input information of the YOLOv8-seg network model. The output information of the YOLOv8-seg network model is the 2D bounding box of the lunar eclipse assembly scenario, the lunar eclipse mask, and the lunar eclipse category. The backbone network of the YOLOv8-seg network model extracts features from the input image and obtains a feature map, which is then converted into a mask using a masking mechanism. M = σ(W·F + b) Where M is the predicted mask, σ is the activation function, W and b are the weights and biases respectively, and F is the feature map; To control the matching of the mask generated by the YOLOv8-seg network model with the real mask, the binary cross-entropy loss function is used as the mask loss function: Where N is the number of samples in the batch, y i p is the true label of the i-th sample, taking the value 0 or 1. i This is the probability predicted by the model, i.e., the probability of belonging to this category, with a value range of 0 ≤ p. i ≤1.

4. The method for precision shaft hole assembly of a robotic arm based on lunar eclipse heuristic scenario observation simulation as described in claim 1, characterized in that: Depending on the different lunar eclipse assembly scenarios, the corresponding robotic arm actions are output in the following manner: Obtain key information on each assembly action and state during the process of expert users performing shaft and hole assembly tasks, form state-action pairs, construct an expert dataset D containing N demonstration samples, each sample consisting of state s i and the corresponding action a i composition: The collected assembly data from expert users was cleaned, aligned, and normalized, and then divided into training and testing sets. Construct an imitation learning policy network model, assemble a behavior policy based on a behavior cloning model and an imitation learning expert, input the state s into the imitation learning policy network model, and thus output the probability distribution of action a: p θ (a|s;θ)=P(a|s;θ) Here, θ represents the model parameters. The state s is propagated forward in the imitation learning policy network model. The input data is convolved by a convolution kernel to extract local features. The convolution output is then nonlinearly mapped by an activation layer, enabling the model to learn more complex features. The feature map is downsampled by a pooling layer. Finally, the extracted local features are mapped to a high-dimensional space by a fully connected layer, integrating global information and ultimately outputting the robotic arm's execution action.

5. The method for precision shaft hole assembly of a robotic arm based on lunar eclipse heuristic scenario observation simulation as described in claim 4, characterized in that: The imitation learning strategy network model compares the action probability distribution generated by forward propagation based on the cross-entropy loss function with the real labels of pre-collected expert user assembly data to measure the difference between the robot arm's predicted actions and the actual actions: Where: L(θ) represents the degree of matching between the probability predicted by the model and the actual observed data, given the parameter θ, π θ (a i |s i ;θ) represents the state s given by parameter θ. i When selecting action a i The probability of; Where η is the learning rate. It is a first-moment estimate with bias correction (related to the gradient). It is the second-order moment estimate with bias correction, where ∈ is a small constant used for numerical stability to prevent the denominator from being zero; The gradient of the loss with respect to the model parameters is calculated by backpropagation algorithm. The parameters of the imitation learning policy network model are updated according to the optimizer to maximize the prediction probability of the correct action, thereby minimizing the loss function L(θ) and improving the performance and prediction accuracy of the imitation learning network model. Where α is the learning rate. It is the gradient of the loss function with respect to the parameter θ; The parameters of the imitation learning strategy network model are gradually optimized through iterative training. After each parameter update, the loss function is checked to see if it has reached the convergence condition or the preset maximum number of iterations.

Images