A mechanical arm screw locking method and system based on non-structured multi-level positioning of screw holes

Abstract

The application belongs to the technical field of mechanical arm assembly, and provides a mechanical arm screw locking method and system based on multi-stage positioning of unstructured screw holes, comprising: acquiring a target image to be locked; performing multi-scale perception fusion on the target image based on different expansion rates to extract the edge morphology of the screw hole in the target image; obtaining a screw hole global edge prediction map according to the extracted screw hole edge morphology and a multi-layer supervision mechanism; performing screw hole center positioning based on the obtained screw hole global edge prediction map to obtain the real attitude of the screw hole; performing screw hole center positioning based on the obtained screw hole global edge prediction map to obtain the position information of the screw hole and the error-containing coarse attitude information; correcting the error-containing coarse attitude information in the attitude alignment network to determine the screw hole attitude signal, and the mechanical arm moves to the screw hole directly above according to the position information and the attitude information of the screw hole to lock the screw.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of mechanical arm assembly, and particularly relates to a mechanical arm screw locking method and system based on multi-level positioning of unstructured screw holes. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of industrial technology, the application of mechanical arms in industrial automation has gradually become a key component. Industrial robots not only can complete repetitive work, but also perform well in handling complex tasks and improving production efficiency; among them, the background of mechanical arms locking screws in unstructured scenes is an important direction of current research.

[0004] In the traditional structured environment, the mechanical arm can usually lock screws through a predefined path and fixed conventional workpieces. However, in unstructured scenes, the working environment may contain various randomly placed workpieces and non-standardized screw holes (screw holes are inclined at any angle), which brings additional challenges for the mechanical arm to perform screw locking operations. SUMMARY

[0005] To solve the above problems, the present application provides a mechanical arm screw locking method and system based on multi-level positioning of unstructured screw holes, which is suitable for unstructured scenes, captures the edge shape of the screw hole in the target image based on edge detection and multi-scale fusion technology, optimizes the detection of the screw hole edge through a multi-layer supervision mechanism, and obtains a global edge prediction map of the screw hole; according to the global edge prediction map, the center point and the coarse pose of the screw hole are obtained, and the real pose of the screw hole is further obtained; the pose of the mechanical arm is determined in combination with the real pose of the screw hole, and the mechanical arm screw locking based on multi-level positioning of unstructured screw holes is completed according to the pose of the mechanical arm, improving the adaptability and operation precision of the mechanical arm in dynamic environments.

[0006] According to some embodiments, the first aspect of the present application provides a mechanical arm screw locking method based on multi-level positioning of unstructured screw holes, which adopts the following technical solution:

[0007] A mechanical arm screw locking method based on multi-level positioning of unstructured screw holes, comprising:

[0008] Obtaining a target image to be locked;

[0009] Performing multi-scale perception fusion on the target image based on different inflation rates to extract the edge shape of the screw hole in the target image;

[0010] According to the extracted screw hole edge shape and the multi-layer supervision mechanism, a global edge prediction map of the screw hole is obtained;

[0011] Based on the obtained screw hole global edge prediction map, the screw hole center is positioned, and the real attitude of the screw hole is obtained;

[0012] Based on the obtained screw hole global edge prediction map, the screw hole center is positioned, and the position information of the screw hole and the coarse attitude information containing errors are obtained;

[0013] The coarse attitude information containing errors is corrected in the attitude alignment network to determine the screw hole attitude signal, and the mechanical arm moves to the screw hole directly above according to the position information and the attitude information of the screw hole to perform screw locking.

[0014] As a further technical limitation, in the process of obtaining the target image to be locked, an original image is collected, a screw hole region anchor frame is identified in the obtained original image, a screw locking region of interest (i.e. ROI region) is extracted in the identified screw hole region anchor frame, and the target image to be locked is obtained.

[0015] As a further technical limitation, a convolutional neural network is used to perform multi-layer convolution operation on the obtained target image to be locked, and image features of the target image to be locked are extracted layer by layer. The multi-scale perception fusion is performed on the extracted multi-layer image features through different inflation rates, and the screw hole edge morphology of the target image is obtained. The edge information of the obtained screw hole edge morphology is learned according to the multi-layer supervision mechanism, the edge detection result of each layer is obtained, and the obtained edge detection result of each layer is weighted and averaged to obtain the screw hole global edge prediction map.

[0016] As a further technical limitation, in the process of screw hole center positioning, the obtained screw hole global edge prediction map is converted into a three-dimensional space to obtain a screw hole global edge three-dimensional prediction map. The screw hole global edge three-dimensional prediction map is denoised by a geometric relationship filtering method, non-edge points are removed, and the minimum error circle fitting method is applied to the edge points for iterative optimization to obtain the optimal screw hole center position. The screw hole center point position information and the coarse attitude information containing errors are obtained.

[0017] As a further technical limitation, based on the coarse position of the screw hole, the pose alignment network is applied according to the RGB image obtained by the camera at the end of the current mechanical arm to predict the correction direction of the next current position and the real position of the screw hole. According to the step, the alignment error between the mechanical arm and the screw hole is corrected, and the correction is iterated in the X and Y directions respectively, and finally corrected to within a reasonable error to complete the correct locking.

[0018] According to some embodiments, the second aspect of the present application provides a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, which adopts the following technical scheme:

[0019] A mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, comprising:

[0020] an image acquisition module configured to acquire a target image to be fastened;

[0021] an edge extraction module configured to perform multi-scale perception fusion on the target image based on different inflation rates, extract a screw hole edge shape in the target image, and obtain a screw hole global edge prediction map according to the extracted screw hole edge shape and a multi-layer supervision mechanism;

[0022] a coarse positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map, and obtain a screw hole real attitude;

[0023] a fine positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map, and obtain position information of the screw hole and error-containing coarse attitude information;

[0024] a fastening module configured to correct the error-containing coarse attitude information in an attitude alignment network to determine a screw hole attitude signal, and move a mechanical arm to directly above the screw hole according to the position information and the attitude information of the screw hole to perform screw fastening.

[0025] According to some embodiments, a third aspect of the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0026] A computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the steps in the method for screw fastening of a mechanical arm based on multi-level positioning of unstructured screw holes according to the first aspect of the present application.

[0027] According to some embodiments, a fourth aspect of the present application provides an electronic device, which adopts the following technical solution:

[0028] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, and the processor implements the steps in the method for screw fastening of a mechanical arm based on multi-level positioning of unstructured screw holes according to the first aspect of the present application when executing the program.

[0029] According to some embodiments, a fifth aspect of the present application provides a computer program product, which adopts the following technical solution:

[0030] A computer program product includes software code, and the program in the software code performs the steps in the method for screw fastening of a mechanical arm based on multi-level positioning of unstructured screw holes according to the first aspect of the present application.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The application captures the edge shape of the screw hole in the target image based on edge detection and multi-scale fusion technology, optimizes the detection of the screw hole edge through a multi-layer supervision mechanism, obtains a screw hole global edge prediction map, acquires the center point and coarse pose of the screw hole according to the global edge prediction map, applies a pose alignment network on the basis of the coarse pose of the screw hole, predicts the correction direction of the current pose and the real pose of the screw hole in the next step, corrects the alignment error between the mechanical arm and the screw hole according to the step length, and finally corrects to the accurate pose to complete the locking, thereby improving the adaptability and operation precision of the mechanical arm in the dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the embodiments of the present application and are incorporated in and constitute a part of this specification. The embodiments of these drawings are set forth to explain the embodiments of the present application and are not meant to limit the present application.

[0034] Figure 1 The figure is a schematic diagram of the mechanical arm screw locking method based on multi-level positioning of unstructured screw holes in the first embodiment of the present application.

[0035] Figure 2 The figure is a schematic diagram of the screw hole edge detection in the first embodiment of the present application.

[0036] Figure 3 The figure is a schematic diagram of the mechanical arm pose prediction in the first embodiment of the present application.

[0037] Figure 4 The figure is a structural block diagram of the mechanical arm screw locking system based on multi-level positioning of unstructured screw holes in the second embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application will be further described below in conjunction with the drawings and embodiments.

[0039] It should be noted that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0040] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0041] In the present application, the terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only a relationship word determined for the convenience of describing the structural relationship of the components or elements of the present application, and cannot be understood as a limitation on the present application.

[0042] In the present application, the terms such as "fixedly connected", "connected", "connected" and the like should be understood broadly, which means that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific researchers or technicians in the art, the specific meaning of the above terms in the present application can be determined according to the specific circumstances, and cannot be understood as a limitation on the present application.

[0043] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0044] It should be noted that the unstructured scene is a data scene that does not conform to the table mode of the traditional relational database; the data structure in such a data scene is irregular or incomplete, there is no predefined data model, and it is not convenient to use a two-dimensional logical table of a database to represent it. The unstructured scene database can flexibly process and query these unstructured data and provide efficient storage and retrieval functions. Due to the diversity of unstructured data, such a database system usually has the characteristics of high flexibility (able to process various formats of files and data types), efficient storage (able to quickly store and retrieve a large amount of data) and convenient query (provides efficient query function to meet the demand of fast response).

[0045] The unstructured scene in the present application can be understood as a dynamic non-stationary, non-horizontal and non-vertical position, or a narrow and irregular position.

[0046] Embodiment one

[0047] The embodiment one of the present application introduces a mechanical arm screw locking method based on multi-level positioning of unstructured screw holes.

[0048] The embodiment aims at automatically performing screw locking task by mechanical arm in unstructured scene, and proposes an intelligent solution to cope with the increasing complex needs and challenges in industrial automation, and improve the adaptability and operation precision of mechanical arm in dynamic environment. Firstly, the embodiment introduces ShedNet network to perform high-precision edge detection on the region of interest, effectively extracts the edge feature points of screw hole based on deep learning technology, and ensures accurate identification of the target region under complex background or shielding conditions. Secondly, the embodiment obtains the center position of the screw hole based on the projection hole center positioning algorithm, fully utilizes the spatial coordinates of the screw hole edge points, and combines the geometric calculation method to determine the center of the screw hole within a very small error range. In order to ensure the optimal motion path of the mechanical arm in the locking task, the embodiment introduces AcNet network to intelligently predict the next pose of the mechanical arm according to the RGB feedback information in the dynamic scene by real-time processing of the environment, iteratively optimize the pose and motion trajectory of the mechanical arm, ensure its high precision and high efficiency, and improve the adaptability of the mechanical arm in complex unstructured environment, so that it can flexibly cope with various unstructured scenes.

[0049] As shown in the mechanical arm screw locking method based on multi-level positioning of unstructured screw hole, the method comprises the following steps: Figure 1

[0050] A, target detection and ROI extraction;

[0051] B, image enhancement processing;

[0052] C, high-precision edge detection;

[0053] D, screw hole center positioning;

[0054] E, mechanical arm pose prediction.

[0055] As one or more embodiments, the embodiment collects several high-quality images, and accurately identifies the anchor frame of the screw hole region by manual annotation. In order to optimize the target detection effect, the embodiment uses K-means clustering algorithm to calculate effective anchor frame. Through clustering analysis of these labeled data, the anchor frame configuration suitable for screw hole size distribution is generated, which significantly improves the target detection performance.

[0056] In the embodiment, YOLOV5 algorithm can be used to extract ROI in the identified anchor frame to lay the foundation for edge detection. In other embodiments, other algorithms are also used to extract the position region.

[0057] ​The data required for screw hole edge detection in this embodiment can be obtained by acquiring images and depth information with a depth camera at the end of the mechanical arm; to ensure the diversity of the data set, the embodiment can set the configuration of the mechanical arm and the depth camera in different experimental environments; by configuring the depth camera to capture screw hole images in a real scene while ensuring the flexibility of the mechanical arm, the screw hole is photographed at different angles and positions; wherein, under different lighting conditions, multiple photographs can be taken based on natural light and artificial lighting to obtain screw hole images under various lighting conditions; at different angles, the screw hole can be photographed from multiple perspectives to capture its different appearance characteristics; in various backgrounds, different backgrounds can be selected to improve the richness of the image and ensure that the model can adapt to various application scenarios.

[0058] During the collection of images, each image is acquired by a depth camera controlled by a mechanical arm, ensuring the quality and stability of the image. In each image, the edge points of the screw hole are generated by clicking the mouse. Using the obtained edge points, an elliptical shape can be fitted by a suitable algorithm, thereby generating a corresponding binary mask, which accurately represents the contour of the screw hole edge.

[0059] To prevent overfitting of the model, data augmentation is introduced in the data set in this embodiment, i.e. rotation (random angle rotation of the image to improve the robustness of the model to screw holes in different directions), scaling (random adjustment of image size to simulate screw hole images at different distances), color transformation (color transformation of the image, including random adjustment of brightness, contrast and saturation, to enhance the model's ability to adapt to changes in lighting) and symmetry transformation (horizontal and vertical flipping of the image to increase the diversity of the data set).

[0060] It should be noted that the image enhancement processing in this embodiment belongs to the prior art and can be understood as prior art known to those skilled in the art, which will not be described here in detail.

[0061] As one or more implementations, the embodiment adopts a screw hole edge detection network ShedNet based on a full convolutional neural network as shown in Figure 2 to realize edge detection of screw hole images. Through the feature extraction module and the multi-scale feature fusion module, the model learns to aggregate feature information under different receptive fields, and the multi-layer supervision module is introduced to enable the model to better capture subtle and complex edge structures.

[0062] In edge detection, different levels of features of the image are important for capturing edges. The embodiment adopts a multi-layer convolution module as a feature extraction network (i.e. Conv Module); given an input image Wherein, H and W represent the height and width of the image respectively, the plurality of feature extraction modules perform multi-layer convolution operations on the input image or the feature map, and gradually extract different levels of feature representations.

[0063] The extracted feature layer is F i = f i (X), i = 1, 2, 3; wherein f i (X) represents the feature mapping of the i-th convolution module (i.e. from conv_1 to conv_3). For each convolution layer, the dimension of the feature map gradually decreases with the deepening of the level, and the features extracted by the previous layers are biased towards details, while the high-level features gradually become more global. To adapt to the task of detecting the edge of the screw hole, three convolution modules are used in this embodiment, thereby reducing the amount of calculation and enhancing the ability to capture small edges; at the same time, during the feature extraction process, reflection padding is used to avoid boundary effects, which makes the original information of the image edge more complete in the convolution operation.

[0064] The screw hole edge usually has different scales and complexities, and a single scale of convolution cannot fully capture all possible edge patterns. This embodiment realizes feature fusion by multi-scale perception of features through different dilation rates, thereby improving the edge detection capability.

[0065] The core of the multi-scale fusion module is the application of dilated convolution. Dilated convolution inserts holes between convolution kernels, increasing the receptive field while keeping the convolution kernel parameters unchanged. Given an input feature map, the multi-scale fusion module extracts multi-scale information through a series of convolution operations with dilation rates, i.e. Wherein, F (l) is the input feature map of the l-th layer, * represents the convolution operation, is the convolution kernel weight with dilation rate d k , and K is the number of different dilation rates. This embodiment sets three dilation rates d1 = 4, d2 = 8, and d3 = 12, respectively, to capture local and global features of the screw hole edge at different scales.

[0066] The features output by the multi-scale convolution are fused with the initial input features through residual connection to generate multi-scale fusion features i.e. The fused feature map is reduced in channel number by 1x1 convolution and obtains the final output as the edge detection final feature map at the current level.

[0067] To ensure that edge information can be effectively learned on different levels of feature maps, the embodiment introduces a multi-layer supervision mechanism; by adding independent supervision signals after each convolutional layer, the network can not only learn edge information on high-level features, but also optimize edge detection capability on low-level and middle-level feature maps.

[0068] Given the input feature map F i , which is the feature information extracted by the feature extraction network and the multi-scale fusion module at the layer, upsample each level of feature map to make the resolution consistent with the input image.

[0069] Upsampling is achieved through a deconvolutional layer (ConvTranspose2d), and the specific upsampling rate depends on the reduction ratio of the feature map. For example, the third layer feature map needs to be upsampled by 8 times, i.e. By concatenating all the upsampled prediction results and further fusing them through a 1x1 convolution, the final edge detection result is obtained, i.e.

[0070] The embodiment uses three main supervision branches, corresponding to different levels of convolutional features, i.e. i = σ(W i *F i ); where W i is the convolution kernel of the supervision branch, σ represents the Sigmoid activation function, and P i is the edge prediction of the i-th layer.

[0071] For each supervision branch, calculate the binary cross-entropy loss between the edge prediction P i of this level and the true edge label Y, where N represents the number of pixels, Y n represents the label of the nth pixel, and P i,n is the prediction value of the nth pixel; the total loss is the weighted sum of the supervision loss of each layer, i.e. where α i is the weight of each level supervision branch, and the edge detection results of different levels are fused by weighted average to generate a global edge prediction map.

[0072] As one or more embodiments, to accurately solve the screw hole center position and the coarse pose (i.e. inaccurate pose), the embodiment realizes screw hole center positioning based on RGB image and depth camera coordinate transformation, converts the image edge detection result to three-dimensional space, filters the detected point cloud data, and solves the screw hole center and its normal vector through fitting and iteration.

[0073] After the edge detection is completed, the detected image coordinate points are converted from the RGB camera coordinate system to the world coordinate system through the coordinate transformation of the depth camera. In this process, the camera intrinsic parameters and extrinsic parameters are known, and the pixel coordinates (u, v) on the image plane can be converted into the point P (x, y, z) in the three-dimensional world coordinate system through the standard inverse perspective projection transformation; the transformation formula is wherein Pw is the point in the world coordinate system, is the transformation matrix from the camera to the world coordinate system, K is the camera intrinsic parameter matrix, and D (u, v) is the depth value of the depth map at the corresponding pixel point.

[0074] In order to eliminate the noise points in the detection, the embodiment adopts a filtering algorithm based on geometric relationship. Since the depth of the hole edge changes continuously, the distance between adjacent points is used as the basis to filter the noise points; that is, the center point of the edge point is taken as the reference point Pref, the polar angle θ of all edge points with the reference point is calculated i = arctan2 (p i,y -P ref,y , p i,x -P ref,x ); after the polar angle of each edge point is calculated, the edge points are sorted according to the polar angle; the distance between adjacent points after sorting is calculated, and the minimum distance dmin is found as the threshold value for subsequent filtering; the parameter k is set as the multiple of the distance threshold value, and any point pair with a distance between adjacent points exceeding kxdmin will be regarded as a noise point and removed; the valid point set Pvalid after filtering will be used for subsequent fitting of the hole center; the non-edge points introduced in the edge detection process due to image noise or detection errors can be effectively removed.

[0075] In order to accurately solve the three-dimensional center of the hole, a circle fitting method based on error minimization is adopted, and the best hole center position is found by iteratively optimizing the normal vector. It is assumed that the normal vector of the hole is initially n0 = [0, 0, 1], and the normal vector is rotated and transformed through the Euler angle rotation matrix R (θ x , θ y , θ z ); the rotated normal vector n is n = R (θ x , θ y , 0) · n0; wherein θ x , θ y are the rotation angles around the X axis and the Y axis (the X axis and the Y axis at this time are the coordinate axes of the robot arm), which are initially set to vary within the range of [-30°, 30°] with a step of 5°; for each rotation angle (θ x , θ y ), the current normal vector n is used for circle fitting. The goal of fitting the hole edge points is to minimize the fitting error, and it is assumed that the parameters of the circle are (x c , y cr) (i.e. the center coordinates and radius), the error is defined as the sum of squared geometric distances of the edge points to the fitted circle, i.e.

[0076] The embodiment finds the normal vector n and the center coordinates (x c , y c ) that minimize the fitting error E by traversing different rotation angles. Under all rotation angles, the normal vector n best and the corresponding screw hole center (x c , y c ) that minimize the error are found. The normal vector n best of the screw hole is the optimal solution, representing the direction of the normal vector of the screw hole. After obtaining the optimal normal vector n best and the center (x c , y c ), the center position of the screw hole is solved by a straight line equation. Given the edge point set P of the screw hole and the normal vector n best , the center P c of the screw hole satisfies P c = P f + t·n best ; where P f is the center of the fitted circle, and t is the displacement along the normal vector direction. By minimizing the sum of squared distances of the edge points to the center point, the displacement t can be solved; specifically: the objective function D is the sum of squared perpendicular distances of all edge points to the center point, i.e. Taking the derivative of the displacement t and minimizing it, the optimal screw hole center Pc is finally solved, i.e. The accurate center position of the screw hole in three-dimensional space and the direction of the normal vector (with error) are obtained.

[0077] As one or more implementations, the embodiment adopts a mechanical arm pose prediction architecture as shown in Figure 3 , designs a screw hole pose alignment network based on a CNN classifier to complete the pose prediction of the mechanical arm, and the network predicts the alignment deviation of the mechanical arm in the X / Y direction according to the RGB image obtained by the camera fixed at the end of the mechanical arm; by continuously predicting the classification results in the X direction and the Y direction, the network can guide the mechanical arm to gradually adjust the angle in the correct direction, realizing the accurate alignment of the mechanical arm and the screw hole.

[0078] The embodiment models the angle correction problem as a classification problem, predicts the angle deviation of the current mechanical arm relative to the screw hole axis, and outputs the corresponding classification result to indicate the correction operation; for each direction (X direction and Y direction), the classification is divided into three categories, i.e.:

[0079] Category 0: The current mechanical arm angle is already in the correct position and does not need to be adjusted.

[0080] Category 1: The current angle deviation is positive, and needs to be rotated by 5° in the positive direction for correction.

[0081] Category -1: The current angle deviation is negative, and needs to be rotated by 5° in the negative direction for correction.

[0082] In this embodiment, the input image is processed through multiple convolutional layers and pooling layers to extract image features of different scales and levels. The extracted feature map is used for subsequent classification tasks. Assuming that the input image is I, the extracted feature map is denoted as F = f(I), where f(·) represents the feature extraction process; for the features in the X direction / Y direction, a classifier module is designed to predict the angle deviation in the corresponding direction. Assuming that the feature in the X direction is represented as F x , and the feature in the Y direction is represented as F y , the classification task can be represented as wherein and represent the predicted classification results in the X direction and the Y direction, respectively, P x and P y are the classification probability distributions, and the network outputs the probability of each category through the Softmax function.

[0083] The loss function of the network is composed of a classification loss and a regularization term. Taking the X direction pose alignment network loss function as an example, it is represented as L = a·L x + λ·R(W), L x is the cross-entropy loss, corresponding to the X direction classification task, represented as where y x,i is the one-hot encoding of the true label, and p x,i is the predicted probability of the corresponding category. R(W) is the L2 regularization term of the network weights, used to prevent overfitting. The weight factors a and λ are adjusted through experiments, and finally set to a = 1.0 and λ = 0.0001. The Adam optimizer is selected, with an initial learning rate of 0.001, and a cosine annealing strategy is used to gradually reduce the learning rate, thereby accelerating the convergence. Through data augmentation methods such as random rotation, translation, and noise superposition, the diversity of training samples is increased, and the generalization ability of the network is improved.

[0084] In addition, to further enhance the feature expression capability, a skip connection is introduced in the feature extraction module to fuse the detailed information of the shallow layer and the semantic information of the deep layer, improving the ability to capture angle deviation features. Channel attention mechanism is introduced on the feature map F to automatically assign weights to different channels to highlight key features related to angle deviation.

[0085] The robot arm will perform the corresponding rotation compensation according to the classification results output by the network. Assuming that the predicted result in the X direction is The prediction result in the Y direction is The rotation compensation amount of the mechanical arm is

[0086] When or , it indicates that the angle of the mechanical arm in the direction is correct and does not need to be adjusted; when or , the mechanical arm needs to rotate 5° in the positive direction; when or , the mechanical arm needs to rotate 5° in the negative direction. Through angle adjustment, the mechanical arm can quickly correct the alignment error with the screw hole within a limited step and realize precise assembly.

[0087] In this embodiment, 400 images are collected within the range of [-30°, 30°] in the x direction and [-30°, 30°] in the y direction, and the collection step is 3°, that is, the x direction angle change is [-5°, 5°] for category 0, and the y direction angle change is [-5°, 5°] for category 0, the x direction angle change is [5°, 30°] for category 1, and the rest are category -1, and the y direction is the same. Through 50 experiments, the screw hole angle offset changes within the range of [-30°, 30°], and 46 times are successful, with a success rate of 92%.

[0088] This embodiment captures the screw hole edge morphology in the target image based on edge detection and multi-scale fusion technology, optimizes the detection of the screw hole edge through a multi-layer supervision mechanism, obtains a screw hole global edge prediction map, acquires the center point and rough pose of the screw hole according to the global edge prediction map, and further obtains the real pose of the screw hole. The real pose of the screw hole is combined to determine the pose of the mechanical arm, and the mechanical arm screw locking based on the multi-level positioning of the unstructured screw hole is completed according to the pose of the mechanical arm, thereby improving the adaptability and operation precision of the mechanical arm in a dynamic environment.

[0089] Embodiment two

[0090] The embodiment two of the present application introduces a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes.

[0091] As shown in Figure 4 a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, comprising:

[0092] An image acquisition module configured to acquire a target image to be locked;

[0093] An edge extraction module configured to perform multi-scale perception fusion on the target image based on different expansion rates to extract the screw hole edge morphology in the target image; and obtain a screw hole global edge prediction map according to the extracted screw hole edge morphology and a multi-layer supervision mechanism.

[0094] a coarse positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain a screw hole real pose;

[0095] a fine positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain position information of the screw hole and coarse pose information containing errors;

[0096] a locking module configured to correct the coarse pose information containing errors in the pose alignment network to determine a screw hole pose signal, and the robot arm moves to the screw hole directly above according to the position information and the pose information of the screw hole to lock the screw.

[0097] The detailed steps are the same as those of the robot screw locking method based on multi-level positioning of unstructured screw holes provided in Embodiment 1, and will not be repeated here.

[0098] Embodiment 3

[0099] The embodiment 3 of the present application provides a computer readable storage medium.

[0100] A computer readable storage medium has a program stored thereon, and the program is executed by a processor to implement the steps in the robot screw locking method based on multi-level positioning of unstructured screw holes according to the embodiment 1 of the present application.

[0101] The detailed steps are the same as those of the robot screw locking method based on multi-level positioning of unstructured screw holes provided in Embodiment 1, and will not be repeated here.

[0102] Embodiment 4

[0103] The embodiment 4 of the present application provides an electronic device.

[0104] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor, and the processor executes the program to implement the steps in the robot screw locking method based on multi-level positioning of unstructured screw holes according to the embodiment 1 of the present application.

[0105] The detailed steps are the same as those of the robot screw locking method based on multi-level positioning of unstructured screw holes provided in Embodiment 1, and will not be repeated here.

[0106] Embodiment 5

[0107] The embodiment 5 of the present application provides a computer program product.

[0108] A computer program product includes software code, and a program in the software code performs the steps in the robot screw locking method based on multi-level positioning of unstructured screw holes according to the embodiment 1 of the present application.

[0109] The detailed steps are the same as those of the mechanical arm screw locking method based on unstructured screw hole multi-stage positioning provided in Embodiment One, and will not be described herein again.

[0110] The above merely provides the preferred embodiments of the present embodiment, and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A mechanical arm screw locking method based on multi-stage positioning of unstructured screw holes, characterized in that, The method comprises the following steps: obtaining a target image to be fastened; performing multi-scale perception fusion on the target image based on different expansion rates to extract the edge shape of the screw hole in the target image; obtaining a screw hole global edge prediction map according to the extracted edge shape of the screw hole and a multi-layer supervision mechanism; performing screw hole center positioning based on the obtained screw hole global edge prediction map to obtain a real pose of the screw hole; performing screw hole center positioning based on the obtained screw hole global edge prediction map to obtain position information of the screw hole and error-containing coarse pose information; correcting the error-containing coarse pose information in a pose alignment network to determine a screw hole pose signal, and moving a mechanical arm to directly above the screw hole according to the position information and the pose information of the screw hole to fasten the screw; the multi-layer supervision mechanism adds independent supervision signals after each convolution layer; a convolutional neural network is used to perform multi-layer convolution operation on the obtained target image to be fastened, to extract image features of the target image to be fastened layer by layer, to perform multi-scale perception fusion on the extracted multi-layer image features based on different expansion rates, to obtain the edge shape of the screw hole in the target image, to learn edge information of the obtained edge shape of the screw hole according to the multi-layer supervision mechanism, to obtain edge detection results of each layer, and to obtain a screw hole global edge prediction map by weighted average fusion of the obtained edge detection results of each layer; in the process of screw hole center positioning, the obtained screw hole global edge prediction map is converted into a three-dimensional space to obtain a screw hole global edge three-dimensional prediction map, the screw hole global edge three-dimensional prediction map is denoised by a geometric relationship filtering method to remove non-edge points, and the denoised screw hole global edge three-dimensional prediction map and a minimum error circle fitting method are used for iterative optimization to obtain an optimal screw hole center position and a real pose of the screw hole; a screw hole axis position is determined according to the obtained real pose of the screw hole, the angle and position of the mechanical arm are adjusted according to the angle deviation between the current mechanical arm and the screw hole axis until the mechanical arm is aligned with the screw hole to determine the pose of the mechanical arm; the moving direction of the mechanical arm on different directional shafts is adjusted according to the positive and negative of the angle deviation between the current mechanical arm and the screw hole axis to correct the alignment error between the mechanical arm and the screw hole, and the angle and position of the mechanical arm are adjusted through rotation of the mechanical arm.

2. The multi-stage positioning based robotic screw driving method of claim 1, wherein, In the process of obtaining the target image to be fastened, an original image is collected, a screw hole region anchor box is marked in the obtained original image, a screw fastening region of interest is extracted in the marked screw hole region anchor box, and the target image to be fastened is obtained.

3. A mechanical arm screw locking system based on multi-stage positioning of unstructured screw holes, adopting the mechanical arm screw locking method based on multi-stage positioning of unstructured screw holes according to any one of claims 1-2, characterized in that, The method comprises the following steps: an image acquisition module configured to obtain a target image to be fastened; an edge extraction module configured to perform multi-scale perception fusion on the target image based on different expansion rates to extract the edge shape of the screw hole in the target image, and to obtain a screw hole global edge prediction map according to the extracted edge shape of the screw hole and a multi-layer supervision mechanism; a coarse positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain a real pose of the screw hole; a fine positioning module configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain position information of the screw hole and error-containing coarse pose information; A locking module configured to correct the error-containing coarse pose information in the pose alignment network to determine a screw hole pose signal, and the robot arm moves to the screw hole directly above according to the position information and the pose information of the screw hole to lock the screw.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method for robot screw locking based on multi-stage positioning of unstructured screw holes according to any one of claims 1-2.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the program to implement the steps of the method for robot screw locking based on multi-stage positioning of unstructured screw holes according to any one of claims 1-2.

6. A computer program product comprising software code, characterized in that, The program in the software code executes the steps of the method for robot screw locking based on multi-stage positioning of unstructured screw holes according to any one of claims 1-2.

Images