Mechanical arm screw locking method and system based on unstructured screw hole multistage positioning

Abstract

The invention belongs to the technical field of mechanical arm assembly, and provides a mechanical arm screw locking method and system based on unstructured screw hole multistage positioning, and the method comprises the steps: obtaining a to-be-locked target image; performing multi-scale perception fusion on the target image based on different expansion rates, and extracting a screw hole edge form in the target image; obtaining a screw hole global edge prediction map according to the extracted screw hole edge form and a multi-layer supervision mechanism; positioning the center of the screw hole based on the obtained global edge prediction map of the screw hole to obtain a real posture of the screw hole; positioning the center of the screw hole based on the obtained global edge prediction map of the screw hole to obtain position information of the screw hole and coarse attitude information containing errors; and correcting coarse attitude information containing errors in the attitude alignment network to determine a screw hole attitude signal, and enabling the mechanical arm to move to a position right above the screw hole according to the position information and the attitude information of the screw hole for screw locking.

Description

Technical Field

[0001] The invention belongs to the technical field of robot arm assembly, and in particular relates to a robot arm screw locking method and system based on multi-level positioning of unstructured screw holes. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of industrial technology, the application of robotic arms in industrial automation has gradually become a key component. Industrial robots can not only complete repetitive tasks, but also perform well in handling complex tasks and improving production efficiency; among them, the background of robotic arms tightening screws in unstructured scenes is an important direction of current research.

[0004] In traditional structured environments, robotic arms can usually perform screw tightening through predefined paths and fixed conventional workpieces. However, in unstructured scenarios, the working environment may contain a variety of randomly placed workpieces and non-standard screw holes (screw holes tilted at any angle), which brings additional challenges to the robotic arm to perform operations such as screw tightening. Summary of the invention

[0005] To solve the above problems, the present invention proposes a method and system for screw locking with a robotic arm based on multi-level positioning of unstructured screw holes, which is suitable for unstructured scenes. The edge morphology of the screw holes in the target image is captured based on edge detection and multi-scale fusion technology, and the detection and optimization of the edges of the screw holes are performed through a multi-layer supervision mechanism to obtain a global edge prediction map of the screw holes; the center point and the rough posture of the screw holes are obtained according to the global edge prediction map, and the real posture of the screw holes is further obtained; the posture of the robotic arm is determined in combination with the real posture of the screw holes, and the screw locking with the robotic arm based on multi-level positioning of the unstructured screw holes is completed according to the posture of the robotic arm, thereby improving the adaptability and operation accuracy of the robotic arm in a dynamic environment.

[0006] According to some embodiments, a first solution of the present invention provides a method for screw locking of a robotic arm based on multi-level positioning of unstructured screw holes, which adopts the following technical solution:

[0007] A method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes, comprising:

[0008] Get the target image to be locked;

[0009] Based on different expansion rates, multi-scale perception fusion is performed on the target image to extract the edge morphology of the screw hole in the target image;

[0010] According to the extracted screw hole edge morphology and multi-layer supervision mechanism, the screw hole global edge prediction map is obtained;

[0011] The center of the screw hole is located based on the obtained global edge prediction map of the screw hole to obtain the true posture of the screw hole;

[0012] The center of the screw hole is located based on the obtained global edge prediction map of the screw hole to obtain the position information of the screw hole and the rough posture information including the error;

[0013] In the posture alignment network, the rough posture information containing errors is corrected to determine the posture signal of the screw hole. The robot arm moves to the top of the screw hole to lock the screw according to the position information and posture information of the screw hole.

[0014] As a further technical limitation, in the process of obtaining the target image to be locked, the original image is captured, the screw hole area anchor frame is identified in the obtained original image, and the screw locking region of interest (i.e., ROI region) is extracted in the identified screw hole area anchor frame to obtain the target image to be locked.

[0015] As a further technical limitation, a convolutional neural network is used to perform multi-layer convolution operations on the acquired target image to be locked, and image features of the target image to be locked are extracted layer by layer. The extracted multi-layer image features are fused at multi-scale perception through different expansion rates to obtain the edge morphology of the screw hole of the target image. The edge information of the edge morphology of the screw hole is learned according to the multi-layer supervision mechanism to obtain the edge detection result of each layer, and the edge detection results of each layer are weighted averaged and fused to obtain a global edge prediction map of the screw hole.

[0016] As a further technical limitation, in the process of positioning the center of the screw hole, the obtained global edge prediction map of the screw hole is converted into three-dimensional space to obtain a three-dimensional prediction map of the global edge of the screw hole. The three-dimensional prediction map of the global edge of the screw hole is denoised by a geometric relationship filtering method to eliminate non-edge points. The edge points are iteratively optimized using the minimum error circle fitting method to obtain the optimal screw hole center position, the screw hole center point position information and the rough posture information containing errors.

[0017] As a further technical limitation, based on the rough posture of the screw hole, the posture alignment network is applied according to the RGB image obtained by the camera at the end of the current robotic arm to predict the correction direction of the current posture and the actual posture of the screw hole in the next step, and the alignment error between the robotic arm and the screw hole is corrected according to the step size. It is iterated in the X and Y directions respectively, and finally corrected to a reasonable error to complete the correct locking.

[0018] According to some embodiments, the second solution of the present invention provides a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, which adopts the following technical solutions:

[0019] A mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, comprising:

[0020] An image acquisition module, which is configured to acquire a target image to be locked;

[0021] An edge extraction module is configured to perform multi-scale perceptual fusion on the target image based on different expansion rates to extract the edge morphology of the screw hole in the target image; and obtain a global edge prediction map of the screw hole based on the extracted edge morphology of the screw hole and the multi-layer supervision mechanism;

[0022] A coarse positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain the real posture of the screw hole;

[0023] A fine positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain screw hole position information and rough posture information containing errors;

[0024] The locking module is configured to correct the rough posture information containing errors in the posture alignment network to determine the posture signal of the screw hole. The robot arm moves to the top of the screw hole according to the position information and posture information of the screw hole to lock the screw.

[0025] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0026] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of a mechanical arm screw locking method based on multi-level positioning of unstructured screw holes as described in the first solution of the present invention.

[0027] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0028] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps in a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in the first solution of the present invention are implemented.

[0029] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0030] A computer program product includes software codes, wherein the program in the software codes executes the steps in a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in the first solution of the present invention.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention captures the edge morphology of the screw hole in the target image based on edge detection and multi-scale fusion technology, optimizes the detection of the screw hole edge through a multi-layer supervision mechanism, and obtains a global edge prediction map of the screw hole; obtains the center point and rough posture of the screw hole according to the global edge prediction map; based on the rough posture of the screw hole, applies a posture alignment network to predict the correction direction of the current posture and the real posture of the screw hole in the next step, corrects the alignment error between the robot arm and the screw hole according to the step size, and finally corrects to the precise posture to complete the locking, thereby improving the adaptability and operation accuracy of the robot arm in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0034] Figure 1 This is a schematic diagram of a mechanical arm screw locking method based on multi-level positioning of unstructured screw holes in the first embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the screw hole edge detection in the first embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the robot arm posture prediction in the first embodiment of the present invention;

[0037] Figure 4 It is a structural block diagram of a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes in the second embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0041] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0042] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0043] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0044] It should be noted that unstructured scenarios are data scenarios that do not conform to the traditional relational database table model; the data structure in this data scenario is irregular or incomplete, there is no predefined data model, and it is not convenient to use a database two-dimensional logical table to represent it. Unstructured scenario databases can flexibly process and query these unstructured data and provide efficient storage and retrieval functions. Due to the diversity of unstructured data, this database system usually has the characteristics of high flexibility (able to process files and data types of various formats), efficient storage (able to quickly store and retrieve large amounts of data), and convenient query (providing efficient query functions to meet the needs of rapid response).

[0045] The unstructured scene in the present invention can be understood as a scene that is dynamic and not static, in a non-horizontal and non-vertical orientation, or in a narrow and unconventional orientation.

[0046] Embodiment 1

[0047] Embodiment 1 of the present invention introduces a method for screw locking of a robotic arm based on multi-level positioning of unstructured screw holes.

[0048] This embodiment proposes an intelligent solution for the automatic screw locking task of the robot arm in an unstructured scene to cope with the increasingly complex needs and challenges in industrial automation and improve the adaptability and operation accuracy of the robot arm in a dynamic environment. First, this embodiment introduces the ShedNet network to perform high-precision edge detection on the area of ​​interest, and effectively extracts the edge feature points of the screw hole based on deep learning technology to ensure that the target area can still be accurately identified under complex backgrounds or occlusion conditions. Secondly, this embodiment obtains the center position of the screw hole based on the projection-based hole center positioning algorithm, makes full use of the spatial coordinates of the edge points of the screw hole, and combines the geometric calculation method to determine the center of the screw hole within a very small error range. In order to ensure that the action path of the robot arm in performing the locking task is optimal, this embodiment introduces the AcNet network, which intelligently predicts the next position of the robot arm according to the RGB feedback information in the dynamic scene in a real-time processing environment, iteratively optimizes the posture and motion trajectory of the robot arm, ensures the high accuracy and efficiency of its operation, and improves the adaptability of the robot arm in a complex unstructured environment, so that it can flexibly cope with various unstructured scenes.

[0049] like Figure 1 The robotic arm screw locking method based on multi-level positioning of unstructured screw holes includes the following steps:

[0050] A. Target detection and ROI extraction;

[0051] B. Image enhancement processing;

[0052] C. High-precision edge detection;

[0053] D. Screw hole center positioning;

[0054] E. Robotic arm posture prediction.

[0055] As one or more implementation modes, this embodiment collects several high-quality images and accurately identifies the anchor frames of the screw hole areas through manual annotation. To optimize the target detection effect, this embodiment uses the K-means clustering algorithm to calculate the effective anchor frames. By clustering analysis of these labeled data, an anchor frame configuration suitable for the screw hole size distribution is generated, which significantly improves the target detection performance.

[0056] In this embodiment, the ROI can be extracted from the marked anchor frame by the YOLOV5 algorithm, laying a foundation for edge detection. In other embodiments, other algorithms are also used to extract the position area.

[0057] In this embodiment, the data required for screw hole edge detection can be obtained by acquiring images and depth information with a depth camera at the end of the robotic arm; to ensure the diversity of the data set, this embodiment can set the configuration of the robotic arm and the depth camera in different experimental environments; by configuring the depth camera to capture screw hole images in real scenes, while ensuring the flexibility of the robotic arm, it is possible to shoot the screw holes at different angles and positions; under different lighting conditions, multiple shots can be taken based on natural light and artificial lighting to obtain screw hole images under a variety of lighting conditions; at different angles, the screw holes can be photographed from multiple perspectives to capture their different appearance features; under a variety of backgrounds, different backgrounds can be selected to improve the richness of the image to ensure that the model can adapt to various application scenarios.

[0058] In the process of collecting images, each image is acquired by a depth camera controlled by a robotic arm to ensure the quality and stability of the image. In each image, the edge point of the screw hole is clicked by the mouse to generate the edge coordinates. Using the obtained edge points, an ellipse shape can be fitted through an appropriate algorithm to generate a corresponding binary mask that accurately represents the contour of the screw hole edge.

[0059] To prevent the model from overfitting, this embodiment introduces data enhancement in the dataset, namely rotation (randomly rotating the image at random angles to improve the robustness of the model to screw holes in different directions), scaling (randomly adjusting the image size to simulate screw hole images at different distances), color transformation (performing color transformation on the image, including random adjustment of brightness, contrast and saturation to enhance the model's adaptability to lighting changes) and symmetry transformation (flipping the image horizontally and vertically to increase the diversity in the dataset).

[0060] It should be noted that the image enhancement processing in this embodiment belongs to the prior art, which can be understood as the prior art that those skilled in the art should know, and will not be described in detail in this embodiment.

[0061] As one or more implementation modes, this embodiment adopts the following Figure 2 The screw hole edge detection network ShedNet based on the fully convolutional neural network shown in the figure realizes the edge detection of the screw hole image. Through the feature extraction module and the multi-scale feature fusion module, the model learns to aggregate the feature information under different receptive fields. At the same time, the introduction of multi-layer supervision modules enables the model to better capture subtle and complex edge structures.

[0062] In edge detection, different levels of image features are important for capturing edges. This embodiment uses a multi-layer convolution module as a feature extraction network (i.e., Conv Module); given an input image Among them, H and W represent the height and width of the image respectively. Multiple feature extraction modules perform multi-layer convolution operations on the input image or feature map to gradually extract feature representations at different levels.

[0063] The extracted feature layer is F i =f i (X), i = 1, 2, 3; where f i (X) represents the feature map of the i-th convolution module (i.e., from conv_1 to conv_3). For each convolution layer, the dimension of the feature map gradually decreases as the layer deepens. The features extracted from the first few layers tend to be details, while the high-level features gradually become more global. To adapt to the task of screw hole edge detection, this embodiment uses three convolution modules to reduce the amount of calculation and enhance the ability to capture small edges; at the same time, in the feature extraction process, reflection filling is used to avoid boundary effects, which makes the original information of the image edge more complete in the convolution operation.

[0064] The edges of screw holes usually have different scales and complexities, and a single-scale convolution cannot fully capture all possible edge forms. This embodiment uses different expansion rates to perform multi-scale perception of features to achieve feature fusion, thereby improving the ability of edge detection.

[0065] The core of the multi-scale fusion module lies in the application of dilated convolution. Dilated convolution increases the receptive field while keeping the convolution kernel parameters unchanged by inserting holes between the convolution kernels. Given an input feature map, the multi-scale fusion module extracts multi-scale information through a series of convolution operations with dilation rates, i.e. Among them, F (l) is the feature map of the lth layer input, * represents the convolution operation, is with expansion rate d k The convolution kernel weight is K, and K is the number of different expansion rates. In this embodiment, three expansion rates d1=4, d2=8, and d3=12 are set respectively to capture the local and global features of the edge of the screw hole at different scales.

[0066] The features of the multi-scale convolution output are fused with the initial input features through residual connections to generate multi-scale fusion features. Right now The fused feature map is subjected to 1×1 convolution to reduce the number of channels and obtain the final output as the final feature map for edge detection at the current level.

[0067] To ensure that edge information can be effectively learned on feature maps at different levels, this embodiment introduces a multi-layer supervision mechanism; by adding independent supervision signals after each convolutional layer, the network can not only learn edge information on high-level features, but also optimize edge detection capabilities on low-level and mid-level feature maps.

[0068] Given an input feature map F i The feature map is the feature information extracted after the layer passes through the feature extraction network and the multi-scale fusion module. The feature maps of each layer are upsampled to make the resolution consistent with the input image.

[0069] Upsampling is achieved through the deconvolution layer (ConvTranspose2d), and the specific upsampling rate depends on the reduction ratio of the feature map. For example, the third layer feature map needs to be upsampled 8 times, that is The final edge detection result is obtained by concatenating all upsampled prediction results and further fusing them through a 1×1 convolution, that is,

[0070] This embodiment adopts three main supervision branches, corresponding to different levels of convolution features, namely, P i =σ(W i *F i ), where W i is the convolution kernel of the supervision branch, σ represents the Sigmoid activation function, P i is the edge prediction of the i-th layer.

[0071] For each supervised branch, calculate the edge prediction result P at that level i The binary cross entropy loss between the true edge label Y, Where N represents the number of pixels, Y n Indicates the label of the nth pixel, P i,n is the predicted value of the nth pixel; the total loss is the weighted sum of the supervision losses of each layer, that is, Among them, α i It is the weight of each level of supervision branch. The edge detection results at different levels are fused by weighted average to generate a global edge prediction map.

[0072] As one or more implementation methods, in order to accurately solve the center position of the screw hole and the rough pose (i.e., inaccurate pose), this embodiment realizes the center positioning of the screw hole based on RGB image and depth camera coordinate transformation, converts the image edge detection results into three-dimensional space, and then filters the detected point cloud data for noise, and solves the screw hole center and its normal vector through fitting and iteration.

[0073] After edge detection is completed, the detected image coordinate points are converted from the RGB camera coordinate system to the world coordinate system through the coordinate transformation of the depth camera. In this process, the camera intrinsic and extrinsic parameters are known, and the pixel coordinates (u, v) on the image plane can be converted to the point P (x, y, z) in the three-dimensional world coordinate system through the standard perspective projection inverse transformation; the transformation formula is Among them, Pw is the point in the world coordinate system, is the transformation matrix from the camera to the world coordinate system, K is the camera intrinsic parameter matrix, and D(u,v) is the depth value of the depth map at the corresponding pixel point.

[0074] In order to eliminate noise points in the detection, this embodiment adopts a filtering algorithm based on geometric relationships. Since the depth of the edge of the screw hole changes continuously, the noise points are filtered based on the distance between adjacent points; that is, the center point of the edge point is used as the reference point Pref, and the polar angle θ between all edge points and the reference point is calculated. i =arctan2(p i,y -P ref,y ,p i,x -P ref,x ); after calculating the polar angle of each edge point, sort the edge points by polar angle; calculate the distance between adjacent points after sorting, and find the minimum distance dmin as the threshold for subsequent filtering; set parameter k as a multiple of the distance threshold, any point pair with a distance between adjacent points exceeding k×dmin will be regarded as a noise point and removed; the filtered valid point set Pvalid will be used for subsequent screw hole center fitting; it can effectively remove non-edge points introduced by image noise or detection errors in the edge detection process.

[0075] In order to accurately solve the three-dimensional center of the screw hole, a circle fitting method based on minimizing the error is used to find the optimal center position of the screw hole by iteratively optimizing the normal vector. Assuming that the normal vector of the screw hole is initially n0=[0,0,1], the Euler angle rotation matrix R(θ x ,θ y ,θ z ) to rotate the normal vector; the rotated normal vector n is n = R (θ x ,θ y ,0)·n0; where,θ x ,θ y is the rotation angle around the X-axis and Y-axis (the X-axis and Y-axis are the coordinate axes of the robot arm at this time), which is initially set to change in the range of [-30°, 30°] with a step size of 5°; for each rotation angle (θ x ,θ y ), and use the current normal vector n to fit the circle. The goal of fitting the edge points of the screw hole is to minimize the fitting error. Assuming that the parameters of the circle are (x c ,y c,r) (i.e., the coordinates of the center and the radius), the error is defined as the sum of the squares of the geometric distances from the edge point to the fitting circle, that is,

[0076] This embodiment traverses different rotation angles to find the normal vector n and the center coordinate (x c ,y c ). Under all rotation angles, find the normal vector n with the smallest error best and the corresponding screw hole center (x c ,y c ). The normal vector n of the screw hole best This is the optimal solution, which represents the normal vector direction of the screw hole. best and the center (x c ,y c ), the center position of the screw hole is solved by the straight line equation. The edge point set P of the screw hole and the normal vector n are known. best , which can represent the center of the screw hole P c Satisfy P c =P f +t·n best ; Among them, P f is the center of the circle obtained by fitting, and t is the displacement along the normal vector direction. The displacement t can be solved by minimizing the sum of the squares of the distances between the edge points of the screw hole and the center point; specifically: the objective function D is to minimize the sum of the squares of the vertical distances from all edge points to the center point, that is, The displacement t is derived and minimized, and finally the optimal screw hole center Pc is solved, that is, Get the exact center position of the screw hole in three-dimensional space and the direction of the normal vector (including error).

[0077] As one or more implementation modes, this embodiment adopts the following Figure 3 The robot arm posture prediction architecture shown in the figure designs a screw hole posture alignment network based on the CNN classifier to complete the posture prediction of the robot arm. The network predicts the alignment bias of the robot arm in the X / Y direction based on the RGB image obtained by the camera fixed at the end of the current robot arm; by continuously predicting the classification results in the X and Y directions, the network can guide the robot arm to gradually adjust the angle in the correct direction to achieve accurate alignment between the robot arm and the screw hole.

[0078] This embodiment models the angle correction problem as a classification problem, by predicting the angle deviation of the current robot arm relative to the screw hole axis and outputting the corresponding classification result to indicate the correction operation; for each direction (X direction and Y direction), the classification is divided into three categories, namely:

[0079] Category 0: The current robot arm angle is in the correct position and does not need to be adjusted.

[0080] Category 1: The current angle deviation is positive and needs to be rotated 5° in the positive direction for correction.

[0081] Category-1: The current angle deviation is negative and needs to be corrected by rotating 5° in the opposite direction.

[0082] In this embodiment, the input image passes through multiple convolutional layers and pooling layers to extract image features of different scales and levels. The extracted feature map is used for subsequent classification tasks. Assuming the input image is I, the extracted feature map is denoted as F = f(I), where f(·) represents the feature extraction process; for the features in the X direction / Y direction, a classifier module is designed to predict the angular deviation in the corresponding direction. Assuming the feature in the X direction is represented by F x , the feature in the Y direction is represented by F y , then the classification task can be expressed as in, and Represent the predicted classification results in the X direction and Y direction respectively, P x and P y To classify the probability distribution, the network outputs the probability of each category through the Softmax function.

[0083] The network loss function consists of classification loss and regularization term. Taking the X-direction posture alignment network loss function as an example, it is expressed as L = α·L x +λ·R(W),L x is the cross entropy loss, corresponding to the X-direction classification task, expressed as where y x,i One-hot encoding of the true label, p x,i is the predicted probability of the corresponding category. R(W) is the L2 regularization term of the network weight, which is used to prevent overfitting. The weight factors α and λ are adjusted through experiments and finally set to α = 1.0 and λ = 0.0001. The Adam optimizer is selected, the initial learning rate is set to 0.001, and the cosine annealing strategy is used to gradually reduce the learning rate to accelerate convergence. Through data enhancement methods such as random rotation, translation, and noise superposition, the diversity of training samples is increased and the generalization ability of the network is improved.

[0084] In addition, to further enhance the feature expression capability, a jump connection is introduced in the feature extraction module to fuse shallow detail information and deep semantic information, thereby improving the ability to capture angle deviation features. A channel attention mechanism is introduced on the feature map F to automatically assign weights to different channels to highlight key features related to angle deviation.

[0085] The robot will perform corresponding rotation compensation according to the classification results output by the network. Assume that the prediction result in the X direction is The prediction result in the Y direction is Then the rotation compensation of the robot arm is

[0086] when or When , it means that the angle of the robot arm in this direction is correct and no adjustment is needed; when or When , the robot arm needs to rotate 5° in the positive direction; when or When the robot arm rotates 5° in the opposite direction, the robot arm can quickly correct the alignment error with the screw hole within a limited number of steps to achieve precise assembly.

[0087] In this embodiment, a total of 400 images were collected in the range of [-30°, 30°] in the x-axis direction and [-30°, 30°] in the y-axis direction, and the collection step was 3°, that is, the x-axis angle change was [-5°, 5°] for category 0, and the y-axis angle change was [-5°, 5°] for category 0, and the x-axis angle change was [5°, 30°] for category 1, and the rest were category -1, and the same applies to the y-axis direction; through 50 experiments, the screw hole angle offset varied in the range of [-30°, 30°], 46 were successful, and the success rate was 92%.

[0088] This embodiment captures the edge morphology of the screw hole in the target image based on edge detection and multi-scale fusion technology, optimizes the detection of the screw hole edge through a multi-layer supervision mechanism, and obtains a global edge prediction map of the screw hole; the center point and rough posture of the screw hole are obtained according to the global edge prediction map, and the real posture of the screw hole is further obtained; the position and posture of the robot arm are determined in combination with the real posture of the screw hole, and the robot arm screw locking based on multi-level positioning of unstructured screw holes is completed according to the position and posture of the robot arm, thereby improving the adaptability and operation accuracy of the robot arm in a dynamic environment.

[0089] Embodiment 2

[0090] The second embodiment of the present invention introduces a mechanical arm screw locking system based on multi-level positioning of unstructured screw holes.

[0091] like Figure 4 A mechanical arm screw locking system based on multi-level positioning of unstructured screw holes is shown, comprising:

[0092] An image acquisition module, which is configured to acquire a target image to be locked;

[0093] An edge extraction module is configured to perform multi-scale perceptual fusion on the target image based on different expansion rates to extract the edge morphology of the screw hole in the target image; and obtain a global edge prediction map of the screw hole based on the extracted edge morphology of the screw hole and the multi-layer supervision mechanism;

[0094] A coarse positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain the real posture of the screw hole;

[0095] A fine positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain screw hole position information and rough posture information containing errors;

[0096] The locking module is configured to correct the rough posture information containing errors in the posture alignment network to determine the posture signal of the screw hole. The robot arm moves to the top of the screw hole according to the position information and posture information of the screw hole to lock the screw.

[0097] The detailed steps are the same as those of the mechanical arm screw locking method based on multi-level positioning of unstructured screw holes provided in Example 1, and will not be repeated here.

[0098] Embodiment 3

[0099] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0100] A computer-readable storage medium stores a program thereon, and when the program is executed by a processor, the steps of a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in the first embodiment of the present invention are implemented.

[0101] The detailed steps are the same as those of the mechanical arm screw locking method based on multi-level positioning of unstructured screw holes provided in Example 1, and will not be repeated here.

[0102] Embodiment 4

[0103] A fourth embodiment of the present invention provides an electronic device.

[0104] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in the first embodiment of the present invention are implemented.

[0105] The detailed steps are the same as those of the mechanical arm screw locking method based on multi-level positioning of unstructured screw holes provided in Example 1, and will not be repeated here.

[0106] Embodiment 5

[0107] Embodiment 5 of the present invention provides a computer program product.

[0108] A computer program product includes software codes, wherein the program in the software codes executes the steps in a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in the first embodiment of the present invention.

[0109] The detailed steps are the same as those of the mechanical arm screw locking method based on multi-level positioning of unstructured screw holes provided in Example 1, and will not be repeated here.

[0110] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes, characterized in that: include: Get the target image to be locked; Based on different expansion rates, multi-scale perception fusion is performed on the target image to extract the edge morphology of the screw hole in the target image; According to the extracted screw hole edge morphology and multi-layer supervision mechanism, the screw hole global edge prediction map is obtained; The center of the screw hole is located based on the obtained global edge prediction map of the screw hole to obtain the true posture of the screw hole; The center of the screw hole is located based on the obtained global edge prediction map of the screw hole to obtain the position information of the screw hole and the rough posture information including the error; In the posture alignment network, the rough posture information containing errors is corrected to determine the posture signal of the screw hole. The robot arm moves to the top of the screw hole to lock the screw according to the position information and posture information of the screw hole.

2. A method for screw locking by a mechanical arm based on multi-level positioning of unstructured screw holes as described in claim 1, characterized in that: In the process of acquiring the target image to be locked, the original image is collected, the screw hole area anchor frame is identified in the obtained original image, and the screw locking area of ​​interest is extracted in the identified screw hole area anchor frame to obtain the target image to be locked.

3. A method for screw locking by a mechanical arm based on multi-level positioning of unstructured screw holes as described in claim 1, characterized in that: A convolutional neural network is used to perform multi-layer convolution operations on the acquired target image to be locked, and the image features of the target image to be locked are extracted layer by layer. The extracted multi-layer image features are fused at multi-scale perception through different expansion rates to obtain the edge morphology of the screw hole of the target image. The edge information of the edge morphology of the screw hole is learned according to the multi-layer supervision mechanism to obtain the edge detection result of each layer. The edge detection results of each layer are weighted averaged and fused to obtain the global edge prediction map of the screw hole.

4. A method for screw locking by a mechanical arm based on multi-level positioning of unstructured screw holes as described in claim 1, characterized in that: In the process of screw hole center positioning, the obtained screw hole global edge prediction map is converted into three-dimensional space to obtain the screw hole global edge three-dimensional prediction map, and the screw hole global edge three-dimensional prediction map is denoised by the geometric relationship filtering method to remove non-edge points. The denoised screw hole global edge three-dimensional prediction map and the minimum error circle fitting method are used to perform iterative optimization to obtain the optimal screw hole center position and the true posture of the screw hole.

5. A method for screw locking by a mechanical arm based on multi-level positioning of unstructured screw holes as described in claim 1, characterized in that: The position of the screw hole axis is determined according to the obtained real posture of the screw hole, and the angle and position of the robot arm are adjusted according to the angular deviation between the current robot arm and the screw hole axis until the robot arm and the screw hole are aligned, thus completing the determination of the robot arm posture.

6. A method for screw locking by a mechanical arm based on multi-level positioning of unstructured screw holes as described in claim 5, characterized in that: According to the positive or negative angle deviation between the current robot arm and the axis of the screw hole, the movement direction of the robot arm on different axes is adjusted respectively, the alignment error between the robot arm and the screw hole is corrected, and the angle and position of the robot arm are adjusted by rotating the robot arm.

7. A mechanical arm screw locking system based on multi-level positioning of unstructured screw holes, characterized in that: include: An image acquisition module, which is configured to acquire a target image to be locked; An edge extraction module is configured to perform multi-scale perceptual fusion on the target image based on different expansion rates to extract the edge morphology of the screw hole in the target image; and obtain a global edge prediction map of the screw hole based on the extracted edge morphology of the screw hole and the multi-layer supervision mechanism; A coarse positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain the real posture of the screw hole; A fine positioning module is configured to perform screw hole center positioning based on the obtained screw hole global edge prediction map to obtain screw hole position information and rough posture information containing errors; The locking module is configured to correct the rough posture information containing errors in the posture alignment network to determine the posture signal of the screw hole. The robot arm moves to the top of the screw hole according to the position information and posture information of the screw hole to lock the screw.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in any one of claims 1-6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of a method for screw locking by a robotic arm based on multi-level positioning of unstructured screw holes as described in any one of claims 1-6.

Images