A method and device for automatic wiring based on visual guidance

Abstract

The present invention discloses a method and device for automatic wiring based on visual guidance, the device includes a wiring operation execution part, a visual detection part and a drive control part; the wiring operation execution part includes a two-dimensional motion platform, a rotation auxiliary mechanism, a six-degree-of-freedom industrial robot and a clamp installed at the end of the industrial robot; the visual detection part includes two industrial cameras, a miniature camera and an image acquisition card; the drive control part motion control card and a robot controller. The present invention adopts visual guidance technology to realize the automatic wiring operation of industrial robots, with high intelligence and flexibility, high success rate of cable terminal positioning and wire hole assembly, strong flexibility, and can be used for industrial application scenarios such as flexible assembly of automatic cable plugging.

Description

Technical Field

[0001] The present invention relates to the field of automatic wiring, and in particular to a method and a device for automatic wiring based on vision guidance. Background Art

[0002] Nowadays, production equipment in manufacturing companies is almost inseparable from the connection of circuit components. Traditional circuit connection generally relies on manual plugging on the plugboard, which consumes a lot of manpower and capital for inefficient and repetitive work. After entering the era of intelligence, many factories have begun to seek human-machine combination or completely replace the old manual plug-in method with robots or other equipment, such as the development of some automatic wiring equipment for optical fiber.

[0003] However, there are three main problems with replacing manual labor with machines to complete wiring. First, the cable is a soft structure with an unstable shape, which makes it difficult to grab. Second, the cable terminal structure is complex, and the postures vary during feeding, which requires posture adjustment. Finally, when plugging the cable terminal into the connector, the gap between the two is very small, so a high positioning accuracy is required.

[0004] The operation of automatic wiring is complicated, and it is difficult to complete the entire process of wire hole plugging by simply relying on mechanical operation. In the face of some complex operations in industrial production, industrial robots are often introduced with industrial cameras for visual guidance operations, such as picking and stacking. The SYNDY system designed by System Robot Automation is mainly used for the connection of wires in electrical equipment, and end-to-end wiring operations can be achieved through manipulators. However, the end effector structure of the device is large, which makes it unable to complete the wiring tasks of a large number of cables and high-density wiring. The vision-based wiring system developed by Kyongmo Koo and others assembles the wiring harness of the car into the car body. Through the laser sensor, the point cloud model is obtained to find the position of the hole and the direction of insertion, and guide the robot to complete the shaft hole matching work, but because the system uses more equipment and the cost is high, it is difficult to be widely used.

[0005] In the research on visual algorithms related to wiring operations, some scholars have proposed using CNN algorithms to identify cable terminals, while others have used color separation to extract cables. The former needs to improve its recognition accuracy and cannot identify the cable body. The latter has the defect of being unable to distinguish cables with different terminals. In terms of wire hole plug-in, many similar or related research works are guided by stereo vision, using depth cameras or binocular stereo vision devices to obtain the three-dimensional position of the target object, and finally provide assembly guidance.

[0006] Therefore, the current research on wiring operations still has the following deficiencies: First, the recognition algorithm for cable terminals needs to further improve the accuracy and precision of detection; second, the problem of cable segmentation for different terminals; third, there is a lack of relevant research on the posture estimation and adjustment of cable terminals. Due to the small size of cable terminals, it is difficult to extract their three-dimensional image information, and it is not easy to estimate their posture through three-dimensional information, which can easily cause large estimation deviations and lead to wiring failures; fourth, currently more stereoscopic vision is used to guide the actuator to plug in the wires, but for flexible cables, the lack of real-time closed-loop control during the plugging process can easily reduce the success rate. Summary of the invention

[0007] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, an object of the present invention is to provide a method and device for automatic wiring based on vision guidance.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A method for automatic wiring based on visual guidance, characterized in that it includes the following steps:

[0010] Establish the coordinate system relationship between the robot and the camera;

[0011] Move the robot to the loading area, obtain the cable image in the loading area, calculate the cable picking posture, and drive the robot to clamp the cable;

[0012] Obtain a side image of the cable terminal, build an angle prediction model for the cable terminal, use the image features of the local color histogram of the cable terminal as input, and the predicted angle as output, and adjust the posture of the cable terminal according to the predicted angle until the posture of the cable terminal is adjusted to a preset correct angle;

[0013] The cable terminal adjusted to the preset correct angle is clamped, and the robot is moved to the wire insertion position to obtain the image of the target connector. By constructing a visual mapping model based on support vector regression, the positional relationship between the connector and the cable terminal in the target connector image is extracted as the image feature, and then the image feature is used as the input of the visual mapping model. The output of the model is to predict the target robot posture, adjust the robot posture, and complete the wire hole assembly.

[0014] Further, the mobile robot goes to the loading area, obtains the cable image of the loading area, calculates the cable picking posture, and drives the robot to clamp the cable, specifically:

[0015] Collect images of cables in the loading area, and use the improved YOLOv5 algorithm to identify and locate the cable terminals in the images;

[0016] After the cable terminal is identified and located, the ROI image of the target cable terminal in the image is intercepted, and the contour edge and the minimum outer bounding rectangle of the cable terminal are extracted through binarization and morphological processing, and then the central axis of the cable terminal is obtained;

[0017] Perform color threshold segmentation on the original image of the collected cable, perform preliminary image segmentation on the cable body, and obtain candidate target areas;

[0018] The central axis of the cable terminal and the candidate target area are obtained for AND operation. The retained pixels are used as seed pixels. The 8-connected domain growth algorithm is used to perform secondary segmentation of the cable body on the image to obtain the final target cable body area.

[0019] According to the target cable line area, it is processed using a refinement algorithm to obtain the line centerline, and sampling is performed on the centerline. The three-dimensional coordinates of the sampling points are obtained through the binocular camera triangulation algorithm. The sampling points are fitted with straight lines to obtain the cable picking posture. The robot is driven to move to the picking position, and the cable picking is completed by controlling the pneumatic gripper.

[0020] Further, the side image of the cable terminal is obtained, and an angle prediction model of the cable terminal is constructed, with the image features of the local color histogram of the cable terminal as input and the predicted angle as output, and the posture of the cable terminal is adjusted according to the predicted angle until the posture of the cable terminal is adjusted to a preset correct angle, specifically:

[0021] Straighten the cable body so that the cable terminal and the cable body are in the same straight line. At the same time, clamp and limit the cable so that it retains one degree of freedom, that is, it can only rotate around its own cable body as the axis;

[0022] Collect the side image of the cable terminal, intercept the local ROI of the cable terminal in the image, track it using the KCF algorithm, extract the color histogram of the local ROI image, and use the Otsu method to segment the color histogram of the foreground area as image feature data;

[0023] By constructing and offline training an angle prediction model for cable terminals, the image feature data is obtained as input, and the model predicts the target angle of the corresponding cable terminal. According to the target angle, the cable terminal is driven to rotate around its own wire body, thereby achieving angle adjustment of the cable terminal. When the cable terminal is adjusted to the correct angle, the adjustment is completed and the subsequent wire hole plug-in operation is carried out.

[0024] Furthermore, the cable terminal adjusted to the preset correct angle is clamped, and the robot is moved to the wire insertion position to obtain the image of the target connector. By constructing a visual mapping model based on support vector regression, the positional relationship between the connector and the cable terminal in the target connector image is extracted as an image feature, and then the image feature is used as the input of the visual mapping model. The output of the model is to predict the target robot posture, adjust the robot posture, and complete the wire hole assembly, specifically:

[0025] The cable with posture adjustment is picked up again, and the robot is moved to the insertion position near the target hole of the connector to collect images of the connector and the cable terminal in the clamped state;

[0026] Identify and locate the connector and cable terminal in the image, then perform binarization on the connector ROI image and the cable terminal ROI image respectively and extract the central pixel point of the connected minimum outer bounding rectangle and the top of the cable terminal. Finally, calculate the image distance from the center point of the top of the cable terminal to the minimum outer bounding rectangle of the connector, and use this as their image position feature data;

[0027] By constructing and offline training a visual mapping model based on support vector regression, the calculated image feature data is used as input, and the model outputs the predicted robot position. By driving the robot to the predicted position, the images of the connector and cable terminals at the current position are recaptured, the target position is predicted again, and then the robot is moved again until the image position error is less than the expected error, thereby realizing closed-loop visual wire insertion guidance.

[0028] Furthermore, template matching based on image grayscale correlation is used to identify and locate connectors and cable terminals.

[0029] Furthermore, the improved YOLOv5 algorithm includes an input end, a backbone network, a Neck network, an enhanced context module ECM structure and an output end.

[0030] Furthermore, the enhanced context module ECM structure is specifically:

[0031] Based on the receptive field module model, its branch structure is adjusted and the original convolution block is replaced with a double-hole convolution structure. At the same time, a single 3×3 convolution branch is used.

[0032] A device for implementing the method, comprising a wiring operation execution part, a visual detection part and a drive control part;

[0033] The wiring operation execution part includes a two-dimensional motion platform, a rotation auxiliary mechanism, a six-degree-of-freedom robot and a clamp installed at the end of the robot;

[0034] The rotation auxiliary mechanism is installed at the moving end of the two-dimensional motion platform. As the two-dimensional motion platform moves in the vertical and horizontal directions, the rotation auxiliary mechanism is used to adjust the posture of the cable terminal. The clamper is installed at the end of the robot and moves with the movement of the end of the robot to clamp the cables to be plugged in the loading area.

[0035] The visual inspection part includes two industrial cameras, a miniature camera and an auxiliary light source. The two industrial cameras are installed in the clamp at the end of the robot to collect images of cables and connectors; the miniature camera is installed on the clamp to collect images of cable terminals and target plug-in holes in the clamping state during hole assembly operations; the auxiliary light source is installed in the clamp at the end of the six-degree-of-freedom industrial robot to adjust the light intensity and the ambient light around the image acquisition area;

[0036] The drive control part is used to drive the two-dimensional motion platform to move and the gears inside the rotation auxiliary mechanism to rotate and drive the six-degree-of-freedom industrial robot to move to the target position.

[0037] Furthermore, the rotation assist mechanism includes a metal base and a rotation assist mechanism body. The rotation assist mechanism body includes three complete gears, one incomplete gear, a limiting mechanism and a clamping mechanism. The limiting mechanism and the clamping mechanism pull the cable body to be plugged in into a straight state, and the rotation of the gear drives the cable terminal placed therein to rotate around its own cable body as the axis, thereby realizing the posture adjustment of the cable terminal.

[0038] Furthermore, the clamp includes a pneumatic clamp and a metal connector, and the industrial camera is installed on the clamp at the end of the six-freedom industrial robot; the micro camera is installed on the clamp and is adjacent to the pneumatic clamp.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] (1) According to the automated wiring task, a network model based on improved YOLOv5 is proposed to improve the detection capability of terminals. A cable segmentation algorithm combining terminal ROI information and color information is proposed, and the cable picking posture is estimated through a binocular camera.

[0041] (2) Aiming at the problem of adjusting the posture of cable terminals, a regression discriminant model based on multi-view feature learning is used to estimate the terminal angle and make adjustments;

[0042] (3) By designing and extracting the image information of the socket contour and the terminal in the image space as features, a mapping model is trained with the robot arm posture to achieve uncalibrated wire plugging visual guidance based on support vector regression. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 1 is a schematic diagram of the overall structure of a device for automatic wiring based on visual guidance according to Embodiment 1 of the present invention;

[0044] Figure 2 This is a flowchart of a method for automatic wiring based on visual guidance according to Embodiment 2 of the present invention;

[0045] Figure 3 It is a schematic diagram of the principle of the pinhole imaging model in Example 2 of the present invention;

[0046] Figure 4 Schematic diagram of the binocular camera principle in Embodiment 2 of the present invention;

[0047] Figure 5 is a schematic diagram of the structure of an enhanced context module (ECM) in Example 2 of the present invention;

[0048] Figure 6 Schematic diagram of the double-atrous convolution structure in Embodiment 2 of the present invention;

[0049] Figure 7 Schematic diagram of the network model of the improved YOLOv5 in Example 2 of the present invention;

[0050] Figure 8 This is a flow chart of the closed-loop visually guided wire insertion operation in Example 2 of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0052] Example 1

[0053] like Figure 1 As shown, a device for automatic wiring based on vision guidance includes a wiring operation execution part, a visual detection part and a drive control part.

[0054] The wiring operation execution part includes a two-dimensional motion platform 2, a rotation auxiliary mechanism 3, a six-degree-of-freedom industrial robot 1 and a clamper 7 installed at the end of the industrial robot.

[0055] The rotation auxiliary mechanism 3 includes a metal base and a rotation auxiliary mechanism body, which is installed on the moving end of the two-dimensional motion platform 2 and can move in the vertical and horizontal directions with the two-dimensional motion platform 2. The two-dimensional motion platform 2 is set on the experimental table through the support frame 8. The rotation auxiliary mechanism body is mainly composed of three complete gears, an incomplete gear, a limit mechanism and a clamping mechanism. The limit mechanism and the clamping mechanism pull the cable body to be plugged into a straight state, and the cable terminal placed therein is driven to rotate with its own cable body as the axis through the rotation of the gear, so as to achieve the posture adjustment of the cable terminal.

[0056] The clamp 7 is installed at the end of the six-degree-of-freedom industrial robot and can move with the movement of the robot end. The clamp 7 is composed of pneumatic grippers and metal connectors and is used to clamp the cables to be plugged in the loading area.

[0057] The visual inspection part includes two industrial cameras 5, a micro camera 6, an auxiliary light source and an image acquisition card 11. The two industrial cameras 5 are installed on the clamp 7 at the end of the six-degree-of-freedom industrial robot 1, and are used to collect images of cables and connectors 9. The micro camera 6 is installed on the clamp 7 and is adjacent to the pneumatic clamp, and is used to collect images of cable terminals and target plug-in holes in a clamped state during hole assembly operations. The two industrial cameras 5 and the micro camera 6 can move with the movement of the robot end. The auxiliary light source 4 is installed on the clamp 7 at the end of the robot and is adjacent to the two industrial cameras 5. The ambient light around the image acquisition area is adjusted by adjusting the light intensity; the industrial camera 5 and the micro camera 6 are connected to the computer 13 through the image acquisition card 11 to realize image acquisition and transmission control.

[0058] The drive control part includes a motion control card 10 and a robot controller 12;

[0059] The motion control card 10 is interconnected with the computer 13, and drives multiple stepper motors to work through digital signals, thereby driving the two-dimensional motion platform 2 to move and the gears inside the rotation auxiliary mechanism 3 to rotate. The robot controller 12 is interconnected with the computer 13 through a network cable. The computer 13 can send a position signal to the robot controller 12 by executing a written robot program, thereby driving the six-degree-of-freedom industrial robot 1 to move to the target position.

[0060] Example 2

[0061] like Figure 2 As shown, a method for automatic wiring based on visual guidance mainly includes the steps of establishing a coordinate system, picking up cables, adjusting the posture of cable terminals, and guiding the cables to be plugged into connectors, as follows:

[0062] Step S1: For two industrial cameras installed on the end gripper of the robot, obtain the conversion relationship between the camera coordinate system and the robot coordinate system;

[0063] Step S2: Move the robot to the loading area, obtain the cable image in the loading area, calculate the cable picking posture by combining the target recognition algorithm, image segmentation algorithm and binocular ranging method, and then drive the robot to clamp the cable;

[0064] Step S3: Clamp the cable onto the rotation auxiliary mechanism, collect the side image of the cable terminal, build an angle prediction model for the cable terminal, use the image features of the local color histogram of the cable terminal as input, and use the predicted angle as output, and drive the rotation auxiliary mechanism to adjust the posture of the cable terminal until the posture of the cable terminal is adjusted to a preset correct angle;

[0065] Step S4: re-grip the cable terminal after the posture adjustment, move the robot to the wire insertion position, capture the image of the target connector through the micro camera installed on the clamp, and construct a visual mapping model based on support vector regression to extract the positional relationship between the connector and the cable terminal in the image as the image feature input, output the predicted target robot arm posture, thereby driving the robot end to move and perform real-time posture adjustment until the robot arm position is adjusted to within the expected image feature error, and the wire hole assembly can be completed.

[0066] Furthermore, if Figure 3 and Figure 4 As shown, step S1 includes the following specific steps:

[0067] Firstly, the camera intrinsic parameters of the two industrial cameras installed on the robot's end gripper are calibrated. Then, a binocular camera triangulation model is constructed to obtain the transformation relationship between the camera coordinate system and the image coordinate system. Finally, the industrial camera and the robot are calibrated by hand and eye to obtain the transformation relationship between the robot coordinate system and the camera coordinate system.

[0068] Furthermore, the intrinsic parameters of the camera can be obtained according to the pinhole imaging model, such as Figure 3 As shown, a point P in space w (X w ,Y w ,Z w ) and its image projection point p(u,v) is related as follows:

[0069]

[0070] Among them, fx and fy represent the focal lengths in the vertical and horizontal directions in the image coordinate system, respectively, u0 and v0 are the offsets of the origin of the pixel coordinate system on the imaging plane. Since their values ​​are only related to the camera structure, they are both camera internal parameters; while R and P represent the transformation relationship between the world coordinate system and the camera coordinate system, which are the external parameters of the camera. Among them, the camera internal parameters can be obtained through Zhang Zhengyou calibration method.

[0071] Since the binocular camera is a follow-up camera installed on the clamping tool at the end of the robotic arm, there is a fixed positional relationship between the camera and the end of the robotic arm:

[0072]

[0073] in, is the transformation matrix from the tool coordinate system to the camera coordinate system, which can be determined by hand-eye calibration; is the transformation matrix between the tool coordinate system and the base coordinate system, so M is the external parameter matrix of the camera.

[0074] Further, see Figure 4 , the binocular camera ranging principle can be derived through trigonometric relationship:

[0075]

[0076] Where T is the distance between the optical centers of the two cameras (the origin of the camera coordinate system), x l and x r are the horizontal pixel coordinate values ​​of the left and right cameras respectively, and Z is the depth distance in the camera coordinate system. It can be further deduced that:

[0077]

[0078] Where X, Y, and Z are the three coordinate values ​​of the left camera in the camera coordinate system (with the lens optical center as the origin).

[0079] The specific process of step S2 is as follows:

[0080] S21, collecting images of cables in the loading area, and identifying and locating the cable terminals in the images by using an improved YOLOv5 algorithm;

[0081] S22, after the target detection and positioning of the cable terminal, intercept the ROI image of the target cable terminal in the image, extract the contour edge and the minimum outer enclosing rectangular frame of the cable terminal through binarization and morphological processing, and then obtain the central axis of the cable terminal;

[0082] S23, performing color threshold segmentation on the collected original image, performing preliminary image segmentation on the cable body, and obtaining a candidate target area;

[0083] S24, combining the processing results of S22 and S23, performing an AND operation on the central axis of the target cable terminal and the candidate cable body area, using the retained pixel points as seed pixel points, and using the 8-connected domain growth algorithm to perform secondary segmentation of the cable body on the image to obtain the final target cable body area;

[0084] S25, performing a refinement algorithm on the obtained target cable line area to obtain the line centerline, then sampling is performed on the centerline, and the three-dimensional coordinates of the sampling points are obtained through a binocular camera triangulation algorithm. The three-dimensional points are fitted with a straight line to obtain the cable picking posture, and the robot is driven to move to the picking position, and the cable picking is completed by controlling the pneumatic gripper;

[0085] Furthermore, for the target recognition and positioning method of the cable terminal in S21, the present invention adopts an improved YOLOv5 target detection algorithm, and the network structure of YOLOv5 includes: input end, backbone network, Neck, Head; among them, the backbone network is CSPDarknet53, which includes Focus, CBL, SPP, BottleNeckCSP modules; the Neck part adopts the FPN+PAN network structure.

[0086] Aiming at the problem that the YOLOv5 model has poor performance in detecting small target defects, an improved YOLOv5 model suitable for small target image detection of cable terminals is established; an enhanced context module (ECM structure) is added to expand the receptive field and embedded between the Neck module and the Head module, see Figure 7 .

[0087] Among them, Figure 5 The figure shows the structure diagram of the enhanced context module ECM. Based on the Receptive Field Block (RFB) model, its branch structure is adjusted, the original convolution block is replaced with a double-hole convolution structure, and a single 3×3 convolution branch is used. Figure 6 As shown in the figure, in order to simulate the group receptive field mechanism in the human visual system, the sampling rate b>a is set. Therefore, the double-hole convolution structure has a larger effective receptive field and has a stronger ability to extract contextual information compared to the original branch structure in RFB.

[0088] The theoretical receptive field of the double-hole convolution structure is calculated as follows:

[0089]

[0090] Among them, R is the receptive field size; s is the step size; f is the size of the convolution kernel, and f′ is the actual effective convolution kernel size of the dilated convolution. Placing the ECM module before the Head prediction layer can further expand the receptive field of high-level information after multi-scale feature fusion.

[0091] Furthermore, the training method for the improved YOLOv5 network model is as follows:

[0092] 1) Multi-scale sampling of cable terminals and image acquisition at various tilt angles, including placing various interference objects similar to cable terminals in the background, adjusting ambient light of different intensities, and acquiring cable terminals at different scales. The dataset contains 968 images in total, which are divided into training set, validation set, and test set in a ratio of 8:1:1;

[0093] 2) After the data set is created, the data is labeled and saved using labelimg software;

[0094] 3) Based on the Pytorch deep learning framework, the YOLOv5 operating environment is built in the Python environment, and the COCO dataset pre-trained weights are used as the initial weight parameters of the model, and the number of iterations is set to 300 for model training.

[0095] Furthermore, in steps S22 to S24, the segmentation and extraction method for the target cable body area of ​​different cable terminals is as follows:

[0096] 1) First, since the cable body has a fixed color, it can be initially separated and extracted by color separation. Therefore, the input RGB channel image is converted into an HSV channel image.

[0097] When performing threshold segmentation on the HSV image, its three channels are extracted separately and subjected to double threshold segmentation, namely:

[0098]

[0099] Among them, [H1, H2], [S1, S2] and [V1, V2] represent the high and low thresholds of the three channels respectively. The threshold range can be formulated by reading the values ​​of the three channels h(x, y), s(x, y) and v(x, y) of the pixels of the cable part in the HSV image.

[0100] 2) Secondly, the improved YOLOv5 detection model described in step S21 is used to identify and locate the cable terminals in the captured image, extract the ROI image of the target terminal, perform binarized threshold segmentation, and perform morphological processing. In order to further accurately extract the information of the terminal area, the binarized terminal image is contour detected and the minimum outer bounding rectangle is extracted. The central axis of the cable terminal is further calculated through the minimum outer bounding rectangle.

[0101] 3) Finally, the wire area preliminarily segmented by the color threshold is ANDed with the extracted cable terminal centerline at the wire connection position to obtain the local pixel points at the connection position between the cable terminal and the wire. These local pixel points are used as seed points to accurately segment the target cable wire area through the 8-connected region growing algorithm. The cable wire segmentation effect of different terminals in the feeding area.

[0102] Furthermore, when using binoculars for ranging, the left and right camera images need to be stereo corrected, including:

[0103] 1) First, the position relationship matrix between the binocular cameras is obtained through binocular positioning

[0104] 2) Then, according to The rotation matrix in Rotate the two images by half on average to reduce the reprojection error between the left and right cameras;

[0105] 3) Finally, construct a correction matrix to correct the two rotated images.

[0106] After completing the stereo correction, the relationship between the left camera coordinate system and the pixel coordinate system can be obtained as follows:

[0107] Where d is the disparity and Q is the reprojection matrix:

[0108]

[0109] Where, T x is the baseline distance, f is the focal length of the left camera, c' x is the horizontal coordinate of the origin of the right camera image, c x and c y are the horizontal and vertical coordinates of the origin of the left camera image, respectively. The method is used to perform stereo correction on the acquired left and right camera images, and the left and right images can be adjusted to line align to reduce the difficulty of matching left and right similar points.

[0110] Furthermore, in step S25, the method for calculating the cable grabbing posture includes:

[0111] First, stereo correction is performed on the cable image collected by binoculars;

[0112] Then, the target cable area is separated and extracted from the left and right images to obtain a binary image of the target cable body;

[0113] Thin the binary image to extract the central axis of the line body, and perform uniform sampling on the central axis;

[0114] The three-dimensional coordinates of the sampling points on the entire cable body are calculated through the binocular ranging method to estimate the posture of the entire cable body. The coordinates of some sampling points at the connection between the cable terminal and the cable body are fitted into a straight line, and finally the picking posture of the cable is estimated.

[0115] The process of step S3 is as follows:

[0116] S31. First clamp the cable onto the rotation auxiliary mechanism, straighten the cable body, make the cable terminal and the cable body on the same straight line, and at the same time, through the clamping mechanism and the limiting mechanism, make the cable retain only one degree of freedom, that is, it can only rotate around its own cable body as the axis, so that the subsequent posture adjustment operation of the cable terminal can be carried out.

[0117] S32. Capture the side image of the cable terminal through a binocular camera, intercept the local ROI of the cable terminal in the image, track it using the KCF algorithm, extract the color histogram of the local ROI image, and use the Otsu method to segment the color histogram of the foreground area as image feature data.

[0118] S33, by constructing and offline training a regression discriminant model based on multi-view feature learning, the image feature data obtained in S32 is used as input, the model predicts the target angle of the corresponding cable terminal, and drives the rotation auxiliary mechanism to rotate through a computer, driving the cable terminal to rotate around its own line body, thereby achieving the angle (posture) adjustment of the cable terminal;

[0119] S34, repeating steps S32 and S33 until the angle of the cable terminal is adjusted to a preset correct angle, then the adjustment is completed and the subsequent wire hole plugging operation can be performed;

[0120] Furthermore, in step S32, the method for extracting image features of a local area of ​​the cable terminal is as follows:

[0121] First, the robot arm is moved to the preset shooting position to collect the side image of the cable terminal. According to the cable terminal area after target recognition and positioning, the local ROI is intercepted by the preset offset value, and the KCF algorithm is used to track and locate the local area in real time.

[0122] The color histogram of the captured cable terminal local image is calculated, and the color histogram of the ROI image is thresholded using the Otsu method to extract the foreground (cable terminal) and background (non-target area), and the foreground area is retained by threshold screening;

[0123] The histogram data of the foreground area is processed as follows:

[0124]

[0125] In the formula, hist(K) is the histogram data, and M represents the number of rows of the ROI image. The image feature data of the local histogram of the cable terminal can be stably obtained through the above method.

[0126] Furthermore, in S33, the method of constructing a regression discrimination model based on multi-view features to adjust the angle of the cable terminal specifically includes:

[0127] 1) The side image of the cable terminal is collected discretely and evenly within a 360° range by rotating the cable terminal;

[0128] 2) Calculate the image feature data of the cable terminal ROI respectively for the collected left and right camera images and record the corresponding cable terminal angle;

[0129] 3) Since the input image feature data is 2D data and the output angle value is 1D data, the present invention uses the quadratic surface equation function as the target regression function:

[0130] a0+a1x+a2y+a3x 2 +a4xy+a5y 2 =z

[0131] 4) The parameters in the regression discriminant model can be solved by least square method by collecting multiple groups of different data offline.

[0132] When the parameterized trained discriminant model is used for online cable terminal posture adjustment, the side image feature data of the terminal obtained in real time by the binocular camera is used as input. The predicted angle value can be obtained through the model, and the rotation adjustment of the cable terminal is performed until the predicted angle is within the expected range.

[0133] The process of step S4 is as follows:

[0134] S41, picking up the cable after the posture adjustment is completed, and moving the robot to the plugging position near the target hole of the connector, and collecting images of the connector and the cable terminal in the clamping state through a micro camera installed on the clamp;

[0135] S42, using a template matching algorithm based on image grayscale correlation to identify and locate the connector and cable terminal in the image, and then binarizing the connector ROI image and the cable terminal ROI image respectively and extracting the central pixel point of the connected minimum outer enclosing rectangular frame and the top of the cable terminal, and finally calculating the image distance from the center point of the top of the cable terminal to the minimum outer enclosing rectangular frame of the connector, and using this as their image position feature data;

[0136] S43, by constructing and offline training an image-position visual mapping model based on support vector regression, taking the image feature data calculated in S42 as input, the model outputs the predicted robot position, and by driving the robot to the predicted position, recollecting the images of the connector and the cable terminal at the current position, thereby performing real-time wiring guidance;

[0137] S44, repeating step S42 and step S43 until the target image feature reaches within the preset expected error, that is, the closed-loop visual guidance of the wire insertion position is completed, and the final wire hole assembly operation is achieved.

[0138] Further, in S42, the method of calculating the image position relationship between the connector and the cable terminal in the image by using the connector image captured by the micro camera specifically includes:

[0139] 1) First, in the image captured by the micro camera at the end of the robotic arm, the area of ​​the connector and cable terminal is identified and located by a template matching method based on image grayscale;

[0140] 2) Secondly, the identified connector area is thresholded and converted into a binary image, and morphological opening and closing operations are performed on it, and its minimum outer bounding rectangle is extracted;

[0141] 3) Then, the identified cable terminal area is also subjected to threshold segmentation and morphological processing, and the ROI of the terminal tip is intercepted, and the centroid pixel of the part is calculated according to the central moment;

[0142] 4) Finally, with the centroid pixel point at the tip of the cable terminal as the center of the circle, n rays are emitted in the fan-shaped area of ​​the upper semicircle, and the distance from the center of each ray to the edge of the minimum outer enclosing rectangle of the connector is calculated and set to x i By using the above method, we can obtain the image feature vector X=(x1,x2,...,x n ), in this embodiment, n=3.

[0143] Furthermore, in S43, the construction of a visual mapping model based on support vector regression is as follows:

[0144]

[0145] Among them, K is the kernel function, s is the number of support vectors, and the input data x is the image position feature vector X=(x1,x2,...,x n ), and the output data f(x) is the corresponding robot posture.

[0146] For the visual mapping model based on support vector regression, the model parameter training steps are as follows:

[0147] 1) Sample data preparation: First, set the working distance of the micro camera to 3.5 cm to 4.5 cm, then set the range of different sampling densities. The closer to the target hole, the greater the sampling density. Finally, set the robotic arm to move with a fixed step length during the sampling process to facilitate the annotation of sample data.

[0148] 2) In the model training process, the acquired data samples and annotation information are processed, the image feature vector is extracted for each image sample, and the corresponding robotic arm posture information is saved;

[0149] 3) Read the obtained image feature vector and its corresponding robot arm posture data and matrix it to obtain the training data and the required format

[0150] 4) Build the designed visual mapping model based on the SVM framework in OpenCV4.1, train the SVR model in the SVM type selected by the model, and select the Gaussian kernel function. After setting the corresponding parameters, start automatic training until the error is less than the preset range or reaches the preset number of iterations.

[0151] Furthermore, this embodiment verifies the trained support vector regression-based visual mapping model and closed-loop visual guidance method. Figure 8 As shown in the figure, the closed-loop vision-guided wire insertion process specifically includes:

[0152] The cable terminal after posture adjustment is re-clamped, and the robot arm is moved to the preset position of the wire hole assembly area; the images of the connector and cable terminal are collected in real time by a micro camera, and their image feature vectors are calculated in real time; the image feature data is input into the support vector regression model, the predicted robot arm posture is output, and the posture is adjusted to achieve closed-loop vision guidance control until the error between the image feature and the target feature is within the expected value, and the robot arm posture adjustment is completed; finally, the wire hole is plugged in with the adjusted robot arm posture to complete the assembly guidance operation.

[0153] As shown in Table 1, this is the comparison result of the wire insertion operation guided by closed-loop vision and open-loop binocular stereo vision in this embodiment.

[0154] Table 1 Experimental results of wire hole mating under closed-loop vision guidance and open-loop vision guidance

[0155]

[0156] In Table 1, the average number of position adjustments refers to the number of position adjustments during the closed-loop vision guidance process. In this experiment, the termination iteration number is set to 4. The guidance process starts from the time the robot picks up the cable from the rotation auxiliary mechanism to the time the cable is inserted into the hole. The success rate is calculated by judging whether the terminal will touch the edge of the target hole to avoid successful entry into the hole by damaging the hole or the cable terminal structure. The experimental results in Table 1 show that although it takes more time under the guidance of closed-loop vision, the success rate can be much higher. Since inserting the wire into the hole is the last step in the wiring operation, the use of closed-loop vision guidance will have a higher success rate and efficiency in actual wiring tasks.

[0157] In summary, compared with the prior art, this embodiment has the following advantages and beneficial effects:

[0158] (1) According to the automated wiring task, a network model based on improved YOLOv5 is proposed to improve the detection capability of terminals. A cable segmentation algorithm combining terminal ROI information and color information is proposed, and the cable picking posture is estimated through a binocular camera.

[0159] (2) Aiming at the problem of adjusting the posture of cable terminals, a regression discriminant model based on multi-view feature learning is used to estimate the terminal angle and make adjustments, which solves the problem of adjusting the cable terminal angle.

[0160] (3) By designing and extracting the image information of the socket contour and the terminal in the image space as features, and training a mapping model with the robot arm posture, uncalibrated wire insertion visual guidance based on support vector regression is realized, thereby improving the success rate of wire hole assembly in the wire insertion operation.

[0161] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for automatic wiring based on visual guidance, characterized in that: The steps include: Establish the coordinate system relationship between the robot and the camera; Move the robot to the loading area, obtain the cable image in the loading area, calculate the cable picking posture, and drive the robot to clamp the cable; Obtain a side image of the cable terminal, build an angle prediction model for the cable terminal, use the image features of the local color histogram of the cable terminal as input, and the predicted angle as output, and adjust the posture of the cable terminal according to the predicted angle until the posture of the cable terminal is adjusted to a preset correct angle; The cable terminal adjusted to the preset correct angle is clamped, and the robot is moved to the wire insertion position to obtain the image of the target connector. By constructing a visual mapping model based on support vector regression, the positional relationship between the connector and the cable terminal in the target connector image is extracted as the image feature, and then the image feature is used as the input of the visual mapping model. The output of the model is to predict the target robot posture, adjust the robot posture, and complete the wire hole assembly; The mobile robot goes to the loading area, obtains the cable image in the loading area, calculates the cable picking posture, and drives the robot to clamp the cable, specifically: Collect images of cables in the loading area, and use the improved YOLOv5 algorithm to identify and locate the cable terminals in the images; After the cable terminal is identified and located, the ROI image of the target cable terminal in the image is intercepted, and the contour edge and the minimum outer bounding rectangle of the cable terminal are extracted through binarization and morphological processing, and then the central axis of the cable terminal is obtained; Perform color threshold segmentation on the original image of the collected cable, perform preliminary image segmentation on the cable body, and obtain candidate target areas; The central axis of the cable terminal and the candidate target area are obtained for AND operation. The retained pixels are used as seed pixels. The 8-connected domain growth algorithm is used to perform secondary segmentation of the cable body on the image to obtain the final target cable body area. According to the target cable area, a refinement algorithm is used to process it to obtain the center line of the line, and sampling is performed on the center line. The three-dimensional coordinates of the sampling points are obtained through the binocular camera triangulation algorithm. The sampling points are fitted with straight lines to obtain the cable picking posture. The robot is driven to move to the picking position, and the cable picking is completed by controlling the pneumatic gripper. The improved YOLOv5 algorithm includes an input end, a backbone network, a Neck network, an enhanced context module ECM structure and an output end; The ECM structure of the enhanced context module is as follows: Based on the receptive field module model, its branch structure is adjusted and the original convolution block is replaced with a double-hole convolution structure. At the same time, a single 3*3 convolution branch is used.

2. The method according to claim 1, characterized in that The side image of the cable terminal is obtained, and an angle prediction model of the cable terminal is constructed, the image features of the local color histogram of the cable terminal are used as input, and the predicted angle is used as output, and the posture of the cable terminal is adjusted according to the predicted angle until the posture of the cable terminal is adjusted to a preset correct angle, specifically: Straighten the cable body so that the cable terminal and the cable body are in the same straight line. At the same time, clamp and limit the cable so that it retains one degree of freedom, that is, it can only rotate around its own cable body as the axis; Collect the side image of the cable terminal, intercept the local ROI of the cable terminal in the image, track it using the KCF algorithm, extract the color histogram of the local ROI image, and use the Otsu method to segment the color histogram of the foreground area as image feature data; By constructing and offline training an angle prediction model for cable terminals, the image feature data is obtained as input, and the model predicts the target angle of the corresponding cable terminal. According to the target angle, the cable terminal is driven to rotate around its own wire body, thereby achieving angle adjustment of the cable terminal. When the cable terminal is adjusted to the correct angle, the adjustment is completed and the subsequent wire hole plug-in operation is carried out.

3. The method according to claim 1, characterized in that The cable terminal adjusted to the preset correct angle is clamped, and the robot is moved to the wire insertion position to obtain the image of the target connector. By constructing a visual mapping model based on support vector regression, the positional relationship between the connector and the cable terminal in the target connector image is extracted as an image feature, and then the image feature is used as the input of the visual mapping model. The output of the model is to predict the target robot posture, adjust the robot posture, and complete the wire hole assembly, specifically: The cable with posture adjustment is picked up again, and the robot is moved to the insertion position near the target hole of the connector to collect images of the connector and the cable terminal in the clamped state; Identify and locate the connector and cable terminal in the image, then perform binarization on the connector ROI image and the cable terminal ROI image respectively and extract the central pixel point of the connected minimum outer bounding rectangle and the top of the cable terminal. Finally, calculate the image distance from the center point of the top of the cable terminal to the minimum outer bounding rectangle of the connector, and use this as their image position feature data; By constructing and offline training a visual mapping model based on support vector regression, the calculated image feature data is used as input, and the model outputs the predicted robot position. By driving the robot to the predicted position, the images of the connector and cable terminals at the current position are recaptured, the target position is predicted again, and then the robot is moved again until the image position error is less than the expected error, thereby realizing closed-loop visual wire insertion guidance.

4. The method according to claim 3, characterized in that Template matching based on image grayscale correlation is used to identify and locate connectors and cable terminals.

5. A device for implementing the method according to any one of claims 1 to 4, characterized in that: It includes wiring operation execution part, visual inspection part and drive control part; The wiring operation execution part includes a two-dimensional motion platform, a rotation auxiliary mechanism, a six-degree-of-freedom robot and a clamp installed at the end of the robot; The rotation auxiliary mechanism is installed at the moving end of the two-dimensional motion platform. As the two-dimensional motion platform moves in the vertical and horizontal directions, the rotation auxiliary mechanism is used to adjust the posture of the cable terminal. The clamper is installed at the end of the robot and moves with the movement of the end of the robot to clamp the cables to be plugged in the loading area. The visual inspection part includes two industrial cameras, a miniature camera and an auxiliary light source. The two industrial cameras are installed in the clamp at the end of the robot to collect images of cables and connectors; the miniature camera is installed on the clamp to collect images of cable terminals and target plug-in holes in the clamping state during hole assembly operations; the auxiliary light source is installed in the clamp at the end of the six-degree-of-freedom industrial robot to adjust the light intensity and the ambient light around the image acquisition area; The drive control part is used to drive the two-dimensional motion platform to move and the gears inside the rotation auxiliary mechanism to rotate and drive the six-degree-of-freedom industrial robot to move to the target position; The rotation assist mechanism includes a metal base and a rotation assist mechanism body. The rotation assist mechanism body includes three complete gears, one incomplete gear, a limiting mechanism and a clamping mechanism. The limiting mechanism and the clamping mechanism pull the cable body to be plugged in into a straight state, and the rotation of the gear drives the cable terminal placed therein to rotate with its own cable body as the axis, thereby realizing the posture adjustment of the cable terminal.

6. The device according to claim 5, characterized in that The clamp comprises a pneumatic clamp and a metal connector, and the industrial camera is installed on the clamp at the end of the six-freedom industrial robot; the micro camera is installed on the clamp and is adjacent to the pneumatic clamp.

Images