An automatic wiring robot based on visual recognition and guidance

Abstract

The application provides an automatic wiring robot based on visual identification and guidance, and relates to the technical field of robots.The automatic wiring robot based on visual identification and guidance comprises an industrial camera and a mechanical arm, the industrial camera is installed at the end of the mechanical arm, the robot is an Eye-In-Hand hand-eye model, and comprises an ECT and an EBT, wherein the ECT is a transformation matrix of a tool coordinate system to a camera coordinate system and is determined through hand-eye calibration; the EBT is a transformation matrix between the tool coordinate system and a base coordinate system and is obtained through conversion calculation according to Euler angle information in a controller during work.Aiming at the demand of automatic wiring in the field of electric appliance manufacturing, an automatic wiring system based on an industrial robot and machine vision technology is designed and realized, through the design and experimental verification of key algorithms, the system can efficiently and accurately complete the identification, grasping, pose estimation and adjustment of a cable terminal and end visual guidance and other key links.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to an automatic wiring robot based on visual recognition and guidance. BACKGROUND

[0002] In the field of modern electrical appliance manufacturing, as the functions of products continue to expand, the complexity of internal circuits also increases. Traditional circuit connection mainly relies on manual operation, which is not only inefficient but also prone to errors, making it difficult to meet the needs of mass production.

[0003] In actual production applications, the commonly used cable types for wiring connection include DuPont wire, optical fiber, network cable, etc. Many research institutions have developed corresponding automatic wiring systems for these flexible cables. For example, the AFM device designed by FiberZone company can automatically complete the connection of optical fiber lines and automatically save the real-time accurate connection of optical fiber communication lines. However, this set of equipment has poor compatibility and expandability, and due to its large structure size, it is not suitable for assembly line production. The SYNDY system designed by SystemRobotAutomation company is mainly applied to the connection of wires in electrical equipment, and the wiring operation from end to end is realized through a mechanical hand. However, the end effector structure of this device is large, which limits its application in tasks with a large number of cables and high-density wiring.

[0004] The KieslingAverex wiring system developed by KieslingMaschinentechnik company is suitable for wiring operation of electrical equipment, but it needs to fix the direction of cable terminals and connectors and other plug-in components during operation, which limits its application in complex wiring connection. The zeta device of Komax company can efficiently complete functions such as stripping and plugging terminals, but the mechanism size is large, and the connector can only be fixed separately. The vision-based wiring system developed by KyongmoKoo et al. uses a laser sensor to obtain a point cloud model, finds the position and insertion direction of the hole, and guides the robot to complete the shaft-hole pairing work, mainly used for the assembly of automobile wiring harness.

[0005] In China, the research and development of wiring robots is still in its infancy, and only a few research units are involved. SimingXing et al. use a robot combined with visual technology to complete the connection of network cables, although the structure is simple, but the operation space is limited, and since there are only three degrees of freedom, the attitude of the curved cable cannot be adjusted, and the scope of application is limited. The automatic wiring system designed by Fei et al. keeps the double heads of the cable in a vertical state through the corresponding mechanism, realizing double-head wiring, but the feeding needs manual operation, which is low in efficiency. SUMMARY

[0006] The application aims to provide an automatic wiring robot based on visual recognition and guidance to solve the problems in the background art.

[0007] Technical scheme: In the first aspect, the application provides the following technical scheme: an automatic wiring method based on visual recognition and guidance, comprising the following steps:

[0008] Calibrate the camera imaging model and the camera and the hand-eye model, the camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane, and the hand-eye model is used to set the industrial robot base coordinate system as the world coordinate system in the camera imaging model;

[0009] The method based on direction feature template matching is used to verify and position the cable terminal after picking up, and the input image of the photographed cable terminal is feature extracted and similarity measured, the corresponding type of target terminal is obtained according to the template image, and the optimal matching perspective image of the cable terminal image to be identified is obtained;

[0010] The angle of the mechanical arm is estimated by using the calibrated coordinate system and the optimal matching perspective image of the cable terminal image to be identified, so as to realize the visual guidance of the mechanical arm and complete the wiring of the cable terminal.

[0011] Further, comprising:

[0012] The method based on direction feature template matching is used to verify and position the cable terminal after picking up, comprising: the camera photographs the cable terminal to be identified, and the direction features in the photographed image are extracted to realize the matching between the template image and the input image, so as to confirm whether the cable terminal is the target terminal, the template image is a pre-set image with multiple specific cable terminal forms, and the input image is the cable terminal image to be identified.

[0013] Further, comprising:

[0014] The input image of the photographed cable terminal is feature extracted and similarity measured, and the corresponding type of target terminal is obtained according to the template image, comprising:

[0015] When it is confirmed that the cable terminal is the target terminal, the camera takes a picture of the target terminal, calculates the gradient value of the color channel corresponding to the picture to form the gradient direction of each pixel, quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs, and quantizes the direction angle feature; the quantized gradient direction feature and the direction angle feature are associated to form two complementary feature sets, which collectively describe the image features of the target terminal; the image corresponding to the target terminal is divided into a plurality of grid units by grid division, each grid unit contains a certain number of pixels and corresponding feature information; in each grid unit, a local feature vector is extracted according to the quantized gradient direction feature and the direction angle feature; the similarity measurement method is used to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; the similarity between the target image and the template image is determined according to the similarity measurement result; and the target terminal corresponding to the target image is found from the template image according to the similarity value.

[0016] Further, comprising:

[0017] The optimal matching view image of the cable terminal image to be identified includes:

[0018] After finding the corresponding type of cable terminal, the camera takes multiple-angle pictures of the target terminal to obtain a series of multi-view multi-angle batch acquisition images, which should cover all views of the target terminal; representative, stable and distinguishable feature information is extracted from the above multi-view images and represented in the form of a vector; similarity is used to measure the similarity between two feature vectors, and a transformation matrix is solved by an image registration method to determine the position and direction of the images relative to the target image, and according to the similarity size, the candidate view images with higher similarity are retained, and finally, the optimal matching view image and its position are determined according to the similarity measurement and the orientation registration result.

[0019] Further, comprising:

[0020] The quantization processing of the calculated gradient direction includes: selecting pixels with a gradient amplitude greater than a certain fixed threshold in the gradient map, dividing the gradient vector corresponding to the selected pixels into 8 directions, and quantizing the direction as a byte string, with the position corresponding to the direction being 1 and the other bits being 0.

[0021] Further, comprising:

[0022] The angle of the mechanical arm is estimated using the calibrated coordinate system and the optimal matching view image of the cable terminal image to be identified, comprising:

[0023] Acquire target image data x, and use the data and the learning model to infer the global state w, which is the rotation angle to be estimated in the wiring task, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot predict the position or pose of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimally matched view image of the cable terminal obtained in the pose adjustment module.

[0024] Further, comprising:

[0025] The use of the calibrated coordinate system and the obtained optimal matching view image of the cable terminal to be identified to estimate the angle of the robot further includes positioning the local area of the cable terminal to be processed. The positioning method includes: first, positioning the entire cable terminal through the target recognition method on the obtained optimal matching view image of the cable terminal, then intercepting the required local area through position offset, the required local area includes terminal contact part, terminal fixed part and terminal feature area, then using a target tracker to track the local area in real time, the target tracker processes each frame of image in real time, and completes the positioning of the terminal local area without increasing the time cost.

[0026] Further, comprising:

[0027] The positioning of the local area of the cable terminal to be processed further includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracts the features of the required local area for each frame of image, and evaluates the similarity between the required local area and the target through the learned related filter, and selects the candidate area with the highest score as the position of the target according to the similarity result of the required local area.

[0028] Further, comprising:

[0029] Based on the feature information extracted from the obtained target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed, and in the tracking process, new target images are continuously collected, and the weight parameters of the related filter are updated.

[0030] Further, comprising:

[0031] The implementation of visual guidance to the robot to complete the wiring of the cable terminal includes: after the positioning of the required local area is completed, path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and the type, specification and other information of the terminal, a suitable wire harness is selected for connection; the wire harness is connected according to the planned path by using an automatic device.

[0032] In a second aspect, the application further provides an automatic wiring system based on visual recognition and guidance, which is obtained according to the wiring method, and specifically comprises a calibration module, a posture adjustment module and an angle estimation module.

[0033] The calibration module is used for calibrating a camera imaging model and a camera and a hand-eye model.

[0034] The posture adjustment module is used for verifying and positioning the cable terminal after picking up by using a method based on direction feature template matching, and performing feature extraction and similarity measurement on the input image of the photographed cable terminal, obtaining a target terminal of a corresponding type according to the template image, and obtaining an optimal matching visual angle image of the cable terminal image to be recognized.

[0035] The angle estimation module is used for estimating the angle of the mechanical arm by using the calibrated coordinate system and the optimal matching visual angle image of the cable terminal image to be recognized, so as to realize visual guidance of the mechanical arm and complete wiring of the cable terminal.

[0036] In a third aspect, the application further provides an automatic wiring robot based on visual recognition and guidance, which comprises an industrial camera and a mechanical arm.

[0037] Advantages: Compared with the prior art, the application has the following advantages:

[0038] The application designs and implements an automatic wiring system based on an industrial robot and machine vision technology to meet the demand for automatic wiring in the field of electric appliance manufacturing.

[0039] Firstly, the application not only considers the similarity between the target terminal image and the template image, so as to find the target terminal corresponding to the input image from the template image according to the similarity value, but also considers the detection and registration of the multi-view images of the target terminal, so as to determine the position and direction of the multi-view images relative to the target image, thereby realizing accurate registration of the images.

[0040] Secondly, in the angle estimation of the mechanical arm, not only the global state w, i.e. the rotation angle to be estimated, is estimated by using a regression prediction model, but also the positioning strategy for the local region of the cable terminal is considered when the positioning problem of the local region is encountered in the model training, so that the local region can be positioned more stably.

[0041] The corresponding experimental results of the application show that the robot can effectively improve the efficiency and accuracy of wiring work, reduce labor cost, and has important industrial application value. With the continuous progress of technology, automated wiring systems will play an increasingly important role in the electrical manufacturing industry, and promote the intelligent transformation of manufacturing industry. Future research can further optimize system performance, expand application range, and provide automated solutions for more types of circuit connections. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A camera imaging model diagram according to the application;

[0043] Figure 2 An Eye-In-Hand hand-eye model diagram according to the application;

[0044] Figure 3 An image feature extraction and similarity measurement flowchart according to the application;

[0045] Figure 4 A multi-view detection method flowchart according to the application. DETAILED DESCRIPTION

[0046] In order to enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0048] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not intended to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0049] In addition, in addition to indicating the orientation or positional relationship, the above-mentioned partial terms can also be used to indicate other meanings, for example, the term "upper" can also be used to indicate a certain dependent relationship or connection relationship in some cases. For those skilled in the art, the specific meaning of these terms in the present application can be understood according to the specific circumstances.

[0050] In addition, the terms "mount", "set", "provided with", "connected", "connected", "sleeved" should be broadly understood. For example, it can be fixedly connected, detachably connected, or integrally constructed; it can be mechanically connected or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium, or internal communication between two devices, elements or components. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0051] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0052] Embodiment 1

[0053] As shown in Figures 1-4 The present application provides an automatic wiring robot based on visual recognition and guidance, which comprises an industrial camera and a mechanical arm, the industrial camera is installed at the end of the mechanical arm;

[0054] The robot is an Eye-In-Hand hand-eye model, which comprises an ECT and an EBT, wherein the ECT is a transformation matrix from the tool coordinate system to the camera coordinate system, which is determined by hand-eye calibration; the EBT is a transformation matrix between the tool coordinate system and the base coordinate system, which is calculated by conversion according to the Euler angle information in the controller during operation;

[0055] The robot further comprises a calibration module, a posture adjustment module and an angle estimation module;

[0056] The calibration module is configured to calibrate a camera imaging model and a camera and hand-eye model, the camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane, and the hand-eye model is configured to set an industrial robot base coordinate system as a world coordinate system in the camera imaging model.

[0057] The posture adjustment module is configured to verify whether the cable terminal after picking up is the target terminal by using a direction feature template matching method, and to perform feature extraction and similarity measurement on an input image of the photographed cable terminal.

[0058] The angle estimation module is configured to estimate the angle of the mechanical arm by using the calibrated coordinate system and the position of the obtained optimal matching image, so as to realize visual guidance of the mechanical arm and complete wiring of the cable terminal.

[0059] The installation mode of the three modules is not limited in the embodiment, as long as the robot can realize the corresponding functions.

[0060] That is, the calibration module of the robot includes the following contents:

[0061] 1) Cable classification and pose estimation, including the following specific contents:

[0062] a) Spatial coordinate transformation model and calibration;

[0063] Determine the coordinate transformation relationship between different objects, including the tool coordinate system of the industrial robot base, the industrial camera imaging model, the binocular stereo vision model and the hand-eye model;

[0064] b) Camera imaging model and camera calibration;

[0065] The camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane, and the camera imaging model can be simplified as a pinhole imaging model, which involves four coordinate systems: a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system; wherein, Ow-XwYwZw represents the world coordinate system, which is set according to the application; the camera coordinate system Oc-XcYcZc takes the camera optical center as the origin, the camera optical axis as the Zc axis, and the Xc axis and the Yc axis are parallel to the X0 axis and the Y0 axis of the image coordinate system; as shown in Figure 1 The image coordinate system O0-X0Y0 takes the center of the imaging device plane a as the coordinate origin, and the X0 axis and the Y0 axis are parallel to the vertical edge and the horizontal edge of the plane a; the pixel coordinate system O-UV is also established on the plane a, and the coordinate origin is shifted to the upper left corner of the plane a, and the U axis and the V axis coincide with the edges of the imaging element; f represents the focal length of the camera;

[0066] c) Hand-eye model and calibration thereof;

[0067] As Figure 2As shown, in actual wiring operation, the spatial pose of the cable needs to be calculated, the binocular camera is installed on the end tool of the industrial robot to realize the calculation of the picking pose. Since the robot is involved, in order to facilitate the conversion between various coordinate systems, the base coordinate system of the industrial robot is set as the world coordinate system in the camera imaging model.

[0068] The pose adjustment module of the robot includes the following contents:

[0069] After the cable is picked up, in order to accurately insert the cable terminal into the corresponding hole, the pose of the clamped cable terminal needs to be adjusted to ensure that the terminal is inserted into the hole with the correct pose. Since the pose of the cable terminal is different in the initial feeding stage, it cannot be guaranteed that the terminal has the correct pose after being picked up by the mechanical arm, so secondary pose adjustment of the clamped cable terminal is required.

[0070] a) Cable terminal re-identification based on template matching;

[0071] After the cable terminal is placed on the rotating platform, in order to accurately extract the target feature and verify whether the picked cable terminal is the target terminal, the metal terminal part needs to be identified again. Considering that the object background is in a closed environment at this stage, the background interference is less and the target terminal does not need to be classified, from the perspective of wiring efficiency and stability, a method based on direction feature template matching is used to position the cable terminal, that is, the cable terminal to be identified is photographed, the direction feature in the image is extracted to realize the matching between the template image (pre-set image with multiple specific cable terminal shapes) and the input image (cable terminal image that needs to be identified), so as to confirm whether the above cable terminal is the target terminal. The core of this method is to convert the image into a directionally encoded form, thereby improving the efficiency and accuracy of matching.

[0072] When it is confirmed that the above cable terminal is the target terminal, the camera photographs the target terminal, and the gradient value of the color channel corresponding to the photograph is calculated to form the gradient direction of each pixel. In order to reduce the influence of noise, the gradient amplitude of each color channel of the input image is calculated for the input RGB image, the Sobel operator is used to calculate the gradient of each pixel of the image, and then the gradient direction of the channel with the largest gradient amplitude is taken as the gradient direction of the pixel, which is calculated as:

[0073]

[0074] In the formula, RGB is the color channel of the color image, represents the gradient direction of the pixel point at position x on each color channel, c x represents the maximum gradient direction.

[0075] b) image feature extraction and similarity measurement;

[0076] The specific steps of step b) are as follows:

[0077] Step b-1, gradient direction feature quantization: the calculated gradient directions are quantized, for example, the continuous gradient direction space is divided into 8 or more discrete direction intervals (or "bins"); the gradient direction of each pixel is classified into one of these direction intervals, thereby obtaining the quantized gradient direction feature;

[0078] Step b-2, direction angle feature calculation and quantization: in addition to the gradient direction feature, the direction angle feature between adjacent pixels or specific pixel pairs can also be calculated.

[0079] This embodiment does not limit the quantization of the direction angle feature, and the angle features can be quantized by calculating the cosine similarity between the image feature vectors for subsequent feature comparison and matching, specifically:

[0080] Cosine similarity is a measure of the consistency of the directions of two vectors. The features of an image can be represented as vectors, and the value range of cosine similarity is [-1, 1]. When the directions of two vectors are exactly the same, the cosine similarity is 1; when the directions of two vectors are exactly opposite, the cosine similarity is -1; when the two vectors are perpendicular, the cosine similarity is 0.

[0081] For two n-dimensional vectors A and B, their cosine similarity can be calculated by the following formula:

[0082] sim(A, B) = (A · B) / (‖A‖ * ‖B‖);

[0083] Where A · B represents the dot product of vectors A and B, and ‖A‖ and ‖B‖ represent the modulus (i.e. length) of vectors A and B, respectively. The cosine value of the angle between the two vectors in n-dimensional space is calculated.

[0084] Step b-3, feature association: the quantized gradient direction feature and the direction angle feature are associated to form two complementary feature sets, which together describe the image features of the target terminal and provide a basis for subsequent feature extraction and similarity measurement.

[0085] Step b-4, grid division: the image corresponding to the target terminal is divided into a grid, which is divided into multiple small regions, or "grid cells", each containing a certain number of pixels and corresponding feature information.

[0086] Step b-5, local feature vector extraction: Within each grid cell, local feature vectors are extracted based on the quantized gradient direction features and the direction angle features. These feature vectors are the key inputs for the subsequent similarity measurement.

[0087] Step b-6, similarity measurement: Using appropriate similarity measurement methods, the local feature vectors of each grid cell in the target image are compared with the local feature vectors of the corresponding grid cells in the template image. Based on the similarity measurement results, the similarity between the target image and the template image is determined. The specific similarity measurement method is not limited in this embodiment, such as cosine similarity, Euclidean distance calculation method can be used as the method in this embodiment.

[0088] Step b-7, result output: The similarity measurement results of the target image and multiple template images are output, which may include the similarity score of each grid cell, the overall similarity score or the number of successfully matched grid cells, etc. These information can be used for subsequent target detection, image recognition or image registration tasks.

[0089] As shown in Figure 3 The above process can realize accurate feature extraction and similarity measurement of the target cable terminal, so as to find the target terminal corresponding to the input image from the template image according to the similarity value. The method process is summarized as follows. This method only uses pixels with gradient amplitude greater than a certain small threshold, and divides their gradient vectors into 8 directions.

[0090] The quantized direction is represented as a byte string, with the position corresponding to the direction being 1 and the other bits being 0. In order to reduce the interference of background noise and small deformation, while retaining more local information, in this method, the tangent direction of the local contour is calculated, and the angle between this direction and the gradient direction of the position is also calculated. This angle is used as one of the new local edge description features, which retains some contour detail information and is also robust to rotation.

[0091] c) Multi-view detection and registration method;

[0092] After finding the corresponding type of cable terminal according to step b), multi-view detection and registration are performed. This method aims to capture the target image from all directions and angles through three-dimensional multi-view image acquisition model, and extract feature vectors from these images for similarity comparison with the multi-view image feature vector set, so as to screen out the optimal matching view image and its position. The specific steps are as follows:

[0093] Step c-1, three-dimensional multi-view image acquisition: using a three-dimensional multi-view image acquisition model, multi-angle shooting of the target terminal to be processed is performed, and a series of multi-view multi-angle batch acquisition images are obtained. These images should cover all possible angles of the target terminal to ensure that the subsequent feature extraction and matching process can be accurately performed.

[0094] The features contained in the single input template image in step b) are often limited. Although the fusion features proposed in the present application have certain robustness to translation, rotation and scaling, the initial pose of the cable terminal is not only uncertain but also has a large variation span. Therefore, if multiple templates are used for matching, the multi-view detection capability of each template also needs to be improved to meet the identification under different angles.

[0095] In order to improve the detection robustness of multi-view, on the basis of the foregoing fusion feature vector and identification method, a multi-view virtual sampling technology is introduced, and its process is as follows Figure 4 The content in the dashed box in the foregoing is consistent. For the input template image, different angle images are generated online according to the three-dimensional multi-view image sampling model, and the feature vectors of the sampling images under each angle are extracted to form a feature vector set for similarity calculation in the object detection stage. This method can effectively improve the detection capability of the target object subjected to affine deformation, and at the same time, a large amount of calculation cost is not introduced.

[0096] Step c-2, feature vector extraction and similarity measurement;

[0097] Step c-2-1, feature vector extraction for multi-view images: representative, stable and distinguishable feature information is extracted from multi-view images. In the form of a vector, it is used for subsequent similarity measurement and registration process.

[0098] Step c-2-2, feature type: in multi-view detection, common feature types include point features (such as corner points, edge points, etc.), line features (such as edge lines, contour lines, etc.) and region features (such as texture regions, shape regions, etc.). Various feature extraction algorithms are used, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), LBP (Local Binary Patterns) and the like.

[0099] Step c-2-3, extraction method: for point features, SIFT or SURF algorithms can be used to detect key points in the image and calculate the direction, scale and other information of these key points to form a feature vector.

[0100] For line features and region features, edge detection, texture analysis and other methods can be used to extract the corresponding feature information.

[0101] Step c-2-4 similarity measure: Similarity measure is a method to measure the degree of similarity between two feature vectors, through which it can be determined whether two feature vectors belong to the same object or scene, thereby achieving accurate image registration and object change detection.

[0102] Step c-2-5 measure method: Cosine similarity measures the degree of similarity between two vectors by calculating the cosine of the angle between them.

[0103] Step c-2-6 In multi-view image registration, cosine similarity is used to measure the degree of similarity between two feature vectors, and the transformation matrix is solved by RANSAC algorithm or the like to determine the position and direction of these images relative to the input image, thereby achieving accurate image registration; and according to the similarity size, the candidate view images with higher similarity are retained.

[0104] Step c-3, select the optimal matching image and position according to the orientation registration: According to the results of similarity measure and orientation registration, determine the optimal matching view image and its position. This step usually involves sorting and selecting multiple candidate view images to find the optimal matching view image of the cable terminal image that needs to be identified.

[0105] In this embodiment, the angle estimation module of the robot includes the following contents:

[0106] According to the calibration of the world coordinate system in the camera imaging model, and according to the image obtained based on three-dimensional multi-view image acquisition in step 2), the required rotation angle is inferred. Generally, the estimation of object pose is predicted by three-dimensional point cloud registration and regression estimation. Although the dual-camera mounted on the end effector of the wiring robot can generate three-dimensional image data, due to the small size of the cable metal terminal and the limited field of view of the dual-camera, there are differences in the images of the metal parts captured by the left and right cameras, resulting in many matching points that cannot be matched, causing a large amount of missing or inaccurate point cloud data, thereby affecting the accuracy of the point cloud data, making it difficult to predict the rotation angle through point cloud registration method;

[0107] However, after the introduction of the peripheral auxiliary rotating mechanism in the present application, the degree of freedom of the cable is reduced due to the introduction of the limiting device, and the pose estimation problem of the cable terminal is converted into the estimation problem of the axial angle, which simplifies the original three-dimensional problem into one-dimensional prediction output, greatly reducing the prediction difficulty, therefore, a simpler visual learning inference model is used to obtain the prediction value.

[0108] The core idea of visual learning of the present application is to obtain target image data x and use these data and a learning model to infer the global state w, which is the rotation angle that needs to be estimated in the wiring task. This is a continuous output value, so a regression prediction model needs to be constructed to help the robot predict the position or pose of the target object based on the output image or other features. The target image data x is the observed image feature value of the cable terminal.

[0109] Step 3-1 Data Collection: The three-dimensional multi-view images collected in the above steps, the feature vectors of the input images, the position or pose of the target terminal, and other data.

[0110] Step 3-2 Model Training: Use machine learning algorithms such as linear regression, support vector regression, neural networks, etc. to train the regression model. During the training process, the model continuously learns and adjusts parameters to minimize the error between the predicted value and the actual value. The predicted value is the minimum value of the model's prediction and inference based on the learned features and rules, and the actual value is the actual angle of the target terminal in the real environment.

[0111] Step 3-3 Prediction Application: In actual application, the robot will use the trained regression model to predict the position or pose of the target object based on the output image or other features. Then, the robot will adjust its position according to the prediction result to more accurately grasp or manipulate the target object.

[0112] Therefore, in summary, when designing the image feature description of the cable terminal, three aspects are usually considered: (1) the image feature data is easy to observe, i.e. it is obtained with less computational cost; (2) the image feature appears stably on the cable terminal, with stability and universality; (3) the image feature changes with the rotation angle of the cable terminal.

[0113] In order to further improve the accuracy of positioning, when the present application encounters the need for local area positioning in the above model training, in order to stably position to the local area, if template matching is continued, since the local area features are not obvious and easy to be confused with other places in the image, and the features will change with the angle when adjusting the rotation angle, multiple template matching is required, which increases the time cost and does not meet the requirements of the wiring task. Therefore, the positioning strategy for the local area of the cable terminal is:

[0114] The target recognition method is firstly used to locate the entire terminal in the three-dimensional multi-view collected image, and then the local area required, such as the terminal contact part, the terminal fixing part and the terminal characteristic area, is intercepted through position offset, and then a target tracker is used to track the local position in real time. The target tracker can process each frame of image in real time, and the positioning of the terminal local area is completed through the following steps without increasing the time cost.

[0115] The target recognition method is not limited in the application, and methods such as multi-view feature extraction, feature fusion and classification, training and optimization can be used.

[0116] In the embodiment, a terminal local area positioning method is provided, which specifically includes the following steps:

[0117] Step S3-1 image preprocessing: the three-dimensional multi-view collected image in step 2) is preprocessed, such as denoising and contrast enhancement, to improve the image quality. Image segmentation techniques such as threshold segmentation and edge detection are used to separate the terminal from the background.

[0118] Step S3-2 feature point detection and matching: feature points such as corner points and edge points are detected in the preprocessed image, which can uniquely represent the shape and position of the terminal. Feature matching algorithms such as SIFT, SURF and ORB are used to match the detected feature points with the feature points in the known terminal model.

[0119] Step S3-3 position and attitude estimation: according to the matched feature points, the position and attitude of the terminal in the image are calculated, including the rotation angle and translation amount. Perspective transformation and affine transformation are used to further accurately position and orient the terminal.

[0120] Step S3-4 post-processing and verification: the estimated position and attitude are post-processed, such as smoothing filtering and de-bouncing, to improve the stability and accuracy of the results. A verification data set is used to evaluate the performance of the positioning method, and adjustments and optimizations are made according to the evaluation results.

[0121] The regression prediction model is not limited in the application, but in the embodiment, KCF algorithm is used in the prediction process to improve the accuracy and real-time performance of terminal angle estimation. The basic idea of the KCF algorithm is: the target position P t in the previous frame F t is sampled for online training of a regressor, which predicts the response of small window sampling data. When the current frame F t+1 is read, the target position P tSurrounding multi-window sampling, and then using the regressor to predict each sample, and responding to the strongest area as the current frame target location P t+1 Finally, let the regressor re-learn the sampling data of this position in real time for the prediction of the next frame image.

[0122] The corresponding parameters include: regularization parameter: set the regularization parameter to control the complexity of the model to prevent overfitting; learning rate: set the learning rate to control the speed of model update in the online update phase.

[0123] And in this embodiment, multi-view fusion strategy can also be carried out, mainly including:

[0124] Feature extraction: in the regression prediction process, the HOG features, edge features, texture features, etc. extracted in the optimal matching image.

[0125] Feature mapping: use the kernel trick in KCF algorithm to map features, positions and poses.

[0126] Circulant matrix generation: based on the target terminal of each input image, use the circulant matrix generation method to generate multiple training samples. These samples include positive samples (i.e. base samples) and fictitious negative samples, which are used for subsequent correlation filter training.

[0127] Feature fusion: fuse the features in different views. Achieved by weighted average, cascading fusion or decision fusion, etc. The fused features will be used for subsequent angle estimation.

[0128] In summary, the terminal angle estimation based on multi-view feature learning in this embodiment includes the following steps: correlation filter training: set the size of the correlation filter. This usually depends on the size and shape of the target area. In terminal angle estimation, the size of the filter may need to be adjusted according to the size and shape of the terminal.

[0129] Candidate region evaluation: in the tracking stage, for each frame of image, extract the features of the target candidate region, and evaluate the similarity between the candidate region and the target through the learned correlation filter.

[0130] Angle estimation: according to the evaluation results of the candidate regions, select the candidate region with the highest score as the location of the target. Then, based on the feature information extracted from this position, use the regression model to estimate the terminal angle.

[0131] Model update: in the tracking process, new target images are constantly collected, and the weights of the correlation filter are updated. Ensure that the model can adapt to the changes of the target and improve the accuracy of angle estimation.

[0132] 4) The process of automatic wiring includes the following steps:

[0133] Step 4-1 path planning: After local positioning is completed, path planning is performed according to the positions of the terminals and the required connection relationship.

[0134] Step 4-2 harness selection: According to the path planning and the type, specification, etc. of the terminals, a suitable harness is selected for connection.

[0135] Step 4-3 automatic wiring: The harness is connected according to the planned path using automated equipment.

[0136] Step 4-4 quality detection: After wiring is completed, quality detection is performed to ensure that the connection is firm and the performance meets the requirements.

[0137] Example 2

[0138] The embodiment provides a technical scheme as follows: An automatic wiring method based on visual recognition and guidance, which comprises the following steps:

[0139] A camera imaging model and a camera and hand-eye model are calibrated, the camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane, and the hand-eye model is used to set an industrial robot base coordinate system as a world coordinate system in the camera imaging model;

[0140] A method based on directional feature template matching is used to verify and position the cable terminal after picking up, feature extraction and similarity measurement are performed on the input image of the photographed cable terminal, the corresponding type of target terminal is obtained according to the template image, and the optimal matching perspective image of the cable terminal image that needs to be identified is obtained;

[0141] The angle of the mechanical arm is estimated by using the calibrated coordinate system and the optimal matching perspective image of the cable terminal image that needs to be identified, so as to realize visual guidance of the mechanical arm and complete wiring of the cable terminal.

[0142] Further, the embodiment further comprises:

[0143] The method based on directional feature template matching is used to verify and position the cable terminal after picking up, comprising: a camera photographs the cable terminal to be identified, and matches the template image and the input image by extracting the directional features in the image after photographing, so as to confirm whether the above-mentioned cable terminal is a target terminal, the template image is a pre-set image with a plurality of specific cable terminal forms, and the input image is a cable terminal image that needs to be identified.

[0144] Further, the embodiment further comprises:

[0145] The feature extraction and similarity measurement of the photographed cable terminal input image are performed to obtain a corresponding type of target terminal according to a template image, and the method comprises the following steps of:

[0146] When it is confirmed that the cable terminal is the target terminal, the camera photographs the target terminal, calculates gradient values of color channels corresponding to the photographed image to form gradient directions of each pixel, quantizes the calculated gradient directions, calculates direction angle features between adjacent pixels or specific pixel pairs and quantizes the direction angle features, associates the quantized gradient direction features and the direction angle features to form two complementary feature sets which collectively describe image features of the target terminal, divides the image of the target terminal into a plurality of grid units by grid division, each grid unit containing a certain number of pixels and corresponding feature information, extracts local feature vectors in each grid unit according to the quantized gradient direction features and the direction angle features, compares the local feature vectors of each grid unit in the target image with the local feature vectors of the corresponding grid units in the template image by using a similarity measurement method, judges the similarity between the target image and the template image according to the similarity measurement result, and finds the target terminal corresponding to the target image from the template image according to the similarity value.

[0147] Further, the embodiment further comprises:

[0148] The optimal matching view image of the cable terminal image to be identified comprises the following steps of:

[0149] After the corresponding type of cable terminal is found, the camera photographs the target terminal from multiple angles to obtain a series of multi-view multi-angle batch acquisition images, and these images should cover all the views of the target terminal, the representative, stable and distinguishable feature information is extracted from the multi-view images and is represented in the form of a vector, the similarity is used to measure the similarity between two feature vectors, a transformation matrix is solved by using an image registration method to determine the position and direction of these images relative to the target image, and according to the similarity size, the candidate view images with high similarity are reserved, finally, the optimal matching view image and its position are determined according to the results of the similarity measurement and the orientation registration.

[0150] Further, the embodiment further comprises:

[0151] The quantization processing of the calculated gradient direction comprises the following steps of: selecting pixels with gradient amplitudes greater than a certain fixed threshold in the gradient map, dividing the gradient vectors corresponding to the selected pixels into 8 directions, and quantizing the direction to a string of one byte, the position corresponding to the direction is 1 and the other bits are 0.

[0152] Further, the embodiment further comprises:

[0153] The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image of the cable terminal image to be identified, comprising:

[0154] The target image data x is obtained, and the global state w is inferred by using the data and the learning model. In the wiring task, the global state w is the rotation angle to be estimated, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot arm predict the position or posture of the target object according to the output image or other features. The target image data x is the feature value corresponding to the optimal matching perspective image of the cable terminal obtained in the posture adjustment module.

[0155] The embodiment further comprises:

[0156] The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image of the cable terminal image to be identified, further comprising positioning the local area of the cable terminal to be processed. The positioning method comprises: first, positioning the entire cable terminal by using the target recognition method on the optimal matching perspective image of the cable terminal, then, intercepting the required local area by position offset, wherein the required local area includes the terminal contact part, the terminal fixed part and the terminal feature area, and then, using a target tracker to track the local area in real time. The target tracker processes each frame of image in real time, and completes the positioning of the local area of the terminal without increasing the time cost.

[0157] Further, the embodiment further comprises:

[0158] The positioning of the local area of the cable terminal to be processed further comprises: setting the size of the filter according to the size and shape of the required local area. The target tracker extracts the features of the required local area for each frame of image, and evaluates the similarity between the required local area and the target by using the relevant filter obtained by learning. According to the similarity result of the required local area, the candidate area with the highest score is selected as the position of the target.

[0159] Further, the embodiment further comprises:

[0160] Based on the feature information extracted from the obtained target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed. In the tracking process, new target images are continuously collected, and the weight parameters of the relevant filter are updated.

[0161] Further, the embodiment further comprises:

[0162] The implementation of visual guidance of the mechanical arm, the wiring of the cable terminal, comprises: after the positioning of the required local area is completed, the path planning is carried out according to the position of the target terminal and the required connection relationship; according to the path planning and the type, specification and other information of the terminal, the appropriate wire harness is selected for connection; the wire harness is connected according to the planned path by using the automatic equipment.

[0163] Embodiment 3

[0164] The application also provides an automatic wiring system based on visual recognition and guidance, which is obtained according to the wiring method, and specifically comprises: a calibration module, a posture adjustment module and an angle estimation module.

[0165] The calibration module is used for calibrating a camera imaging model and a camera and hand-eye model, the camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane, and the hand-eye model is used for setting an industrial robot base coordinate system as a world coordinate system in the camera imaging model.

[0166] The posture adjustment module is used for verifying and positioning the cable terminal after picking up by using a method based on direction feature template matching, extracting features and measuring similarity of the input image of the photographed cable terminal, obtaining the target terminal of the corresponding type according to the template image, and obtaining the optimal matching visual angle image of the cable terminal image to be identified.

[0167] The angle estimation module is used for estimating the angle of the mechanical arm by using the calibrated coordinate system and the optimal matching visual angle image of the cable terminal image to be identified, so as to realize the visual guidance of the mechanical arm and complete the wiring of the cable terminal.

[0168] Specifically, further comprising:

[0169] The method based on the direction feature template matching for verifying and positioning the cable terminal after picking up comprises: photographing the cable terminal to be identified by the camera, and matching the template image and the input image by extracting the direction features in the image after photographing, so as to confirm whether the cable terminal is the target terminal, the template image is a pre-set image with a plurality of specific cable terminal shapes, and the input image is the cable terminal image to be identified.

[0170] Further, the embodiment further comprises:

[0171] The feature extraction and similarity measurement of the input image of the photographed cable terminal, and obtaining the target terminal of the corresponding type according to the template image, comprise:

[0172] When the cable terminal is confirmed as the target terminal, the camera takes a picture of the target terminal, calculates the gradient value of the color channel corresponding to the picture to form the gradient direction of each pixel, quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs, and quantizes the direction angle feature; the quantized gradient direction feature and the direction angle feature are associated to form two complementary feature sets, which collectively describe the image features of the target terminal; the image corresponding to the target terminal is divided into a plurality of grid units by grid division, each grid unit containing a certain number of pixels and corresponding feature information; in each grid unit, a local feature vector is extracted according to the quantized gradient direction feature and the direction angle feature; the similarity measurement method is used to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; the similarity between the target image and the template image is determined according to the similarity measurement result; and the target terminal corresponding to the target image is found from the template image according to the similarity value.

[0173] Further, the embodiment further includes:

[0174] The optimal matching perspective image obtained from the image of the cable terminal to be identified includes:

[0175] After finding the corresponding type of cable terminal, the camera takes multiple-angle pictures of the target terminal to obtain a series of multi-perspective and multi-angle batch collection images, which should cover all perspectives of the target terminal; representative, stable and distinguishable feature information is extracted from the above multi-perspective images and represented in the form of a vector; the similarity is used to measure the similarity between two feature vectors, and the transformation matrix is solved by the image registration method to determine the position and direction of the images relative to the target image, and according to the similarity size, the candidate perspective images with higher similarity are retained, and finally, the optimal matching perspective image and its position are determined according to the similarity measurement and the orientation registration result.

[0176] Further, the embodiment further includes:

[0177] The quantization processing of the calculated gradient direction includes: selecting the pixels in the gradient map with a gradient amplitude greater than a certain fixed threshold, dividing the gradient vector corresponding to the selected pixels into 8 directions, and quantizing the direction as a string of one byte, with the position corresponding to the direction being 1 and the other bits being 0.

[0178] Further, the embodiment further includes:

[0179] The angle of the mechanical arm is estimated using the calibrated coordinate system and the optimal matching perspective image obtained from the image of the cable terminal to be identified, including:

[0180] The target image data x is obtained, and the global state w is inferred using the data and the learning model. In the wiring task, the global state w is the estimated rotation angle, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot predict the position or pose of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimally matched view image of the cable terminal obtained in the pose adjustment module.

[0181] Further, the embodiment also includes:

[0182] The use of the calibrated coordinate system and the obtained optimal matching view image of the cable terminal to estimate the angle of the robot includes positioning the local area of the cable terminal to be processed. The positioning method includes: first, positioning the entire cable terminal by the target recognition method on the obtained optimal matching view image of the cable terminal, then extracting the required local area by position offset, the required local area includes the terminal contact part, the terminal fixed part and the terminal feature area, then using the target tracker to track the local area in real time, the target tracker processes each frame of image in real time, and completes the positioning of the terminal local area without increasing the time cost.

[0183] Further, the embodiment also includes:

[0184] The positioning of the local area of the cable terminal to be processed also includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracts the features of the required local area for each frame of image, and evaluates the similarity between the required local area and the target by using the learned related filter, and selects the candidate area with the highest score as the position of the target according to the similarity result of the required local area.

[0185] Further, the embodiment also includes:

[0186] Based on the feature information extracted from the obtained target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed, and in the tracking process, new target images are continuously collected, and the weight parameters of the related filter are updated.

[0187] Further, the embodiment also includes:

[0188] The implementation of visual guidance for the robot to complete the wiring of the cable terminal includes: after the positioning of the required local area is completed, the path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and the type, specification and other information of the terminal, a suitable wire harness is selected for connection; the wire harness is connected according to the planned path by using the automatic equipment

[0189] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended that the scope of the application be limited to the detailed description contained herein or the specific examples described therein but rather that the scope of the application encompasses all modifications permissible under the patent statutes and to equitably apply the teachings of the application to the fullest extent possible.

[0190] The foregoing description, for purposes of clarity, describes only select embodiments of the application in detail so as to enable those skilled in the art to

[0191] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0192] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application.

[0193] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An automatic wiring method based on visual recognition and guidance, characterized in that: The method includes the following steps: The camera imaging model and the camera and hand-eye model are calibrated. The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot's base coordinate system as the world coordinate system in the camera imaging model. The method based on directional feature template matching is used to verify and locate the picked-up cable terminals. Feature extraction and similarity measurement are performed on the input image of the captured cable terminals. The target terminal of the corresponding type is obtained according to the template image, and the optimal matching view image with the image of the cable terminal to be identified is obtained. The angle of the robotic arm is estimated by using the calibrated coordinate system and the optimal matching view image with the cable terminal image to be identified, thereby realizing visual guidance of the robotic arm and completing the wiring of the cable terminal. The optimal matching viewpoint image obtained with the cable terminal image to be identified includes: After locating the corresponding type of cable terminal, the camera takes multiple photos of the target terminal from various angles to acquire a series of batch images from multiple perspectives. These images should cover all perspectives of the target terminal. Representative, stable, and distinguishable feature information is extracted from the above multi-view images and represented in vector form. Similarity is used to measure the degree of similarity between two feature vectors, and the transformation matrix is ​​solved by image registration method to determine the position and orientation of these images relative to the target image. Based on the similarity, candidate perspective images with high similarity are retained. Finally, based on the results of similarity measurement and orientation registration, the optimal matching perspective image and its position are determined. The method of estimating the angle of the robotic arm using the calibrated coordinate system and the optimal matching view image of the cable terminal to be identified also includes locating the local area of ​​the cable terminal to be processed. The locating method includes: first, locating the entire cable terminal using the optimal matching view image of the cable terminal obtained by the target recognition method; then, cropping the required local area by position offset. The required local area includes the terminal contact part, the terminal fixing part, and the terminal feature area. Then, a target tracker is used to track the local area in real time. The target tracker processes each frame of the image in real time to complete the local area of ​​the terminal without increasing the time cost. The localization of the cable terminal to be processed further includes: setting the size of the filter according to the size and shape of the required local area; the target tracker extracts the features of the required local area for each frame of image, and evaluates the similarity between the required local area and the target by using the learned correlation filter; and selects the candidate area with the highest score as the location of the target based on the similarity result of the required local area.

2. The automatic wiring method based on visual recognition and guidance according to claim 1, characterized in that: The method of verifying and locating the picked-up cable terminal using a directional feature template matching method includes: taking a picture of the cable terminal to be identified with a camera, and extracting directional features from the picture to match the template image with the input image, thereby confirming whether the cable terminal is the target terminal. The template image is a pre-set image with various specific cable terminal shapes, and the input image is the image of the cable terminal to be identified.

3. The automatic wiring method based on visual recognition and guidance according to claim 2, characterized in that: The process of extracting features and measuring similarity from the input image of the captured cable terminal, and obtaining the target terminal of the corresponding type based on the template image, includes: Once the cable terminal is confirmed to be the target terminal, the camera takes a picture of the target terminal and calculates the gradient value of the color channel corresponding to the captured image to form the gradient direction of each pixel. The calculated gradient direction is quantized, and the directional angle feature between adjacent pixels or specific pixel pairs is calculated and quantized. The quantized gradient direction feature and directional angle feature are correlated to form two complementary feature sets, which together describe the image features of the target terminal. The image corresponding to the target terminal is divided into grids, and the image is divided into multiple grid units. Each grid unit contains a certain number of pixels and corresponding feature information. Within each grid unit, local feature vectors are extracted based on the quantized gradient direction feature and directional angle feature. A similarity measurement method is used to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image. Based on the similarity measurement result, the degree of similarity between the target image and the template image is determined. Thus, the target terminal corresponding to the target image is found from the template image based on the similarity value.

4. The automatic wiring method based on visual recognition and guidance according to claim 3, characterized in that: The step of quantizing the calculated gradient direction includes: selecting pixels in the gradient map whose gradient magnitude is greater than a certain fixed threshold, dividing the gradient vector corresponding to the selected pixel into 8 directions, and representing the quantized direction as a one-byte string, with the corresponding position being 1 and the other bits being 0.

5. The automatic wiring method based on visual recognition and guidance according to claim 3 or 4, characterized in that: The estimation of the robotic arm's angle using the calibrated coordinate system and the optimal matching viewpoint image with the cable terminal image to be identified includes: Acquire target image data x And use this data and the learning model to infer the global state. w In wiring tasks, the global state w This refers to the rotation angle that needs to be estimated. This is a continuous output value. Therefore, a regression prediction model is constructed to help the robotic arm predict the position or orientation of the target object based on the output image or other features. x These are the feature values ​​corresponding to the optimal matching viewpoint image of the cable terminal obtained in the attitude adjustment module.

6. The automatic wiring method based on visual recognition and guidance according to claim 5, characterized in that: Based on the feature information extracted from the target location, the angle of the cable terminal to be processed is estimated using the trained regression prediction model. During the tracking process, new target images are continuously collected and the weight parameters of the relevant filters are updated.

7. The automatic wiring method based on visual recognition and guidance according to claim 6, characterized in that: The process of visually guiding the robotic arm to complete the wiring of cable terminals includes: after locating the required local area, performing path planning based on the position of the target terminal and the required connection relationship; selecting appropriate wire harnesses for connection based on the path planning and the type and specification information of the terminals; and using automated equipment to connect the wire harnesses according to the planned path.

8. An automatic wiring system based on visual recognition and guidance, characterized in that: The system is obtained by the wiring method according to any one of claims 1-7, specifically including: a calibration module, an attitude adjustment module, and an angle estimation module; The calibration module is used to calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system as the world coordinate system in the camera imaging model. The posture adjustment module is used to verify and locate the picked-up cable terminals using a method based on directional feature template matching, and to extract features and measure similarity of the input image of the captured cable terminals. Based on the template image, the module obtains the target terminal of the corresponding type and the optimal matching view image with the image of the cable terminal to be identified. The angle estimation module is used to estimate the angle of the robotic arm using the calibrated coordinate system and the optimal matching view image of the cable terminal image to be identified, thereby realizing visual guidance of the robotic arm and completing the wiring of the cable terminal.

9. An automated wiring robot based on visual recognition and guidance, characterized in that: The robot includes an industrial camera and a robotic arm. The camera is mounted at the end of the robotic arm and performs hand-eye calibration using an eye-in-hand method. The robot also employs the automatic wiring system based on vision recognition and guidance as described in claim 8 to achieve automatic wiring of the robotic arm.

Images