Automatic wiring robot based on visual identification and guidance

Abstract

The invention provides an automatic wiring robot based on visual identification and guidance, and relates to the technical field of robots. According to the automatic wiring robot based on visual identification and guidance, the robot comprises an industrial camera and a mechanical arm, and the industrial camera is installed at the tail end of the mechanical arm; the robot is an Eye-In-Hand hand-eye model and comprises an ECT and an EBT, and the ECT is a transformation matrix from a tool coordinate system to a camera coordinate system and is determined through hand-eye calibration; eBT is a transformation matrix between a tool coordinate system and a base coordinate system, and is obtained through conversion calculation according to Euler angle information in a controller during operation. The automatic wiring system based on an industrial robot and a machine vision technology is designed and realized aiming at the requirement of automatic wiring in the field of electric appliance manufacturing, and through design and experimental verification of a key algorithm, the system can efficiently and accurately complete identification, grabbing, pose estimation and adjustment of a cable terminal, so that the automation degree of the cable terminal is improved. And tail end visual guidance and other key links.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, in particular to an automatic wiring robot based on visual recognition and guidance. Background Art

[0002] In the field of modern electrical appliance manufacturing, as product functions continue to expand, the complexity of internal circuits has also increased. Traditional circuit connections mainly rely on manual operations, which is not only inefficient but also prone to errors and difficult to meet the needs of large-scale production.

[0003] In actual production applications, the types of cables commonly used for line connections include DuPont cables, optical fibers, network cables, etc. Many research institutions have developed corresponding automatic wiring systems for these flexible cables. For example, the AFM device designed by FiberZone can automatically complete the connection of optical fiber lines and automatically save the real-time and accurate connection status of optical fiber communication lines. However, the compatibility and scalability of this equipment are poor, and due to its large structure, it is not suitable for assembly line production. The SYNDY system designed by System Robot Automation is mainly used for wire connections in electrical equipment, and end-to-end wiring operations are achieved through manipulators. However, the end effector structure of the device is large, which limits its application in tasks involving large numbers of cables and high-density wiring.

[0004] The Kiesling Averex wiring system developed by Kiesling Maschinentechnik is suitable for wiring operations of electrical equipment, but the direction of plug-in parts such as cable terminals and connectors needs to be fixed during operation, which limits its application in complex line connections. Komax's zeta equipment can efficiently complete functions such as stripping and plugging terminals, but the mechanism size is large and the connector can only be fixed individually. The vision-based wiring system developed by Kyongmo Koo et al. uses a laser sensor to obtain a point cloud model, find the position and insertion direction of the hole, and guide the robot to complete the shaft-hole pairing work. It is mainly used for the assembly of automotive wiring harnesses.

[0005] In China, the research and development of wiring robots is still in its infancy, with only a few research units involved. Siming Xing and others used robots combined with visual technology to complete the wiring of network cables. Although the structure is simple, the operating space is limited, and since there are only three degrees of freedom, the posture of the bent cable cannot be adjusted, and the scope of application is limited. The automatic wiring system designed by Yu Fei and others uses the corresponding mechanism to keep the two ends of the cable in a vertical state, realizing double-end wiring, but the loading requires manual operation and is inefficient. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide an automatic wiring robot based on visual recognition and guidance to solve the problems raised in the above-mentioned background technology.

[0007] Technical solution: In the first aspect, the present invention provides the following technical solution: an automatic wiring method based on visual recognition and guidance, the method comprising the following steps:

[0008] Calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system as the world coordinate system in the camera imaging model.

[0009] The method based on directional feature template matching is used to verify and locate the picked-up cable terminals, and the input images of the captured cable terminals are subjected to feature extraction and similarity measurement. The corresponding type of target terminals are obtained according to the template image, and the image with the best matching viewing angle with the cable terminal image to be identified is obtained.

[0010] The angle of the robot arm is estimated using the calibrated coordinate system and the optimally matched viewing angle image of the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

[0011] Further, including:

[0012] The method based on directional feature template matching is used to verify and locate the cable terminal after picking, including: a camera takes a picture of the cable terminal to be identified, and matches the template image with the input image by extracting the directional features in the image after the photo is taken, so as to confirm whether the above-mentioned cable terminal is the target terminal, the template image is a pre-set image with a variety of specific cable terminal forms, and the input image is the cable terminal image that needs to be identified.

[0013] Further, including:

[0014] The feature extraction and similarity measurement of the input image of the captured cable terminal are performed to obtain a target terminal of a corresponding type according to the template image, including:

[0015] When it is confirmed that the cable terminal is the target terminal, the camera takes a photo of the target terminal, calculates the gradient value of the color channel corresponding to the photographed image to form the gradient direction of each pixel; quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs and quantizes it; associates the quantized gradient direction feature and direction angle feature to form two complementary feature sets, which jointly describe the image features of the target terminal; grids the image corresponding to the target terminal, and divides the image into multiple grid units, each grid unit containing a certain number of pixels and corresponding feature information; in each grid unit, extracts a local feature vector based on the quantized gradient direction feature and direction angle feature; uses a similarity measurement method to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; judges the similarity between the target image and the template image based on the similarity measurement result; thus, finds the target terminal corresponding to the target image from the template image based on the similarity value.

[0016] Further, including:

[0017] The step of obtaining the best matching viewing angle image with the cable terminal image to be identified includes:

[0018] After finding the corresponding type of cable terminal, the camera shoots the target terminal from multiple angles to obtain a series of multi-view and multi-angle batch acquisition images, which should cover all view angles of the target terminal; extract representative, stable and distinguishable feature information from the above multi-view images and express it in the form of vectors; use similarity to measure the similarity between two feature vectors, and solve the transformation matrix through the image registration method to determine the position and direction of these images relative to the target image, and retain the candidate view images with higher similarity according to the similarity size. Finally, determine the best matching view image and its position according to the results of similarity measurement and orientation registration.

[0019] Further, including:

[0020] The quantization processing of the calculated gradient direction includes: selecting pixels in the gradient map whose gradient amplitude is greater than a fixed threshold, dividing the gradient vector corresponding to the selected pixel into 8 directions, and the quantized direction is represented as a one-byte string, the position of the corresponding direction is 1, and the other bits are 0.

[0021] Further, including:

[0022] The method of estimating the angle of the robot arm by using the calibrated coordinate system and the obtained optimally matching viewing angle image with the cable terminal image to be identified includes:

[0023] The target image data x is obtained, and these data and the learning model are used to infer the global state w. In the wiring task, the global state w is the rotation angle that needs to be estimated, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot arm predict the position or posture of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimal matching perspective image of the cable terminal obtained in the posture adjustment module.

[0024] Further, including:

[0025] The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image obtained with the cable terminal image to be identified, and the local area of ​​the cable terminal to be processed is positioned. The positioning method adopted includes: firstly positioning the entire cable terminal by using the target recognition method to obtain the optimal matching perspective image of the cable terminal, and then intercepting the required local area by position offset, the required local area includes the terminal contact part, the terminal fixing part and the terminal feature area, and then using a target tracker to track the local area in real time online. The target tracker processes each frame of the image in real time to complete the positioning of the local area of ​​the terminal without increasing the time cost.

[0026] Further, including:

[0027] The method of locating the local area of ​​the cable terminal to be processed also includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracting the features of the required local area for each frame of the image, and evaluating the similarity between the required local area and the target through the learned correlation filter, and selecting the candidate area with the highest score as the position of the target based on the similarity result of the required local area.

[0028] Further, including:

[0029] Based on the feature information extracted from the target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed. During the tracking process, new target images are continuously collected and the weight parameters of the relevant filters are updated.

[0030] Further, including:

[0031] The method of achieving visual guidance of the robot arm to complete the wiring of the cable terminals includes: after the positioning of the required local area is completed, path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and information such as the type and specification of the terminal, a suitable wire harness is selected for connection; and the wire harness is connected according to the planned path using automated equipment.

[0032] In a second aspect, the present invention further provides an automatic wiring system based on visual recognition and guidance, which is obtained according to the wiring method described above, and specifically comprises: a calibration module, a posture adjustment module and an angle estimation module;

[0033] The calibration module is used to calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system to the world coordinate system in the camera imaging model.

[0034] The posture adjustment module is used to verify and locate the picked-up cable terminal by using a method based on directional feature template matching, and to extract features and measure similarity of the input image of the captured cable terminal, obtain the target terminal of the corresponding type according to the template image, and obtain the best matching perspective image with the cable terminal image to be identified;

[0035] The angle estimation module is used to estimate the angle of the robot arm using the calibrated coordinate system and the obtained optimal matching perspective image with the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

[0036] In a third aspect, the present invention also provides an automatic wiring robot based on visual recognition and guidance, the robot comprising: an industrial camera, a robotic arm, the camera being installed at the end of the robotic arm, hand-eye calibration being performed by an Eye-In-Hand method, and the automatic wiring system based on visual recognition and guidance being used to realize automatic wiring of the robotic arm.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0038] Aiming at the demand for automated wiring in the field of electrical manufacturing, the present invention designs and implements an automatic wiring system based on industrial robots and machine vision technology. Through the design and experimental verification of key algorithms, the system can efficiently and accurately complete key links such as cable terminal recognition, grasping, posture estimation and adjustment, and terminal vision guidance.

[0039] First of all, the present invention not only considers the similarity between the target terminal image and the template image, and finds the target terminal corresponding to the input image from the template image according to the similarity value, but also considers the detection and alignment of the multi-view images of the target terminal, so as to determine the position and direction of these multi-view images relative to the target image, thereby achieving accurate alignment of the images.

[0040] Secondly, in the angle estimation of the robot arm, the present invention not only considers the use of a regression prediction model to infer the global state, the global state w is the rotation angle that needs to be estimated, but also considers on this basis that when the positioning problem of the local area is encountered in the above-mentioned model training, the positioning strategy of the local area of ​​the cable terminal can obtain a more stable positioning of the local area.

[0041] The experimental results corresponding to the present invention show that the robot can effectively improve the efficiency and accuracy of wiring operations, reduce labor costs, and has important industrial application value. With the continuous advancement of technology, automated wiring systems will play an increasingly important role in the electrical manufacturing industry and promote the intelligent transformation of the manufacturing industry. Future research can further optimize system performance, expand the scope of application, and provide automated solutions for more types of circuit connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the camera imaging model of the present invention;

[0043] Figure 2 Schematic diagram of the Eye-In-Hand hand-eye model of the present invention;

[0044] Figure 3 This is a flow chart of image feature extraction and similarity measurement according to the present invention;

[0045] Figure 4 This is a flow chart of the multi-viewing angle detection method described in the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0049] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0050] In addition, the terms "installed", "set", "provided with", "connected", "connected", and "socketed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] Example 1

[0053] like Figure 1-Figure 4 As shown, an embodiment of the present invention provides an automatic wiring robot based on visual recognition and guidance, the robot includes an industrial camera and a mechanical arm, and the industrial camera is installed at the end of the mechanical arm;

[0054] The robot is an Eye-In-Hand hand-eye model, including ECT and EBT. ECT is the transformation matrix from the tool coordinate system to the camera coordinate system, which is determined by hand-eye calibration; EBT is the transformation matrix between the tool coordinate system and the base coordinate system, which is calculated and converted according to the Euler angle information in the controller during operation.

[0055] The robot also includes a calibration module, a posture adjustment module and an angle estimation module;

[0056] The calibration module is used to calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system to the world coordinate system in the camera imaging model.

[0057] The posture adjustment module is used to verify whether the picked-up cable terminal is the target terminal by using a method based on directional feature template matching, and to perform feature extraction and similarity measurement on the input image of the captured cable terminal;

[0058] The angle estimation module is used to estimate the angle of the robot arm by using the calibrated coordinate system and the position of the obtained optimal matching image, thereby realizing visual guidance of the robot arm and completing the wiring of the cable terminal.

[0059] This embodiment does not limit the installation methods of the above three modules, as long as the robot can achieve the corresponding functions.

[0060] That is, the robot calibration module includes the following:

[0061] 1) Cable classification and pose estimation, including the following specific contents:

[0062] a) Spatial coordinate transformation model and calibration;

[0063] Determine the coordinate transformation relationship between different objects, including the tool coordinate system of the industrial robot base, the industrial camera imaging model, the binocular stereo vision model and the hand-eye model;

[0064] b) Camera imaging model and camera calibration;

[0065] The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The camera imaging model can be simplified as a pinhole imaging model, involving four coordinate systems: the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. Among them, Ow-XwYwZw represents the world coordinate system, which is set according to the application. The camera coordinate system Oc-XcYcZc takes the camera optical center as the origin, the camera optical axis as the Zc axis, and the Xc axis and Yc axis are parallel to the X0 axis and Y0 axis of the image coordinate system, respectively. Figure 1 As shown; the image coordinate system O0-X0Y0 takes the center of the imaging component plane α as the coordinate origin, and the X0 axis and Y0 axis are parallel to the vertical side and horizontal side of the plane α respectively; the pixel coordinate system O-UV is also established on the plane α, and the coordinate origin is offset to the upper left corner of the plane α, and the U axis and V axis coincide with the edge of the imaging component; f represents the focal length of the camera;

[0066] c) Hand-eye model and its calibration;

[0067] like Figure 2As shown in the figure, in the actual wiring operation, it is necessary to calculate the spatial posture of the cable. The binocular camera is installed on the end tool of the industrial robot to realize the calculation of the picking posture. Since the robot is involved, in order to facilitate the conversion between various coordinate systems, the base coordinate system of the industrial robot is set as the world coordinate system in the camera imaging model.

[0068] The robot's posture adjustment module includes the following:

[0069] After the cable is picked up, in order to accurately insert the cable terminal into the corresponding hole, the cable terminal needs to be adjusted to ensure that the terminal is inserted into the hole with the correct posture. Since the postures of the cable terminals are different in the initial feeding stage, it is impossible to ensure that the terminals have the correct posture for inserting into the hole after being picked up by the robot arm. Therefore, the cable terminal needs to be adjusted for the second time after being clamped.

[0070] a) Cable terminal re-identification based on template matching;

[0071] After the cable terminal is placed on the rotating platform, in order to accurately extract the target features and verify whether the picked cable terminal is the target terminal, it is necessary to perform secondary identification on its metal terminal part. Considering that the object background is in a closed environment at this stage, there is less background interference and there is no need to classify the target terminal, from the perspective of wiring efficiency and stability, a method based on directional feature template matching is used to locate the cable terminal, that is, take a photo of the cable terminal to be identified, and extract the directional features in the image to achieve the matching between the template image (pre-set image with multiple specific cable terminal forms) and the input image (cable terminal image to be identified), so as to confirm whether the above cable terminal is the target terminal. The core of this method is to convert the image into a directional encoding form, thereby improving the efficiency and accuracy of matching.

[0072] When it is confirmed that the cable terminal is the target terminal, the camera takes a photo of the target terminal, and calculates the gradient value of the color channel corresponding to the photographed image to form the gradient direction of each pixel. In order to reduce the influence of noise, for the input RGB image, the gradient amplitude of each color channel of the input image is calculated respectively, and the Sobel operator is used to calculate the gradient of each pixel of the image. Then, the gradient direction of the channel with the largest gradient amplitude is used as the gradient direction of the pixel. The calculation is expressed as:

[0073]

[0074] In the formula, RGB is the color channel of the corresponding color image, Indicates the gradient direction of the pixel at position x on each color channel, c x Indicates the direction of the maximum gradient.

[0075] b) Image feature extraction and similarity measurement;

[0076] The specific steps of step b) are as follows:

[0077] Step b-1, quantizing gradient direction features: quantizing the calculated gradient direction, for example, dividing the continuous gradient direction space into 8 or more discrete direction intervals (or "boxes"); the gradient direction of each pixel will be classified into one of these direction intervals, thereby obtaining a quantized gradient direction feature;

[0078] Step b-2, calculation and quantification of directional angle features: In addition to the gradient directional features, the directional angle features between adjacent pixels or specific pixel pairs can also be calculated.

[0079] This embodiment does not limit the quantization of the directional angle feature. These angle features can be quantized by calculating the cosine similarity between image feature vectors to facilitate subsequent feature comparison and matching. Specifically:

[0080] Cosine similarity is a measure of the consistency of the directions of two vectors. The features of an image can be represented as vectors. The range of cosine similarity is [-1,1]. When the directions of the two vectors are exactly the same, the cosine similarity is 1; when the directions of the two vectors are completely opposite, the cosine similarity is -1; when the two vectors are perpendicular, the cosine similarity is 0.

[0081] For two n-dimensional vectors A and B, their cosine similarity can be calculated by the following formula:

[0082] sim(A,B)=(A·B) / (‖A‖*‖B‖);

[0083] Where A·B represents the dot product of vectors A and B, and ‖A‖ and ‖B‖ represent the modulus (i.e., length) of vectors A and B. Calculate the cosine of the angle between two vectors in n-dimensional space.

[0084] Step b-3, feature association: associate the quantized gradient direction features and direction angle features to form two complementary feature sets, which jointly describe the image features of the target terminal and provide a basis for subsequent feature extraction and similarity measurement.

[0085] Step b-4, grid division: grid division is performed on the image corresponding to the target terminal, and the image is divided into multiple small areas, or called "grid units", each grid unit contains a certain number of pixels and corresponding feature information.

[0086] Step b-5, local feature vector extraction: In each grid cell, local feature vectors are extracted based on the quantized gradient direction features and direction angle features. These feature vectors are the key inputs for subsequent similarity measurement.

[0087] Step b-6 Similarity measurement: Use an appropriate similarity measurement method to compare the local feature vector of each grid cell in the target image with the local feature vector of the corresponding grid cell in the template image; based on the similarity measurement result, determine the degree of similarity between the target image and the template image; this embodiment does not limit the specific similarity measurement method, such as cosine similarity and Euclidean distance calculation methods can be used as methods in this embodiment.

[0088] Step b-7, result output: Output the similarity measurement results between the target image and multiple template images, which may include the similarity score of each grid cell, the overall similarity score, or the number of successfully matched grid cells. This information can be used for subsequent tasks such as target detection, image recognition, or image registration.

[0089] like Figure 3 As shown, through the above process, accurate feature extraction and similarity measurement of the target cable terminal can be achieved, so as to find the target terminal corresponding to the input image from the template image according to the similarity value. The process of the method is summarized as follows. This method only uses pixels with gradient amplitude greater than a certain small threshold, and their gradient vectors are binned into 8 directions.

[0090] The quantized direction is represented as a one-byte string, with the corresponding direction position as 1 and the other bits as 0. In order to reduce the interference of background noise and small deformations while retaining more local information, the method in this paper calculates the tangent direction of the local contour and obtains the angle between the direction and the gradient direction of the position. This angle is used as one of the new local edge description features, which retains some contour details and is robust to rotation.

[0091] c) Multi-view detection and registration methods;

[0092] After finding the corresponding type of cable terminal according to step b), multi-view detection and registration are performed. This method aims to capture the target image in all directions and at multiple angles through a three-dimensional multi-view image acquisition model, extract feature vectors from these images, and compare the similarity with the multi-view image feature vector set, so as to screen out the best matching view image and its position. The specific steps are as follows:

[0093] Step c-1, 3D multi-view image acquisition: Use the 3D multi-view image acquisition model to shoot the target terminal to be processed from multiple angles to obtain a series of multi-view and multi-angle batch acquisition images. These images should cover all possible perspectives of the target terminal to ensure that the subsequent feature extraction and matching process can be carried out accurately.

[0094] The features contained in the single input template image in step b) are often limited. Although the fusion features proposed in the present invention have certain robustness to translation, rotation and scaling, the initial posture of the cable terminal is not only uncertain but also has a large variation span. Therefore, if multiple templates are used for matching, the multi-view detection capability of each template needs to be improved to meet the requirements of being able to complete recognition under different viewing angles.

[0095] In order to improve the robustness of multi-view detection, based on the aforementioned fusion feature vector and recognition method, multi-view virtual sampling technology is introduced. The process is as follows: Figure 4 The content of the dotted box in is consistent with that in . For the input template image, images under different perspectives are generated online according to the 3D multi-view image sampling model. Then, the feature vectors of the sampled images under each perspective are extracted to form a feature vector set for similarity calculation in the object detection stage. This method can effectively improve the detection capability of target objects that undergo affine deformation without introducing a large amount of computational cost.

[0096] Step c-2, feature vector extraction and similarity measurement;

[0097] Step c-2-1 extracts feature vectors from multi-view images: extracts representative, stable and distinguishable feature information from multi-view images and expresses it in the form of vectors for subsequent similarity measurement and registration processes.

[0098] Step c-2-2 Feature type: In multi-view detection, commonly used feature types include point features (such as corner points, edge points, etc.), line features (such as edge lines, contour lines, etc.) and region features (such as texture regions, shape regions, etc.). They are obtained through various feature extraction algorithms, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), LBP (Local Binary Patterns), etc.

[0099] Step c-2-3 Extraction method: For point features, algorithms such as SIFT or SURF can be used to detect key points in the image, and calculate the direction, scale and other information of these key points to form feature vectors.

[0100] For line features and area features, the corresponding feature information can be extracted through edge detection, texture analysis and other methods.

[0101] Step c-2-4 Similarity measurement: Similarity measurement is a method to measure the similarity between two feature vectors. Through similarity measurement, it can be determined whether the two feature vectors belong to the same object or scene, thereby achieving accurate image registration and object change detection.

[0102] Step c-2-5: Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them.

[0103] In step c-2-6, in multi-view image registration, cosine similarity is used to measure the similarity between two feature vectors, and the transformation matrix is ​​solved by algorithms such as RANSAC to determine the position and direction of these images relative to the input image, thereby achieving accurate image registration; and based on the size of the similarity, candidate view images with higher similarity are retained.

[0104] Step c-3, select the best matching image and position according to the orientation registration: Determine the best matching view image and its position according to the similarity measurement and the orientation registration results. This step usually involves sorting and selecting multiple candidate view images to find the best matching view image with the cable terminal image to be identified.

[0105] In this embodiment, the angle estimation module of the robot includes the following contents:

[0106] According to the world coordinate system calibration in the camera imaging model, the required rotation angle is inferred based on the image based on three-dimensional multi-view image acquisition obtained in step 2). Usually, the estimation of the object posture is predicted by three-dimensional point cloud registration and regression estimation. Although the binocular camera equipped with the end effector of the wiring robot can generate three-dimensional image data, due to the small size of the cable metal terminal and the limited field of view of the binocular camera, the metal part images collected by the left and right cameras are different, resulting in many matching points that cannot correspond, causing a large amount of point cloud data to be missing or inaccurately calculated, thereby affecting the accuracy of the point cloud data, making it difficult to predict the rotation angle through the point cloud registration method;

[0107] However, after the peripheral auxiliary rotation mechanism is introduced in this application, the degree of freedom of the cable is reduced due to the introduction of the limit device, and the posture estimation problem of the cable terminal is transformed into the axial angle estimation problem, which simplifies the original three-dimensional problem into a one-dimensional prediction output, greatly reducing the prediction difficulty. Therefore, a simpler visual learning reasoning model is used to obtain the prediction value;

[0108] The core idea of ​​visual learning in this application is to obtain target image data x and use these data and learning models to infer the global state w. In the wiring task, the global state w is the rotation angle that needs to be estimated. This is a continuous output value. Therefore, it is necessary to build a regression prediction model to help the robot arm predict the position or posture of the target object based on the output image or other features. The target image data x is the image feature value of the observed cable terminal;

[0109] Step 3-1 Data collection: The three-dimensional multi-view images collected in the above steps, the feature vector of the input image, the position or posture of the target terminal and other data.

[0110] Step 3-2 Model training: Use machine learning algorithms (such as linear regression, support vector regression, neural network, etc.) to train the regression model. During the training process, the model continuously learns and adjusts parameters to minimize the error between the predicted value and the actual value. The minimized predicted value is the minimum value that the model predicts and infers the input data based on the learned features and rules, and the actual value is the actual angle of the target terminal in the real environment.

[0111] Step 3-3 Prediction application: In actual applications, the robot arm will use the trained regression model to predict the position or posture of the target object based on the output image or other features. The robot arm will then adjust its position based on the prediction results to more accurately grasp or manipulate the target object.

[0112] Therefore, in summary, when designing the image feature description of the cable terminal, three aspects are usually considered: (1) the image feature data is easy to observe, that is, it is obtained with less computational cost; (2) the image features appear stably on the cable terminal, and are stable and universal; (3) the image features change with the rotation angle of the cable terminal.

[0113] In order to further improve the accuracy of positioning, when the present application encounters the problem of positioning a local area in the above-mentioned model training, in order to stably locate the local area, if the present application continues to use template matching, the local area features are not obvious and are easily confused with other places in the image, and the features will change with the angle when the rotation angle is adjusted. This requires multi-template matching, which increases the time cost and does not meet the requirements of the wiring task. Therefore, the positioning strategy for the local area of ​​the cable terminal in the present application is:

[0114] First, the target recognition method is used to locate the entire terminal in the three-dimensional multi-view image, and then the required local area is intercepted by position offset, such as the terminal contact part, the terminal fixed part, and the terminal feature area. Then, the target tracker is used to track the local position in real time online. The target tracker can process each frame of the image in real time and complete the positioning of the local area of ​​the terminal through the following steps without increasing the time cost.

[0115] This application does not limit the target recognition method, and methods such as multi-view feature extraction, feature fusion and classification, training and optimization can be used.

[0116] In this embodiment, a method for positioning a local area of ​​a terminal is provided, which specifically includes the following steps:

[0117] Step S3-1 Image preprocessing: Preprocess the image collected by three-dimensional multi-view in step 2), such as denoising, contrast enhancement, etc., to improve image quality. Use image segmentation technology (such as threshold segmentation, edge detection, etc.) to separate the terminal from the background.

[0118] Step S3-2 Feature point detection and matching: Detect feature points (such as corner points, edge points, etc.) in the preprocessed image. These feature points can uniquely represent the shape and position of the terminal. Use feature matching algorithms (such as SIFT, SURF, ORB, etc.) to match the detected feature points with the feature points in the known terminal model.

[0119] Step S3-3 Position and posture estimation: Calculate the position and posture (including rotation angle, translation, etc.) of the terminal in the image based on the matched feature points. Use geometric transformations such as perspective transformation and affine transformation to further refine the position and posture of the terminal.

[0120] Step S3-4 Post-processing and verification: Post-process the estimated position and attitude, such as smoothing filtering, de-jittering, etc., to improve the stability and accuracy of the results. Use the verification data set to evaluate the performance of the positioning method, and adjust and optimize it according to the evaluation results.

[0121] This application does not limit the above regression prediction model, but in this embodiment, in order to improve the accuracy and real-time performance of terminal angle estimation, the KCF algorithm is used in the prediction process. The basic idea of ​​the KCF algorithm is: t The target position P in the image t Feature sampling is performed to train a regressor online, which predicts the response of the small window sampled data in the current frame F t+1 When the image is read in, the target position P of the previous frame is tMulti-window sampling is performed around, and then the regressor is used to predict each sample, and the area with the strongest response is taken as the location of the target in the current frame P t+1 Finally, the regressor is allowed to relearn the sampled data at that position in real time for prediction of the next frame of image.

[0122] The corresponding parameters include: Regularization parameter: Set the regularization parameter to control the complexity of the model and prevent overfitting; Learning rate: In the online update phase, set the learning rate to control the speed of model update.

[0123] In this embodiment, a multi-view fusion strategy can also be performed, mainly including:

[0124] Feature extraction: In the regression prediction process, HOG features, edge features, texture features, etc. are extracted from the optimal matching image.

[0125] Feature mapping: Use the kernel technique in the KCF algorithm to map features, positions, and postures.

[0126] Circular matrix generation: Based on the target terminal of each input image, a number of training samples are generated using the circulant matrix generation method. These samples include positive samples (i.e., base samples) and fictitious negative samples for subsequent correlation filter training.

[0127] Feature fusion: Fuse features from different views. This is achieved through weighted averaging, cascade fusion, or decision fusion. The fused features will be used for subsequent angle estimation.

[0128] In summary, the terminal angle estimation based on multi-view feature learning in this embodiment includes the following steps: Correlation filter training: Setting the size of the correlation filter. This usually depends on the size and shape of the target area. In terminal angle estimation, the size of the filter may need to be adjusted according to the size and shape of the terminal.

[0129] Candidate region evaluation: In the tracking stage, for each frame of the image, the features of the target candidate region are extracted, and the similarity between the candidate region and the target is evaluated through the learned correlation filter.

[0130] Angle estimation: Based on the evaluation results of the candidate regions, the candidate region with the highest score is selected as the target location. Then, based on the feature information extracted from this location, a regression model is used to estimate the terminal angle.

[0131] Model update: During the tracking process, new target images are continuously collected and the weights of relevant filters are updated to ensure that the model can adapt to changes in the target and improve the accuracy of angle estimation.

[0132] 4) The automatic wiring process includes the following steps:

[0133] Step 4-1 Path planning: After the local positioning is completed, path planning is performed according to the location of the terminals and the required connection relationship.

[0134] Step 4-2 Wire harness selection: Select the appropriate wire harness for connection based on the path planning and terminal type, specifications and other information.

[0135] Step 4-3 Automatic wiring: Use automated equipment to connect the wire harnesses along the planned path.

[0136] Step 4-4 Quality inspection: After the wiring is completed, perform quality inspection to ensure that the firmness and performance of the connection meet the requirements.

[0137] Example 2

[0138] This embodiment provides the following technical solution: an automatic wiring method based on visual recognition and guidance, the method comprising the following steps:

[0139] Calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system as the world coordinate system in the camera imaging model.

[0140] The method based on directional feature template matching is used to verify and locate the picked-up cable terminals, and the input images of the captured cable terminals are subjected to feature extraction and similarity measurement. The corresponding type of target terminals are obtained according to the template image, and the image with the best matching viewing angle with the cable terminal image to be identified is obtained.

[0141] The angle of the robot arm is estimated using the calibrated coordinate system and the optimally matched viewing angle image of the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

[0142] Furthermore, this embodiment also includes:

[0143] The method based on directional feature template matching is used to verify and locate the cable terminal after picking, including: a camera takes a picture of the cable terminal to be identified, and matches the template image with the input image by extracting the directional features in the image after the photo is taken, so as to confirm whether the above-mentioned cable terminal is the target terminal, the template image is a pre-set image with a variety of specific cable terminal forms, and the input image is the cable terminal image that needs to be identified.

[0144] Furthermore, this embodiment also includes:

[0145] The feature extraction and similarity measurement of the input image of the captured cable terminal are performed to obtain a target terminal of a corresponding type according to the template image, including:

[0146] When it is confirmed that the cable terminal is the target terminal, the camera takes a photo of the target terminal, calculates the gradient value of the color channel corresponding to the photographed image to form the gradient direction of each pixel; quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs and quantizes it; associates the quantized gradient direction feature and direction angle feature to form two complementary feature sets, which jointly describe the image features of the target terminal; grids the image corresponding to the target terminal, and divides the image into multiple grid units, each grid unit containing a certain number of pixels and corresponding feature information; in each grid unit, extracts a local feature vector based on the quantized gradient direction feature and direction angle feature; uses a similarity measurement method to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; judges the similarity between the target image and the template image based on the similarity measurement result; thus, finds the target terminal corresponding to the target image from the template image based on the similarity value.

[0147] Furthermore, this embodiment also includes:

[0148] The step of obtaining the best matching viewing angle image with the cable terminal image to be identified includes:

[0149] After finding the corresponding type of cable terminal, the camera shoots the target terminal from multiple angles to obtain a series of multi-view and multi-angle batch acquisition images, which should cover all view angles of the target terminal; extract representative, stable and distinguishable feature information from the above multi-view images and express it in the form of vectors; use similarity to measure the similarity between two feature vectors, and solve the transformation matrix through the image registration method to determine the position and direction of these images relative to the target image, and retain the candidate view images with higher similarity according to the similarity size. Finally, determine the best matching view image and its position according to the results of similarity measurement and orientation registration.

[0150] Furthermore, this embodiment also includes:

[0151] The quantization processing of the calculated gradient direction includes: selecting pixels in the gradient map whose gradient amplitude is greater than a fixed threshold, dividing the gradient vector corresponding to the selected pixel into 8 directions, and the quantized direction is represented as a one-byte string, the position of the corresponding direction is 1, and the other bits are 0.

[0152] Furthermore, this embodiment also includes:

[0153] The method of estimating the angle of the robot arm by using the calibrated coordinate system and the obtained optimally matching viewing angle image with the cable terminal image to be identified includes:

[0154] The target image data x is obtained, and these data and the learning model are used to infer the global state w. In the wiring task, the global state w is the rotation angle that needs to be estimated, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot arm predict the position or posture of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimal matching perspective image of the cable terminal obtained in the posture adjustment module.

[0155] This embodiment also includes:

[0156] The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image obtained with the cable terminal image to be identified, and the local area of ​​the cable terminal to be processed is positioned. The positioning method adopted includes: firstly positioning the entire cable terminal by using the target recognition method to obtain the optimal matching perspective image of the cable terminal, and then intercepting the required local area by position offset, the required local area includes the terminal contact part, the terminal fixing part and the terminal feature area, and then using a target tracker to track the local area in real time online. The target tracker processes each frame of the image in real time to complete the positioning of the local area of ​​the terminal without increasing the time cost.

[0157] Furthermore, this embodiment also includes:

[0158] The method of locating the local area of ​​the cable terminal to be processed also includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracting the features of the required local area for each frame of the image, and evaluating the similarity between the required local area and the target through the learned correlation filter, and selecting the candidate area with the highest score as the position of the target based on the similarity result of the required local area.

[0159] Furthermore, this embodiment also includes:

[0160] Based on the feature information extracted from the target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed. During the tracking process, new target images are continuously collected and the weight parameters of the relevant filters are updated.

[0161] Furthermore, this embodiment also includes:

[0162] The method of achieving visual guidance of the robot arm to complete the wiring of the cable terminals includes: after the positioning of the required local area is completed, path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and information such as the type and specification of the terminal, a suitable wire harness is selected for connection; and the wire harness is connected according to the planned path using automated equipment.

[0163] Example 3

[0164] The present invention also provides an automatic wiring system based on visual recognition and guidance, which is obtained according to the wiring method described above, and specifically includes: a calibration module, a posture adjustment module and an angle estimation module;

[0165] The calibration module is used to calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system to the world coordinate system in the camera imaging model.

[0166] The posture adjustment module is used to verify and locate the picked-up cable terminal by using a method based on directional feature template matching, and to extract features and measure similarity of the input image of the captured cable terminal, obtain the target terminal of the corresponding type according to the template image, and obtain the best matching perspective image with the cable terminal image to be identified;

[0167] The angle estimation module is used to estimate the angle of the robot arm using the calibrated coordinate system and the obtained optimal matching perspective image with the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

[0168] Specifically, further, including:

[0169] The method based on directional feature template matching is used to verify and locate the cable terminal after picking, including: a camera takes a picture of the cable terminal to be identified, and matches the template image with the input image by extracting the directional features in the image after the photo is taken, so as to confirm whether the above-mentioned cable terminal is the target terminal, the template image is a pre-set image with a variety of specific cable terminal forms, and the input image is the cable terminal image that needs to be identified.

[0170] Furthermore, this embodiment also includes:

[0171] The feature extraction and similarity measurement of the input image of the captured cable terminal are performed to obtain a target terminal of a corresponding type according to the template image, including:

[0172] When it is confirmed that the cable terminal is the target terminal, the camera takes a photo of the target terminal, calculates the gradient value of the color channel corresponding to the photographed image to form the gradient direction of each pixel; quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs and quantizes it; associates the quantized gradient direction feature and direction angle feature to form two complementary feature sets, which jointly describe the image features of the target terminal; grids the image corresponding to the target terminal, and divides the image into multiple grid units, each grid unit containing a certain number of pixels and corresponding feature information; in each grid unit, extracts a local feature vector based on the quantized gradient direction feature and direction angle feature; uses a similarity measurement method to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; judges the similarity between the target image and the template image based on the similarity measurement result; thus, finds the target terminal corresponding to the target image from the template image based on the similarity value.

[0173] Furthermore, this embodiment also includes:

[0174] The step of obtaining the best matching viewing angle image with the cable terminal image to be identified includes:

[0175] After finding the corresponding type of cable terminal, the camera shoots the target terminal from multiple angles to obtain a series of multi-view and multi-angle batch acquisition images, which should cover all view angles of the target terminal; extract representative, stable and distinguishable feature information from the above multi-view images and express it in the form of vectors; use similarity to measure the similarity between two feature vectors, and solve the transformation matrix through the image registration method to determine the position and direction of these images relative to the target image, and retain the candidate view images with higher similarity according to the similarity size. Finally, determine the best matching view image and its position according to the results of similarity measurement and orientation registration.

[0176] Furthermore, this embodiment also includes:

[0177] The quantization processing of the calculated gradient direction includes: selecting pixels in the gradient map whose gradient amplitude is greater than a fixed threshold, dividing the gradient vector corresponding to the selected pixel into 8 directions, and the quantized direction is represented as a one-byte string, the position of the corresponding direction is 1, and the other bits are 0.

[0178] Furthermore, this embodiment also includes:

[0179] The method of estimating the angle of the robot arm by using the calibrated coordinate system and the obtained optimally matching viewing angle image with the cable terminal image to be identified includes:

[0180] The target image data x is obtained, and these data and the learning model are used to infer the global state w. In the wiring task, the global state w is the rotation angle that needs to be estimated, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot arm predict the position or posture of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimal matching perspective image of the cable terminal obtained in the posture adjustment module.

[0181] Furthermore, this embodiment also includes:

[0182] The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image obtained with the cable terminal image to be identified, and the local area of ​​the cable terminal to be processed is positioned. The positioning method adopted includes: firstly positioning the entire cable terminal by using the target recognition method to obtain the optimal matching perspective image of the cable terminal, and then intercepting the required local area by position offset, the required local area includes the terminal contact part, the terminal fixing part and the terminal feature area, and then using a target tracker to track the local area in real time online. The target tracker processes each frame of the image in real time to complete the positioning of the local area of ​​the terminal without increasing the time cost.

[0183] Furthermore, this embodiment also includes:

[0184] The method of locating the local area of ​​the cable terminal to be processed also includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracting the features of the required local area for each frame of the image, and evaluating the similarity between the required local area and the target through the learned correlation filter, and selecting the candidate area with the highest score as the position of the target based on the similarity result of the required local area.

[0185] Furthermore, this embodiment also includes:

[0186] Based on the feature information extracted from the target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed. During the tracking process, new target images are continuously collected and the weight parameters of the relevant filters are updated.

[0187] Furthermore, this embodiment also includes:

[0188] The method of achieving visual guidance of the robot arm to complete the wiring of the cable terminals includes: after the positioning of the required local area is completed, path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and information such as the type and specification of the terminal, a suitable wire harness is selected for connection; and the wire harness is connected according to the planned path using automated equipment.

[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

[0191] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0193] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An automatic wiring method based on visual recognition and guidance, characterized in that: The method comprises the following steps: Calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is the process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system as the world coordinate system in the camera imaging model. The method based on directional feature template matching is used to verify and locate the picked-up cable terminals, and the input images of the captured cable terminals are subjected to feature extraction and similarity measurement. The corresponding type of target terminals are obtained according to the template image, and the image with the best matching viewing angle with the cable terminal image to be identified is obtained. The angle of the robot arm is estimated using the calibrated coordinate system and the optimally matched viewing angle image of the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

2. The automatic wiring method based on visual recognition and guidance according to claim 1, characterized in that: The method based on directional feature template matching is used to verify and locate the cable terminal after picking, including: a camera takes a picture of the cable terminal to be identified, and matches the template image with the input image by extracting the directional features in the image after the photo is taken, so as to confirm whether the above-mentioned cable terminal is the target terminal, the template image is a pre-set image with a variety of specific cable terminal forms, and the input image is the cable terminal image that needs to be identified.

3. The automatic wiring method based on visual recognition and guidance according to claim 2 is characterized in that: The feature extraction and similarity measurement of the input image of the captured cable terminal are performed to obtain a target terminal of a corresponding type according to the template image, including: When it is confirmed that the cable terminal is the target terminal, the camera takes a photo of the target terminal, calculates the gradient value of the color channel corresponding to the photographed image to form the gradient direction of each pixel; quantizes the calculated gradient direction, calculates the direction angle feature between adjacent pixels or specific pixel pairs and quantizes it; associates the quantized gradient direction feature and direction angle feature to form two complementary feature sets, which jointly describe the image features of the target terminal; grids the image corresponding to the target terminal, and divides the image into multiple grid units, each grid unit containing a certain number of pixels and corresponding feature information; in each grid unit, extracts a local feature vector based on the quantized gradient direction feature and direction angle feature; uses a similarity measurement method to compare the local feature vector of each grid unit in the target image with the local feature vector of the corresponding grid unit in the template image; judges the similarity between the target image and the template image based on the similarity measurement result; thus, finds the target terminal corresponding to the target image from the template image based on the similarity value.

4. The automatic wiring method based on visual recognition and guidance according to claim 3 is characterized in that: The step of obtaining the best matching viewing angle image with the cable terminal image to be identified includes: After finding the corresponding type of cable terminal, the camera shoots the target terminal from multiple angles to obtain a series of multi-view and multi-angle batch acquisition images, which should cover all view angles of the target terminal; extract representative, stable and distinguishable feature information from the above multi-view images and express it in the form of vectors; use similarity to measure the similarity between two feature vectors, and solve the transformation matrix through the image registration method to determine the position and direction of these images relative to the target image, and retain the candidate view images with higher similarity according to the similarity size. Finally, determine the best matching view image and its position according to the results of similarity measurement and orientation registration.

5. The automatic wiring method based on visual recognition and guidance according to claim 3 is characterized in that: The quantization processing of the calculated gradient direction includes: selecting pixels in the gradient map whose gradient amplitude is greater than a fixed threshold, dividing the gradient vector corresponding to the selected pixel into 8 directions, and the quantized direction is represented as a one-byte string, the position of the corresponding direction is 1, and the other bits are 0.

6. The automatic wiring method based on visual recognition and guidance according to claim 4 or 5, characterized in that: The method of estimating the angle of the robot arm by using the calibrated coordinate system and the obtained optimally matching viewing angle image with the cable terminal image to be identified includes: The target image data x is obtained, and these data and the learning model are used to infer the global state w. In the wiring task, the global state w is the rotation angle that needs to be estimated, which is a continuous output value. Therefore, a regression prediction model is constructed to help the robot arm predict the position or posture of the target object based on the output image or other features. The target image data x is the feature value corresponding to the optimal matching perspective image of the cable terminal obtained in the posture adjustment module.

7. The automatic wiring method based on visual recognition and guidance according to claim 6 is characterized in that: The angle of the robot arm is estimated by using the calibrated coordinate system and the optimal matching perspective image obtained with the cable terminal image to be identified, and the local area of ​​the cable terminal to be processed is positioned. The positioning method adopted includes: firstly positioning the entire cable terminal by using the target recognition method to obtain the optimal matching perspective image of the cable terminal, and then intercepting the required local area by position offset, the required local area includes the terminal contact part, the terminal fixing part and the terminal feature area, and then using a target tracker to track the local area in real time online. The target tracker processes each frame of the image in real time to complete the positioning of the local area of ​​the terminal without increasing the time cost.

8. The automatic wiring method based on visual recognition and guidance according to claim 7 is characterized in that: The method of locating the local area of ​​the cable terminal to be processed also includes: setting the size of the filter according to the size and shape of the required local area, the target tracker extracting the features of the required local area for each frame of the image, and evaluating the similarity between the required local area and the target through the learned correlation filter, and selecting the candidate area with the highest score as the position of the target based on the similarity result of the required local area.

9. The automatic wiring method based on visual recognition and guidance according to claim 8, characterized in that: Based on the feature information extracted from the target position, the trained regression prediction model is used to estimate the angle of the cable terminal to be processed. During the tracking process, new target images are continuously collected and the weight parameters of the relevant filters are updated.

10. The automatic wiring method based on visual recognition and guidance according to claim 9, characterized in that: The method of achieving visual guidance of the robot arm to complete the wiring of the cable terminals includes: after the positioning of the required local area is completed, path planning is performed according to the position of the target terminal and the required connection relationship; according to the path planning and information such as the type and specification of the terminal, a suitable wire harness is selected for connection; and the wire harness is connected according to the planned path using automated equipment.

11. An automatic wiring system based on visual recognition and guidance, characterized in that: The system is obtained according to the wiring method according to any one of claims 1 to 10, and specifically comprises: a calibration module, a posture adjustment module and an angle estimation module; The calibration module is used to calibrate the camera imaging model and the camera and hand-eye model. The camera imaging model is a process of projecting a three-dimensional object in space onto a two-dimensional plane. The hand-eye model is used to set the industrial robot base coordinate system to the world coordinate system in the camera imaging model. The posture adjustment module is used to verify and locate the picked-up cable terminal by using a method based on directional feature template matching, and to extract features and measure similarity of the input image of the captured cable terminal, obtain the target terminal of the corresponding type according to the template image, and obtain the best matching perspective image with the cable terminal image to be identified; The angle estimation module is used to estimate the angle of the robot arm using the calibrated coordinate system and the obtained optimal matching perspective image with the cable terminal image to be identified, thereby achieving visual guidance of the robot arm and completing the wiring of the cable terminal.

12. An automatic wiring robot based on visual recognition and guidance, characterized in that: The robot comprises: an industrial camera and a robotic arm, wherein the camera is mounted at the end of the robotic arm, hand-eye calibration is performed by an Eye-In-Hand method, and the automatic wiring system based on visual recognition and guidance described in claim 11 is used to realize automatic wiring of the robotic arm.

Images