A soft wire assembly positioning method based on a 2D surface haptic twin network

Abstract

The application discloses a soft wire assembly positioning method based on a 2D surface haptic twin network, which is used for the assembly of a mobile phone soft wire, and the data of a haptic dot matrix is obtained by pressing a haptic sensor at an ideal buckling position, the haptic data information is taken as standard data information, and the haptic dot matrix data around the ideal buckling position is recorded; the correct haptic data information and the incorrect haptic data information are both converted into images, and the soft wire position calculation is regarded as an image registration problem between the correct haptic image and the incorrect haptic image; meanwhile, a deep twin neural network is provided, which extracts features of the haptic image and then calculates the ideal buckling position of the soft wire by using a regression network; the method can improve the assembly efficiency and buckling success rate in the assembly task of the mobile phone soft wire. The application can also be used in the assembly application when vision is blocked. The application can also be applied to other fields of robot precision assembly.

Description

Technical Field

[0001] This invention belongs to the field of precision robot assembly technology, and provides an artificial intelligence assembly method for 3C assembly applications, which is used for the rapid positioning and accurate fastening of mobile phone flexible cables in 3C assembly. Background Technology

[0002] Robotic intelligent assembly has been widely applied in various fields of industrial production, mainly for the manufacturing of various electrical appliances, the assembly of automotive parts, computers, toys, and various electromechanical products. For example, in the electrical appliance manufacturing and assembly industry, robotic intelligent assembly has basically replaced manual assembly, which not only improves assembly efficiency and quality but also saves manpower and reduces labor costs. With the continuous advancement and development of robotic intelligent assembly technology, intelligent assembly is gradually playing a role in more fields and is being used in various complex and precision assembly tasks (such as 3C assembly).

[0003] Tactile sensors are generally used to provide contact information about object surfaces. They have been widely applied in robotic assembly research, including object classification, wear defect detection, shape perception, and material roughness recognition. In addition, tactile sensors are used for tactile interaction in dexterous manipulation, shape perception, and tactile feedback-related robot operation strategies. In recent years, researchers have combined visual and tactile information in robotic assembly tasks, such as grasping unknown objects based on vision and touch, learning multimodal representations of vision and touch in robotic shaft and hole assembly tasks, classifying surface materials using tactile and visual information, and grasping tiny objects that are subject to soft magnetic interference with tactile perception. In these works, tactile information is typically treated as a one-dimensional sequential signal before being fused with visual information.

[0004] In most robotic assembly tasks, tactile sensors are fixed to the end effector of the robot and generate sufficient contact information upon contact with the target surface. Previous research has largely used one-dimensional signals to describe the contact information of tactile sensors. In this patent, we use two-dimensional (2D) tactile signals instead of one-dimensional ones. In mobile phone flexible cable assembly, the 2D image used to process tactile signals is called a tactile texture or tactile image. Compared to the raw 1D data, the tactile image provides both positional and intensity information around the touch point. Tactile images have been widely discussed in material classification, fabric defect detection, object orientation sensing, object shape sensing, surface roughness recognition, and grasping posture estimation. These traditional methods mostly employ deep neural networks (DNNs) as object classification or recognition models, where a single tactile image serves as input and the output is a category or range. Other methods combine a single tactile image and visual information into a feature vector for object classification or material separation. Unlike these previous methods, our goal is to retrieve the distance between the sensed flexible cable position and a reference flexible cable position, using an image registration method to solve this problem.

[0005] Image registration techniques were initially used to align two similar images: a reference image and a sensed image. It has been widely discussed in the computer vision community, including in object recognition, object tracking, and medical image analysis. Image registration techniques can be systematically classified into three categories: feature point matching, shape-based alignment, and deep learning methods. Early image registration techniques primarily relied on feature point registration and shape matching. However, due to the very small surface area of ​​mobile phone flexible flat cables and the sparse output of tactile sensor signals, with signal intensity periodically varying with touch position and direction, the tactile images sensed by the tactile sensor lack indistinguishable shapes or points. Therefore, these traditional point registration and shape matching methods are unsuitable for tactile image registration in mobile phone flexible flat cables. Image registration techniques based on deep Siamese neural networks (DSN) rely on two parallel and similar neural network architectures to jointly connect the reference image and the sensed image to an objective function. In the field of object tracking, the reference image can be a standard pattern or object image patch from a previous video frame, and the sensed image is from a subsequent video frame.

[0006] Because the robot's vision is easily obstructed when installing small parts, existing DSN target recognition and tracking models cannot achieve the high-precision positioning of 0.2mm required for flexible flat cable installation, and cannot handle the delicate operation of flexible flat cable fastening. Therefore, a DSN regression model is needed to learn the difference between a pair of tactile images: incorrect touch and correct touch, rather than a traditional object recognition model or heatmap-guided object tracking model. Summary of the Invention

[0007] To address the issue of mobile phone flexible flat cables being obstructed during robotic vision assembly due to their small size, this invention proposes a flexible flat cable assembly and positioning method based on a 2D surface tactile twin network. A tactile sensor is used to press at the ideal mating position to obtain a tactile signal, which is then used as a standard signal. The tactile signals surrounding the ideal mating position are also recorded. Both correct and incorrect tactile signals are converted into images, and the calculation of the flexible flat cable position is treated as an image registration problem between the correct and incorrect tactile images. A deep twin neural network is proposed, which extracts features from the tactile images and uses a regression network to calculate the ideal mating position of the flexible flat cable. This method can improve assembly efficiency and mating success rate in mobile phone flexible flat cable assembly tasks; it can be used in assembly applications where vision is obstructed, and can also be applied to other fields of robotic precision assembly.

[0008] This invention relates to a flexible flat cable assembly and positioning method based on a 2D surface tactile twin network. By designing a tactile operation module and combining it with a tactile sensor, tactile data at different locations is collected. Furthermore, by combining a method for converting tactile information into images with an assembly skill learning method based on a deep twin neural network, flexible flat cable assembly and positioning is achieved.

[0009] The tactile data at different positions includes: tactile signals at the correct position between the tactile sensor and the flexible cable, and error tactile signals at the error position where the randomly moving robotic arm around the correct position reaches the error position.

[0010] The method for converting tactile information into images involves filtering the collected tactile data and then converting it into an image.

[0011] The assembly skill learning method based on a deep Siamese neural network takes as input images corresponding to correct tactile data, images corresponding to incorrect tactile data, and offsets Δx, Δy, and Δθ of the incorrect position relative to the correct position along the x-axis, y-axis, and around the z-axis. The method obtains predicted offsets for the three degrees of freedom from the two input images, and uses the squared difference between these offsets and the predicted offsets as a loss function for training, making the predicted distance values ​​closer to the true offsets.

[0012] Finally, the tactile image of the correct position input by the autoencoder as the backbone network and the real-time tactile information are used to extract features respectively. The extracted n-dimensional feature vectors are concatenated and then connected to three fully connected layers. Finally, the output is the error of movement and rotation along the x-axis and y-axis.

[0013] The advantages of this invention are:

[0014] 1. The tactile positioning method in this invention solves the problem of visual sensors being blocked during assembly, enabling the robot to accurately position dynamic mobile phone flexible flat cables. This invention improves the robot's recognition accuracy of mobile phone flexible flat cables, meeting the high-precision requirements of 3C assembly; it also improves the assembly efficiency and success rate of mobile phone flexible flat cables, solving problems such as low robot positioning efficiency and high assembly difficulty.

[0015] 2. The method of this invention utilizes the high precision and low latency of tactile sensors to achieve rapid and accurate online compensation for assembly errors. It obtains the displacement during the snap-fit ​​process of the mobile phone's flexible flat cable through tactile detection, and also has the function of detecting robot pressing pressure. On the one hand, it achieves real-time compensation of the displacement detected by tactile sensing; on the other hand, it detects the magnitude of the pressing pressure, which can reduce damage to the flexible flat cable during the snap-fit ​​process and prevent cost losses and machine damage caused by collisions between assembled parts. Attached Figure Description

[0016] Figure 1 This is a flowchart of the flexible flat cable assembly and positioning method based on a 2D surface tactile twin network according to the present invention.

[0017] Figure 2 This is a tactile learning model based on a deep Siamese neural network. Detailed Implementation

[0018] This invention relates to a flexible flat cable assembly and positioning method based on a 2D surface tactile twin network, which involves one functional module and two methods. The functional module is a tactile operation module, used to collect tactile data at different locations in conjunction with tactile sensors (step S1 in the method section below). The two methods are a method for converting tactile information into images (step S2 in the method section below) and an assembly skill learning method based on a deep twin neural network (steps S3 and S4 in the method section below).

[0019] Based on the above-mentioned tactile operation module and the two methods, the present invention forms a soft flat cable assembly and positioning method based on a 2D surface tactile twin network. The specific steps are as follows:

[0020] S1: Collect tactile data in the mobile phone flexible flat cable assembly scenario and create a training dataset.

[0021] The tactile sensor used is a pressure-sensitive sensor with 4*11 tactile sensing units distributed on it. To select more effective tactile signals, s tactile data are collected at each acquisition position of the tactile sensor, where s > 1. The tactile sensor, in conjunction with the tactile operation module, collects tactile data from different positions. The acquisition process is as follows: the user operates the robotic arm to the ideal position for fastening the mobile phone's flexible flat cable. At this point, the tactile sensor is in the correct position between itself and the flexible flat cable, and the sensor collects s ​​tactile signals at this correct position.

[0022] Subsequently, the robotic arm is randomly moved around the correct position to another position (error position), and tactile signals are collected s times by the tactile sensor. These tactile signals are the error tactile signals.

[0023] Furthermore, record the offsets Δx, Δy, and Δθ of the current error position relative to the correct position on the x-axis, y-axis, and around the z-axis.

[0024] The collected correct position tactile signals and error tactile signals, as well as the offset of the current error position relative to the correct position in three degrees of freedom, are used as the data required for training the network, thus obtaining the training dataset.

[0025] During the above data collection process, the tactile operation module acquired two pieces of information:

[0026] 1. The contact signal between the tactile sensor and the mobile phone's flexible cable at the current error location is detected by the tactile sensor, which is the aforementioned error tactile signal;

[0027] 2. The offsets Δx, Δy, and Δθ of the current error position relative to the correct position on the x-axis, y-axis, and around the z-axis.

[0028] In this embodiment, the robotic arm is a collaborative robotic arm. The mobile phone flexible cable is a flat and flexible part, which is not suitable for common end effectors such as grippers. Therefore, a suction cup is used to pick it up, and a tactile sensor is fixed at the end of the actuator to perform the data sensing task after pressing.

[0029] S2: Preprocess the one-dimensional tactile data in the tactile data set established in step 1 and convert it into tactile images.

[0030] Considering that each tactile sensing unit of the tactile sensor may contain white noise and flicker noise during the recording period, a moving average window with a step size of L is used to filter the signal of each sensor's tactile sensing unit. Besides using a moving average window filter, other filtering methods are also applicable to this patent, such as 3*3 mean filters, 3*3 median filters, and Kalman filters to filter noise in the tactile sensor signal.

[0031] After filtering and preprocessing the s tactile data collected from the 44 tactile sensing units on the tactile sensor, the resulting 44 preprocessed data are converted into grayscale images according to the following formula:

[0032] F max =MAX{F1,F2,F3,…F n}, n=44

[0033] F min =MIN{F1,F2,F3,…F n}, n=44

[0034]

[0035] In the formula, F i This is the force value of each tactile sensing unit of the tactile sensor obtained after filtering, where i = 1 to n, F max With F min G represents the maximum and minimum forces. i It is the grayscale value of the corresponding tactile sensing unit in the image.

[0036] To facilitate DSN training, all 4*11 tactile images are converted into m*m images, where m represents the image size, typically 256 or 224.

[0037] S3: The collected tactile data is processed through a deep twin neural network for feature extraction and training.

[0038] Figure 2 A deep twin neural network (DSN) haptic learning model is presented, with ResNet50 as the backbone. The core structure of this model is a deep twin neural network composed of an autoencoder and a regression module. The two channels of this deep twin neural network have identical network structures and share the same network parameters. Unlike traditional DSN target recognition or tracking models, this model uses a CNN-based autoencoder to synchronously encode real-time haptic images and reference haptic images. The autoencoder converts each m*m image into a feature vector of length n, combines them into a one-dimensional feature vector, and inputs it into the regression module. For training the network structure, the input during the training phase consists of the grayscale images corresponding to the correct haptic data and the grayscale images corresponding to the error haptic data obtained in step S2. The input also includes the recorded offsets of the three degrees of freedom. The predicted offsets of the three degrees of freedom are obtained from the two input images, and the squared difference between these predicted and recorded offsets is used as the loss function for training, making the predicted distance values ​​closer to the true offsets.

[0039] Therefore, as Figure 2As shown, the tactile image (correct sample data) at the correct position, which is used as the input of the autoencoder (such as AlexNet, VGG16, ResNet50, etc.) to the backbone network, and the real-time tactile information (acquired sample data) are used to extract features respectively. The extracted n-dimensional feature vectors are concatenated and then connected to three fully connected layers. Finally, the output is the error that needs to be moved along the x-axis and y-axis and the rotation error.

[0040] S4: Perform image registration between the sensed tactile image and the tactile image at the reference position.

[0041] Using the deep Siamese neural network trained in step S3, the robot predicts the position differences Δx, Δy, and Δθ between the current position (sensing) and the ideal position (reference) of the flexible cable from an image pair. The robot then uses a... t =a t-1 After -f(Δx, Δy, Δθ) moves to a new position, the robotic arm will again obtain a new offset through the tactile sensor. This process is iterated until Ω(Δx, Δy, Δθ) == TRUE is satisfied.

[0042] Among them, a t =a t-1 -f(Δx, Δy, Δθ) represents the robot's next position, a t a t-1 is the position of the robotic arm at times t, t-1; f() is a fixed matrix used to transfer the motion distance measured in the tactile image plane to the robot's motion space; Ω() refers to the set of offsets.

[0043] If Ω(Δx,Δy,Δθ)==TRUE, then the sensed tactile image can be considered well aligned with the tactile image at the ideal position. Set the maximum permissible error d for the x-axis, y-axis, and θ. x d y d θ Given angles of 0.2mm, 0.2mm, and 1.5° respectively, we have:

[0044]

[0045] Based on the above method, the specific process for completing the assembly task in the mobile phone flexible flat cable assembly scenario is as follows:

[0046] A. Use visual inspection to identify mobile phone flexible cables on the material table;

[0047] B. The robotic arm precisely picks up the center position of the mobile phone flexible flat cable on the material table;

[0048] C. Place the mobile phone flexible cable on the secondary positioning platform;

[0049] D. Use an RGB camera to detect the snap-fit ​​position, and move the robotic arm to the approximate snap-fit ​​position of the assembly line mobile phone stand;

[0050] E. Press the male connector of the flexible ribbon cable that needs to be fastened with the tactile sensor at the approximate fastening position;

[0051] F. Obtain the tactile image of the press to estimate the distance between the accurate location and the current location;

[0052] G. The robotic arm moves again to the secondary positioning stage to pick up the mobile phone's flexible cable;

[0053] H. Based on the distance calculated by E, move the flexible flat cable to the correct calculated position and fasten the flexible flat cable.

Claims

1. A method for assembling and positioning flexible flatbed cables based on a 2D surface tactile twin network, characterized in that: By designing a tactile operation module and combining it with a tactile sensor, tactile data from different locations is collected. Furthermore, by combining a method for converting tactile information into images with an assembly skill learning method based on a deep twin neural network, soft flat cable assembly positioning is achieved. The tactile data at different positions includes: tactile signals at the correct position between the tactile sensor and the flexible cable, and error tactile signals at the error position where the randomly moving robotic arm around the correct position reaches the error position; The method for converting tactile information into images involves filtering the collected tactile data and then converting it into an image. The assembly skill learning method based on deep Siamese neural network takes as input the image corresponding to the correct tactile data, the image corresponding to the incorrect tactile data, and the offsets Δx, Δy, and Δθ of the error position relative to the correct position on the x-axis, y-axis, and around the z-axis. The method obtains the predicted offsets of the three degrees of freedom from the two input images, and uses the squared difference of the offsets of the three degrees of freedom as the loss function for training, so that the predicted distance value is closer to the true offset. Finally, the tactile image of the correct position input by the autoencoder as the backbone network and the real-time tactile information are used to extract features respectively. The extracted n-dimensional feature vectors are concatenated and then connected to three fully connected layers. Finally, the output is the error of movement and rotation along the x-axis and y-axis.

2. The method for assembling and positioning flexible flatbed cables based on a 2D surface tactile twin network as described in claim 1, characterized in that: The specific steps are as follows: S1: Collect tactile data in the mobile phone flexible flat cable assembly scenario and create a training dataset; The robotic arm is moved to the ideal position for fastening the mobile phone's flexible flat cable. At this point, the tactile sensor is in the correct position between the tactile sensor and the flexible flat cable. The tactile sensor collects s ​​tactile signals at its correct position, where s > 1. Subsequently, the robotic arm is randomly moved around the correct position to the error position, and tactile signals are collected s times by the tactile sensor. These tactile signals are the error tactile signals. Furthermore, record the offsets Δx, Δy, and Δθ of the current error position relative to the correct position on the x-axis, y-axis, and around the z-axis; The correct position tactile signal and the error tactile signal collected above, as well as the offset of the current error position relative to the correct position in three degrees of freedom, are used as the data required to train the network, and the training dataset is obtained. S2: Filter the one-dimensional tactile data in the tactile data set established in step 1 to remove noise from the tactile sensor signal and convert it into a tactile image; After filtering, the preprocessed data is converted into a grayscale image using the following formula: F max =MAX{F1,F2,F3,…F n },n=44 F min =MIN{F1,F2,F3,…F n },n=44 In the formula, F i This is the force value of each tactile sensing unit of the tactile sensor obtained after filtering, where i = 1 to n, F max With F min G represents the maximum and minimum force values. i It is the grayscale value of the corresponding tactile sensing unit in the image; S3: The collected tactile data is processed through a deep Siamese neural network for feature extraction and training; The tactile learning model of the deep twin neural network uses a CNN-based autoencoder to synchronously encode real-time tactile images and reference tactile images. The autoencoder converts each image into a feature vector, combines them into a one-dimensional feature vector, and inputs it into the regression module. During the training phase, the model input consists of the grayscale images corresponding to the correct tactile data and the grayscale images corresponding to the error tactile data obtained in step S2. The input also includes the recorded offsets of the three degrees of freedom. The predicted offsets of the three degrees of freedom are obtained from the two input images, and the squared difference between the predicted offsets and the recorded offsets of the three degrees of freedom is used as the loss function for training. S4: Perform image registration between the sensed tactile image and the tactile image at the reference position; Using the deep Siamese neural network trained in step S3, predict the position differences Δx, Δy, and Δθ between the current position of the flat cable and the ideal position of the flat cable coupling from the images. Based on a... t =a t-1 After -f(Δx, Δy, Δθ) moves to a new position, the robotic arm will again obtain a new offset through the tactile sensor, iterating this process until Ω(Δx, Δy, Δθ) == TRUE; where a t =a t-1 -f(Δx, Δy, Δθ) represents the robot's next position, a t a t-1 Ω is the position of the robotic arm at times t and t-1; f() is a fixed matrix used to transfer the motion distance measured in the haptic image plane to the robot's motion space; Ω() refers to the set of offsets. d x d y d θ The maximum permissible errors for the x-axis, y-axis, and θ, respectively.

3. The method for assembling and positioning flexible flatbed cables based on a 2D surface tactile twin network as described in claim 1 or 2, characterized in that: After image conversion of the tactile signals, all images are converted into m*m images, where m takes the value of 256 or 224.

4. The method for assembling and positioning flexible flatbed cables based on a 2D surface tactile twin network as described in claim 1, characterized in that: The specific process for completing the assembly task in the scenario of assembling mobile phone flexible flat cables is as follows: A. Use visual inspection to identify mobile phone flexible cables on the material table; B. The robotic arm precisely picks up the center position of the mobile phone flexible flat cable on the material table; C. Place the mobile phone flexible cable on the secondary positioning platform; D. Use an RGB camera to detect the snap-fit ​​position, and move the robotic arm to the approximate snap-fit ​​position of the assembly line mobile phone stand; E. Press the male connector of the flexible ribbon cable that needs to be fastened with the tactile sensor at the approximate fastening position; F. Obtain the tactile image of the press to estimate the distance between the accurate location and the current location; G. The robotic arm moves again to the secondary positioning stage to pick up the mobile phone's flexible cable; H. Based on the distance calculated by E, move the flexible flat cable to the correct calculated position and fasten the flexible flat cable.

Images