Mobile phone flexible cable positioning method based on visual-tactile information fusion

Abstract

The present application relates to the field of target pose estimation, and particularly relates to a mobile phone soft wire positioning method based on visual and tactile information fusion under mobile phone assembly. The method performs space-time alignment on the collected visual and tactile information, performs bilinear interpolation on the visual data, and performs sliding average window denoising on the tactile data, then uses different convolutional neural networks to process the visual and tactile information respectively, the two networks output soft wire predicted poses based on visual information and tactile information respectively, and finally through multi-modal information decision-level fusion, the two results are fused in a weighted manner to obtain the final soft wire positioning result. The positioning method reduces the defect that the traditional single mode is not complete in information representation, and through the multi-modal decision-level fusion mode, a result with higher soft wire positioning accuracy and better robustness is obtained.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target pose estimation, and particularly relates to a mobile phone flexible flat cable positioning method based on visual and tactile information fusion in a mobile phone flexible flat cable assembly process. BACKGROUND

[0002] For the field of 3C assembly, with the development of deep learning and robot technology, assembly is being converted from manual operation in the past to automatic operation. Due to the high cost of manual assembly and poor generalization of automatic assembly, more and more people have begun to explore the application of intelligent assembly in recent years. In traditional robot automatic assembly, only single modal information is applied, and single modal information cannot well and completely represent information in the whole assembly task. Traditional visual positioning methods are more suitable for tasks with low precision requirements, and are not ideal for mobile phone part assembly scenes because the visual field of mobile phone assembly is narrow, the assembly precision requirement is high, and the visual method is easily affected by light changes. SUMMARY

[0003] The main purpose of the application is to solve the problems of high cost of manual assembly, poor positioning accuracy of traditional assembly methods, and weak generalization performance. A mobile phone flexible flat cable positioning method based on visual and tactile information fusion is provided. By fusing visual and tactile dual modal information, the defect of incomplete single modal information is overcome. Multi-modal information can represent the task from multiple dimensions. By decision-level fusion of visual and tactile information, the flexible flat cable is first positioned using visual information alone, then positioned using tactile information alone, and finally the two positions are fused to output the final position. This method can effectively improve the positioning accuracy of the flexible flat cable and is a practical technology for mobile phone assembly.

[0004] The technical scheme for achieving the purpose of the application is as follows:

[0005] A mobile phone flexible flat cable positioning method based on visual and tactile information fusion uses a visual sensor and a tactile sensor to realize more accurate positioning of a mobile phone flexible flat cable by fusing multi-modal information. The method comprises the following steps:

[0006] Step (1) collects visual sensor data and tactile sensor data, and pre-processes the collected data for subsequent deep learning model training;

[0007] Step (2) processes the information of the two modalities before fusion, including alignment of the time and space information of the visual and tactile modalities, and encoding of the information of the visual and tactile modalities;

[0008] Step (3) uses a single convolutional neural network-based target detection network to perform soft-wiring positioning on the visual modal information processed in step (2), and the network outputs the detection result and the detection frame of the soft-wiring;

[0009] Step (4) performs edge detection on the detection frame in the result of step (3), calculates the pixel coordinate position and the rotation angle of the soft-wiring in the image, and converts the image coordinates into the mechanical arm space coordinates through the hand-eye calibration technology;

[0010] Step (5) inputs the haptic encoding information processed in step (2) into a deep learning position prediction network to predict the target pose of the soft-wiring, and the network can output the predicted position of the soft-wiring in the mechanical arm space;

[0011] Step (6) performs decision-level fusion on the positioning results of steps (4) and (5), adopts a weighted fusion manner for the two results, and outputs the final predicted pose of the mobile phone soft-wiring.

[0012] Further, step (1) is specifically: collecting and saving visual and tactile information, for visual data, collecting 1920*1080 size images of the visual sensor, and naming each image according to the timestamp, and saving the RGB image in JPG format. For haptic data, collect the data of the haptic sensor, and name and save each collected haptic information in the format of the timestamp.

[0013] Further, step (2) is specifically: aligning the visual and tactile data in step (1) in the time and space dimensions, for the time dimension, selecting the data with the closest timestamp in the two modalities to complete the time alignment;

[0014] For the mechanical arm operation, collecting haptic information will cause the problem of visual sensor occlusion, therefore, for spatial alignment, the visual data and the haptic data are aligned through the collected haptic information and the visual image before collecting the haptic information, thereby completing the alignment of the multi-modal information;

[0015] For the aligned data, the image data first needs to be reduced to 224*224 size by using the bilinear interpolation method, wherein the bilinear interpolation formula is:

[0016]

[0017] wherein Q 11 =(x1,y1), Q 12 =(x1,y2), Q 21 =(x2,y1), Q 22= (x2, y2), P = (x, y), f is the coordinate value of the point, R1 and R2 are intermediate results when interpolating to the x-axis, and thus the interpolation formula can reduce the picture;

[0018] For tactile data, a total of 10 data before and after the alignment timestamp are selected for smoothing and denoising processing, and a sliding average filter is used for smoothing, according to the formula Where F(k) represents the value filtered by the sliding average window at time k, k is valued as {1, 2, 3…10}, and F is the final tactile data after mean processing. The visual and tactile information is preprocessed before fusion through the above steps.

[0019] Further, step (3) is specifically: using a YOLO v5 target detection network based on a convolutional neural network to detect the mobile phone flexible flat cable, YOLO v5 is used to process visual information, and outputs the detection result and the detection frame for the mobile phone flexible flat cable. The training input of the network is the image data of 224*224 size processed in step (2), and the training data Label is the target category and the standard target frame pixel coordinates. The network loss function includes three parts of classification loss, confidence loss and positioning loss. The classification loss calculates the loss of positive and negative samples by cross-entropy loss. The confidence loss first calculates the intersection area IOU of the prediction frame and the Label input frame, and then calculates the target confidence loss by cross-entropy loss. The positioning loss is calculated by using GIOU loss function. The output of the network model is the category detection result and the prediction frame coordinates of the mobile phone flexible flat cable, which is used for subsequent positioning.

[0020] Further, step (4) is specifically: using a hand-eye calibration technique to position the mobile phone flexible flat cable. First, the Sobel edge operator is used to calculate the specific soft flat cable frame and the rotation angle, then the center point of the frame is calculated, and finally the camera-robot coordinate system of the eye on the hand is calibrated. The main purpose is to convert the camera and robot end coordinate system. According to the formula T3XT1P0 = P3, where P0 is any point on the calibration board, T1 is the known camera external parameter matrix, T3 is the known end-to-robot base coordinate system conversion matrix of the robot, P3 is the coordinate position of the calibration board P0 converted to the base coordinate system, and X is the hand-eye calibration matrix to be solved.

[0021] Then move the robot, for the same point on the calibration board, P0 and the corresponding P3 do not change, and T1 and T3 change, so there is the expression By solving the above two expressions, we can get Solving the equation can get the hand-eye calibration matrix X.

[0022] Through the matrix, the YOLO v5 detection result center point coordinates can be converted into the robot position pv = (x, y, r), wherein x, y, r are the calculated soft-wiring position and rotation angle, which will be used in subsequent multi-modal fusion.

[0023] Further, step (5) is specifically: using the method of CNN+LSTM to perform target position prediction training, first duplicating the tactile information in step (2) 20 times, then splicing the duplicated results, such as F = f1 + f2 + … + f20. 20 Wherein F is the final splicing result, f i is the same tactile data;

[0024] Then the tactile information F is taken as the input of the LSTM model, wherein Label is the X, Y axis distance and rotation angle R of the current soft-wiring position and ideal position, when the LSTM processes the data, two fully connected layers are connected, and the output size of the last fully connected layer is 3*1, which respectively represents the predicted distance of X, Y axis and R angle, wherein the loss function function Loss of the network model is mean square error loss, such as formula At this step, the soft-wiring pose of the mobile phone can be predicted by the tactile information, and the result is represented as p f = (Δx, Δy, Δr).

[0025] Further, step (6) is specifically: the results of step (4) and step (5) are fused at the decision level, and the final soft-wiring pose Pos = (Pos x , Pos y , Pos r ) is outputted, wherein the fusion formula is:

[0026]

[0027]

[0028]

[0029] The final positioning result of the fused soft-wiring can be calculated by the above formula.

[0030] The visual and tactile information is fused at the decision level, the visual and tactile information is preprocessed first, including bilinear interpolation of visual data and moving average window filtering of tactile data, then YOLO v5 and CNN+LSTM are used to predict the soft wire position of the visual and tactile information respectively, and the results of the two predictions are fused at the decision level, and the final positioning result is output in a weighted manner. The method improves the positioning accuracy of the assembly parts, reduces the defects of incomplete traditional single modal information representation, and improves the robustness of the positioning result in the assembly process through the multi-modal fusion positioning method. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the positioning method of the application;

[0032] Figure 2 is a comparison chart of the positioning results of the mobile phone soft wire using the method of the application and the traditional visual detection positioning results;

[0033] Figure 2 (a) is a traditional visual detection positioning result chart, Figure 2 (b) is a positioning result chart of the method of the application. DETAILED DESCRIPTION

[0034] The content of the application will be further described in detail below in combination with the embodiments and the drawings, but is not limited to the application.

[0035] EMBODIMENT

[0036] The mobile phone soft wire positioning method based on visual and tactile information fusion, as shown in Figure 1 , the visual and tactile sensor information is collected, the positioning results of the two sensors are fused at the decision level, the positioning accuracy and robustness of the mobile phone soft wire are improved, Figure 1 is a specific flowchart of the positioning method of the application. This embodiment further illustrates the flow by positioning the mobile phone soft wire, and the positioning method includes the following steps:

[0037] Step (1) collect the visual sensor data and tactile sensor data, and save the collected data;

[0038] Step (2) process the information of the two modalities respectively, including multi-modal spatio-temporal alignment, and encoding the information of the visual and tactile modalities respectively;

[0039] Step (3) the visual modality information output by step (2) is predicted by using YOLO v5 neural network model to predict the position of the soft wire, and the detection result and detection frame of the soft wire are output;

[0040] Step (4) performs edge detection on the detection frame in the result of step (3), and converts the mobile phone flexible cable coordinates into mechanical arm space coordinates through a hand-eye calibration technique;

[0041] Step (5) inputs the haptic encoding information output by step (2) into a CNN+LSTM network to predict the target pose of the flexible cable, and outputs the distance between the current flexible cable position and the ideal position.

[0042] Step (6) performs decision-level fusion on the positioning results of steps (4) and (5), adopts a weighted fusion manner for the two results, and outputs the final predicted pose of the mobile phone flexible cable.

[0043] In this embodiment, for visual data, 1920*1080 size images are collected from a visual sensor, and each image is named according to the timestamp and saved as an RGB image in JPG format; for haptic data, data from a haptic sensor is collected, and each collected haptic information is named and saved in the format of a timestamp.

[0044] In this embodiment, the visual and haptic data of step (1) are spatio-temporally aligned. For the time layer, the data with the closest timestamps in the two modalities are selected to complete the time alignment. However, for the mechanical arm operation, the collection of haptic information may cause the problem of visual sensor occlusion. Therefore, for spatial alignment, the visual data and the haptic data are aligned through the collected haptic information and the visual image before collecting the haptic information, thereby completing the alignment of the multi-modal information.

[0045] In this embodiment, for image data, the original picture needs to be first reduced to 224*224 size by using a bilinear interpolation method, wherein the bilinear interpolation formula is:

[0046]

[0047]

[0048]

[0049] wherein Q 11 =(x1,y1), Q 12 =(x1,ψ2), Q 21 =(x2,y1), Q 22 =(x2,y2), P=(x,y), f is the coordinate value of a point, and R1 and R2 are intermediate results when interpolating the x-axis. Thus, the picture can be reduced by the interpolation formula.

[0050] In this embodiment, for haptic data, a total of 10 data before and after the aligned timestamp are selected for smoothing and denoising processing, and a sliding average filter is used for smoothing according to the formula Wherein F(k) represents the value filtered through the sliding average window at time k, k takes values of {1, 2, 3…10}, and F is the final tactile data after mean processing. The visual-tactile information is preprocessed before fusion through the above steps.

[0051] In this embodiment, step (3) uses a YOLO v5 target detection network to detect the soft wire of the mobile phone. The input of the network is the 224*224 size image data processed in step (2), and the output of the network model is the category detection result and the prediction box coordinates of the soft wire of the mobile phone, which is used for subsequent positioning.

[0052] In this embodiment, step (4) uses a hand-eye calibration technique to position the soft wire of the mobile phone. First, the Sobel edge operator is used to calculate the specific soft wire frame and rotation angle, then the center point of the frame is calculated, and finally the camera-robot coordinate system of the eye on the hand is calibrated. The main purpose is to convert the camera and robot end coordinate system, and according to the hand-eye calibration matrix X, the YOLO v5 detection result center point coordinates can be converted into the robot position p v =(x,y,r), where x, y, r are the calculated soft wire position and rotation angle, which will be used in subsequent multi-modal fusion.

[0053] In this embodiment, step (5) uses CNN+LSTM to predict the position of the soft wire. First, the tactile information in step (2) is copied 20 times, and then the copied results are spliced, such as F=f1+f2+…+f 20 , where F is the final spliced result, and f i is the same tactile data. Then the tactile information F is taken as the input of the LSTM model, and the output result is represented as p f =(Δx,Δy,Δr).

[0054] In this embodiment, the results of step (4) and step (5) are fused at the decision level, and the final soft wire pose Pos is output through fusion, and the fusion formula is:

[0055]

[0056]

[0057]

[0058] In this embodiment, the final soft wire prediction position can be output through step (6).

[0059] Comparative example:

[0060] ReferenceFigure 2 The positioning results of the mobile phone flexible flat cable using the method of the application are compared with the positioning results using the traditional visual detection method, Figure 2 (a) The positioning result map of the traditional visual detection method. As can be seen from the map, the positioning effect of the traditional visual detection method on the flexible flat cable is not ideal, and the specific performance is that the positioning position of the flexible flat cable cannot coincide with the correct position. As can be seen from the map, the position of the flexible flat cable has a larger error compared with the ideal position in X, Y and the rotation axis.

[0061] Figure 2 (b) The positioning result map of the method of the application. As can be seen from the map, the positioning effect of the method of the application can be more accurately coincided with the correct positioning position, and satisfactory results are achieved in X, Y and the rotation axis, especially the positioning accuracy of the flexible flat cable is greatly improved in the rotation axis.

[0062] After Figure 2 As a result of comparison, the positioning accuracy of the method of the application is higher than that of the traditional visual detection method, the positioning effect is better, and the robustness and generalization ability are also stronger.

Claims

1. A mobile phone flexible flat cable positioning method based on visual-touch information fusion, characterized in that, By fusing information from visual and tactile sensors, and employing neural network models to locate the mobile phone's flexible flat cable, the method specifically includes the following steps: Step (1) Collect visual sensor data and tactile sensor data, and perform necessary preprocessing on the two types of data for subsequent deep learning model training; Step (2) performs pre-fusion processing on the information of the two modalities, including aligning the temporal and spatial information of the visual and tactile modalities, and encoding the information of the visual and tactile modalities respectively; Step (3) uses a separate convolutional neural network-based target detection network to locate the soft flat cable based on the visual modal information processed in step (2). The network outputs the detection results and detection boxes of the soft flat cable. Step (4) performs edge detection on the detection box in the result of step (3), calculates the pixel coordinate position and rotation angle of the soft flat cable in the image, and converts the image coordinates into the spatial coordinates of the robotic arm through hand-eye calibration technology; Step (5) involves inputting the tactile encoding information processed in step (2) into a deep learning position prediction network to predict the pose of the soft flat cable target. This network can output the predicted position of the soft flat cable in the robotic arm space. Step (6) performs decision-level fusion of the positioning results from steps (4) and (5), and uses a weighted approach to fuse the two results to output the final predicted pose of the mobile phone flexible flat cable.

2. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 1, characterized in that, Step (1) specifically involves: The visual and tactile information is collected and saved. For visual data, images of 1920*1080 size are collected from the visual sensor, and each image is named according to the timestamp and saved as RGB images in JPG format. For tactile data, data from tactile sensors is collected, and each collected tactile information is named and saved in a timestamp format.

3. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 2, characterized in that, Step (2) specifically involves: Align the visual-tactile data in step (1) in a temporal and spatial manner. In terms of time, select the data with the closest timestamps in the two modalities to complete the time alignment. For robotic arm operations, the collection of tactile information can cause visual sensor occlusion. Therefore, for spatial alignment, visual data and tactile data are aligned by using the collected tactile information and the visual image before the tactile information is collected, thus completing the alignment of multimodal information. For the aligned data, image data first needs to be reduced to a size of 224*224 using bilinear interpolation. The bilinear interpolation formula is as follows: Q 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2), P = (x, y), f is the coordinate value of the point, R1 and R2 are the intermediate results of interpolation on the x-axis, and the image can be reduced by this interpolation formula; For tactile data, a total of 10 data points before and after the aligned timestamps were selected for smoothing and noise reduction. A moving average filter was used for smoothing, according to the formula... Where F(k) represents the value after the moving average window filtering at time k, and the value of k is {1,2,3…10}. F is the final tactile data after mean processing. The above steps are used to preprocess the visual-tactile information before fusion.

4. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 3, characterized in that, Step (3) specifically involves: The YOLO v5 object detection network based on convolutional neural network is used to detect mobile phone flex cables. YOLO v5 is used to process visual information and output the detection results and detection boxes for mobile phone flex cables. The training input of this network is the 224*224 image data after processing in step (2). Its training data Label is the target category and the pixel coordinates of the standard target box. Its network loss function includes three parts: classification loss, confidence loss and localization loss. The classification loss uses cross-entropy loss to calculate the loss of positive and negative samples. The confidence loss first calculates the intersection area IOU between the predicted box and the Label input box, and then uses cross-entropy loss to calculate the target confidence loss. The localization loss is calculated using the GIOU loss function. The output of this network model is the category detection result and predicted box coordinates for mobile phone flex cables. This result is used for subsequent localization.

5. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 4, characterized in that, Step (4) specifically involves: The mobile phone flexible flat cable is located using hand-eye calibration technology. First, the Sobel edge operator is used to calculate the specific border and rotation angle of the flexible flat cable. Then, the center point of the border is calculated. Finally, hand-eye calibration is performed on the camera-robotic arm coordinate system with the eye on the hand. This mainly involves transforming the coordinate systems of the camera and the robotic arm end effector. According to the formula T3XT1P0=P3, where P0 is any point on the calibration board, T1 is the known camera extrinsic parameter matrix, T3 is the transformation matrix of the robotic arm end effector to the robotic arm base coordinate system known from the factory, P3 is the coordinate position of the calibration board under the transformed base coordinate system of P0, and X is the hand-eye calibration matrix to be determined. Then, the robotic arm is moved. For the same point on the calibration plate, P0 and the corresponding P3 do not change, while T1 and T3 change. Therefore, the expression is: By combining the two expressions above, we can obtain Solving this equation yields the hand-eye calibration matrix X; This matrix can be used to convert the center point coordinates of the YOLO v5 detection results into the robotic arm position p. v = (x, y, r), where x, y, and r are the calculated soft ranking position and rotation angle, which will be used in subsequent multimodal fusion.

6. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 5, characterized in that, Step (5) specifically involves: The target location prediction training is performed using CNN+LSTM. First, the tactile information in step (2) is copied 20 times, and then the copied results are concatenated, such as F = f1 + f2 + ... + f 20 Where F is the final splicing result, f i For the same tactile data; Then, the tactile information F is used as the input to the LSTM model, where Label represents the X and Y axis distances and rotation angle R between the current position of the flat cable and the ideal position. After the LSTM processes the data, two fully connected layers are connected. The output of the last fully connected layer is 3*1, representing the predicted distances along the X, Y axes and the R angle, respectively. The loss function of the network model, Loss, is the mean squared error loss, as shown in the formula. This step allows for the prediction of the pose of the mobile phone's flexible flat cable using tactile information, and the result is represented as p. f = (Δx, Δy, Δr).

7. The mobile phone flexible flat cable positioning method based on visual-touch information fusion as described in claim 6, characterized in that, Step (6) specifically involves: The results of steps (4) and (5) are fused at the decision level, and the final soft cable pose Pos = (Pos) is output through the fusion. x Pos y Pos r The fusion formula is: The final positioning result of the fused flexible flat cable can be calculated using the above formula.

Images