A robot screwing state detection method
By designing a flexible tactile sensor and a neural network, the problem of detecting the tightening status of bolts and nuts in robot assembly was solved, enabling high-precision autonomous tightening operations by the robot.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2024-05-17
- Publication Date
- 2026-07-03
AI Technical Summary
In the current process of robot assembly, it is difficult to accurately estimate the tightening state between bolts and nuts, resulting in inaccurate tightening control.
A flexible tactile sensor was designed, combining a long short-term memory network and a convolutional neural network. By eliminating artifact noise, a mapping relationship between tactile data and twisting state was constructed, enabling the robot to autonomously detect twisting state.
It improves the accuracy and safety of the robot's autonomous twisting operation, and the accuracy and mean square error of twisting execution status detection are better than other networks.
Smart Images
Figure CN118342515B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot operation technology, and in particular to a method for detecting the turning state of a robot. Background Technology
[0002] With the development of intelligent robot assembly, the requirements for the level of autonomous robot assembly are becoming increasingly higher. Among them, bolt tightening, as one of the typical tasks of robot assembly, requires accurate estimation of the tightening state (alignment and tightening) between the bolt and nut in order to carry out precise planning and control, which is still a challenging task.
[0003] In robot assembly, visual feedback is the most commonly used source of information for identifying the turning state, and the target posture is calculated using computer vision methods such as target recognition and segmentation. However, visual sensors have limitations in assembly due to lighting and occlusion. Using tactile sensors is an important means of overcoming the limitations of computer vision and can improve the performance of robot operations.
[0004] Researchers both domestically and internationally have primarily focused on two main approaches to robotic screwing operations: state detection-based models and process control-based models. State detection-based research on robotic screwing includes methods such as visual detection and force detection multi-mode fusion. Technically, most current screwing control algorithms target precise bolt positioning or bolt-nut alignment control, neglecting the bolt's tightness. Therefore, detecting the screwing state is a crucial technical problem that urgently needs to be solved. Summary of the Invention
[0005] In view of this, the embodiments of this application provide a robot twisting state detection method to realize the twisting state detection of a robot arm.
[0006] A method for detecting the turning state of a robot, the method comprising:
[0007] Create a flexible tactile sensor, determine an algorithm for eliminating sensor data artifact noise, and establish a mapping relationship between tactile data and twisting state based on the flexible tactile sensor and the elimination algorithm.
[0008] Based on the temporal characteristics of tactile data, a temporal tactile data processing module is constructed;
[0009] Based on the mapping relationship between tactile data and twisting state, the temporal tactile data processing module is invoked to construct a twisting state estimation network;
[0010] Based on the aforementioned twisting state estimation network and tactile data, the robot performs autonomous assembly operations.
[0011] In the above method, creating a flexible tactile sensor includes:
[0012] The flexible tactile sensor has a flexible tactile sensing array, which consists of an upper electrode layer, an intermediate dielectric layer, and a lower electrode layer. The upper electrode layer and the lower electrode layer both use polyimide (PI) film as the flexible substrate of the sensor array. A 200nm thick copper layer is deposited on the 150μm thick PI film as a conductive material layer, and a 1.2mm thick silicone rubber PDMS is used as the intermediate dielectric layer.
[0013] Laser cutting technology is used to fill the gap between the upper and lower electrode layers with air as a dielectric.
[0014] A PDMS frustum with a height of 2mm is used as the force-bearing unit of the flexible tactile sensor unit;
[0015] Each three-dimensional sensor unit in the flexible tactile sensor has five capacitive sensing elements, which are covered by PDMS prisms.
[0016] The algorithm for eliminating sensor data artifact noise in the above method includes:
[0017] The pressure-time curves recorded for each electrode were filtered using a mean filter with a window size of 3 to eliminate artifact noise in the sensor data.
[0018]
[0019] Where y[i] represents the filtered tactile data, x[i+j] represents the input tactile data, is the size of the filtering window, j represents the position of the filtering window on the input tactile data, and M represents the size of the filtering window.
[0020] In the above method, the tactile sensor includes two three-dimensional sensor units, and the twisting state includes a tightened state and a twisting state.
[0021] The mapping relationship between tactile data and twisting state is represented as follows:
[0022] Data = {(c1,c2,c3,……,c 10 ), label}
[0023] Where (c1,c2,c3,……,c 10 The input tactile data is represented by , and the label is the label for the tightening state, indicating the tightened state or the state in which it is being tightened.
[0024] In the above method, the step of constructing a temporal tactile data processing module based on the temporal characteristics of tactile data includes:
[0025] Based on the temporal characteristics of tactile data, a temporal tactile data processing module is constructed using Long Short-Term Memory (LSTM).
[0026] The tactile data processing module based on Lstm contains two Lstm network layers with a neuron size of 3, an input data length of 64, and 128 hidden layer neurons.
[0027] A single-layer Lstm network layer consists of multiple Lstm units, each Lstm unit containing a cell state and three gates, namely: an input gate, a forget gate, and an output gate.
[0028] The information stored in the state of the input gate determines the information, including two parameters i. t and i obtained from the sigmoid activation function t The value used to determine the update is obtained using the tanh activation function. The new candidate vector is represented by the following expression:
[0029] i t =sigmoid(ω i ·[h t-1 ,x t ]+b i )
[0030]
[0031] Among them, i t ω is the update weight for new information. i and b i For the weights and biases of the input gate, ω t and b t Weights and biases of memory cells;
[0032] The forget gate determines whether information about the cell state is forgotten and updates the information, including the hidden state h from the previous time step. t-1 Input x with the current state t The forgetting probability f is obtained by feeding it into the sigmoid activation function. t As shown in the following formula:
[0033] f t =sigmoid(ω f ×[h t-1 ,x t ]+b f )
[0034] Where, ω f and b f The weights and biases of the input gate;
[0035] Output gate outputs memory unit information:
[0036] h t =o t ×tanh(C t )
[0037] Among them, h t For the hidden state of the next layer, o t This is the output content.
[0038] In the above method, the step of constructing a twisting state estimation network by calling a temporal tactile data processing module based on the mapping relationship between tactile data and twisting state includes:
[0039] Before the temporal tactile data processing module, an attention-based convolutional layer and pooling layer are added, with 3 convolutional layers and 3 pooling layers. The input of the convolutional layer is tactile data of length 10×1. Each convolutional layer has a kernel size of 3×3. The convolutional layer extracts tactile data features from the input 1-dimensional tactile data and maps the 1-dimensional tactile data to 64-dimensional tactile data features.
[0040] Pooling layers are used to reduce the dimensionality of the tactile data features output by the convolutional layers;
[0041] In the above method, the tactile data features extracted by the convolutional layer are characterized as follows:
[0042]
[0043] Where 'a' represents the tactile data features after the convolution operation. This represents the tactile data feature mapping of the j-th neuron in the l-th layer, where σ represents the ReLU activation function. For bias; F l This represents the number of feature maps on the tactile data feature map of layer l. This represents the weight matrix between the f-th feature map and the j-th neuron in the l-th layer.
[0044] The above method utilizes pooling layers to reduce the dimensionality of the tactile data features output from the convolutional layers, including:
[0045] The pooling layer uses the following formula to reduce the dimensionality of the tactile data features output by the convolutional layer:
[0046]
[0047] Where A represents the tactile data feature, i,j represent the index positions on the tactile data feature, x,y represent the positions in the pooling window, f represents the size of the pooling window, s0 is the stride, and p is the number of filling layers. When p is infinite, the maximum value is taken within the pooling layer.
[0048] In the above method, based on the twisting state estimation network and tactile data, the robot performs autonomous assembly operations, including:
[0049] The system uses a binocular camera to acquire RGBD images of the working area, and uses a cumulative probability Hough transform (PPHT) algorithm to identify external hexagonal bolts and nuts, to obtain the position and orientation information of the center point of the bolts and nuts to be grasped in an unknown environment.
[0050] The robot arm is controlled to grasp the top of the bolt and reach above the nut. The robot's motor drives the arm downward to align with the threaded hole. After the nut and the tail surface of the bolt are in full contact, the visual tactile data changes and generates a vertically upward contact force and a tangential force. The robot arm's posture in the task coordinates is calculated using the tactile data.
[0051] The robot arm's end joint slowly rotates downwards, inputting tactile information with temporal characteristics into a screw-tightening state estimation network. This network then determines and outputs the screw-tightening state of the screw and nut, enabling the robot to autonomously perform the screw-tightening task.
[0052] This method acquires contact information of the twisting operation by designing a tactile sensor, and then proposes a new neural network to identify the twisting execution state under uncertain conditions, guiding the robot to complete the autonomous twisting task and improving the accuracy and safety of the robot's autonomous operation. Attached Figure Description
[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0054] Figure 1 This is an operation flowchart provided in the embodiments of the present invention;
[0055] Figure 2 This is a design diagram of the tactile sensor in an embodiment of the present invention;
[0056] Figure 3 This is a network diagram for determining the robot's turning execution state in an embodiment of the present invention;
[0057] Figure 4 This is a comparison diagram of the confusion matrix between the method of this invention and other methods. Detailed Implementation
[0058] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0059] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0060] This invention provides a method for detecting the turning state of a robot. Please refer to the following embodiments. Figure 1 The method includes the following steps:
[0061] Step 101: Create a flexible tactile sensor, determine an algorithm for eliminating sensor data artifact noise, and establish a mapping relationship between tactile data and twisting state based on the flexible tactile sensor and the elimination algorithm.
[0062] like Figure 2 As shown, the flexible tactile sensing array consists of an upper electrode layer ②, an intermediate dielectric layer ③, and a lower electrode layer ④. The upper and lower electrode layers use polyimide (PI) as the flexible substrate for the sensor array. A 200nm thick copper layer is deposited on a 150μm thick PI film as a conductive material layer. To prevent contact between the upper and lower electrode layers under external pressure, a 1.2mm thick PDMS silicone rubber is used as the intermediate dielectric layer.
[0063] Simultaneously, using laser cutting technology, air is used as the dielectric to correspond to the gap between the upper and lower electrode points, and a 2mm high polydimethylsiloxane (PDMS) frustum is used as the force-bearing unit of the sensor unit, such as... Figure 2 As shown in ①.
[0064] Each three-dimensional sensor unit in the flexible tactile sensor has five capacitive sensing elements, which are covered by PDMS prisms.
[0065] The pressure-time curves recorded for each electrode were filtered using a mean filter with a window size of 3 to eliminate artifact noise in the sensor data.
[0066]
[0067] Where y[i] represents the filtered tactile data, x[i+j] represents the input tactile data, is the size of the filtering window, j represents the position of the filtering window on the input tactile data, and M represents the size of the filtering window.
[0068] Accurately grasping the execution state of a robot's turning motion is fundamental to achieving autonomous turning. Therefore, a turning state representation method based on tactile data is proposed. The tactile sensor used in this invention includes two three-dimensional sensor units. The turning state is divided into two types: a tightened state and a turning in progress state, which are used as judgment labels. The representation is as follows:
[0069] Data = {(c1,c2,c3,……,c 10 ), label}
[0070] Where (c1,c2,c3,……,c 10 The input tactile data is represented by , and the label is the label for the tightening state, indicating the tightened state or the state in which it is being tightened.
[0071] Step 102: Based on the temporal characteristics of tactile data, construct a temporal tactile data processing module.
[0072] Based on the temporal characteristics of tactile data, this paper designs a temporal tactile data processing module using Long Short-Term Memory (LSTM). The LSTM-based tactile data processing module comprises two LSTM network layers with a neuron size of 3, an input data length of 64, and 128 hidden layer neurons. A single-layer LSTM network consists of multiple LSTM units, each containing a cell state and three gates: an input gate, a forget gate, and an output gate.
[0073] The input tactile sequence is fed into the LSTM network step by step. For each time step, the LSTM network calculates the values of the input gate, forget gate, and output gate based on the current input and the hidden state of the previous time step.
[0074] The input gate determines the information stored in the cell state, including two parameters i. t and i obtained from the sigmoid activation function t Used to determine the update value; and obtained from the tanh activation function. This represents a new candidate vector. Its expression is as follows:
[0075] i t =sigmoid(ω i ·[h t-1 ,x t ]+b i )
[0076]
[0077] Among them, i t ω is the update weight for new information. i and bi bi represents the weights and biases of the input gate, ω t and b t Weights and biases of memory cells.
[0078] The forget gate determines whether information about the cell state has been forgotten, and then updates the information by updating the hidden state h from the previous time step. t-1 Input x with the current state t The forgetting probability f is obtained by feeding it into the sigmoid activation function. t As shown in the following formula:
[0079] f t =sigmoid(ω f ×[h t-1 ,x t ]+b f )
[0080] Where, ω f and b f The weights and biases of the input gates.
[0081] Output gate outputs memory unit information:
[0082] h t =o t ×tanh(C t )
[0083] Among them, h t For the hidden state of the next layer, o t This is the output content.
[0084] Step 103: Based on the mapping relationship between tactile data and twisting state, call the temporal tactile data processing module to construct the twisting state estimation network.
[0085] like Figure 3 As shown, to better understand tactile data, convolutional and pooling layers based on an attention mechanism are added before the temporal tactile data processing module. There are three convolutional layers and three pooling layers. The input to the convolutional layers is 10×1 tactile data. Each convolutional layer has a kernel size of 3×3. These convolutional layers extract tactile features from the 1-dimensional input tactile data and map the 1-dimensional tactile data to 64-dimensional tactile features, resulting in richer features. The convolutional feature extraction formula can be described as follows:
[0086]
[0087] Where 'a' represents the tactile data features after the convolution operation. This represents the tactile data feature mapping of the j-th neuron in the l-th layer, where σ represents the ReLU activation function. For bias; Fl This represents the number of feature maps on the tactile data feature map of layer l. This represents the weight matrix between the f-th feature map and the j-th neuron in the l-th layer.
[0088] After convolution, pooling layers are used to reduce the feature dimension of the convolutional layer output, which effectively reduces parameter computation and improves robustness. The pooling layer uses the following formula to reduce the dimensionality of the tactile data features output by the convolutional layer:
[0089]
[0090] Where A represents the tactile data feature, i,j represent the index positions on the tactile data feature, x,y represent the positions in the pooling window, f represents the size of the pooling window, s0 is the stride, and p is the number of filling layers. When p is infinite, the maximum value is taken within the pooling layer.
[0091] Step 104: Based on the twisting state estimation network and tactile data, perform the robot's autonomous assembly operation.
[0092] Considering the task scenario of assembling M10 screws and nuts, we divide the control architecture into three stages: remote, contact, and tightening. Vision is used to calculate the arm posture reference in the task coordinates, and a haptic force feedback loop is applied to control the start and stop of tightening. The vision and robot control routines are executed sequentially.
[0093] The system uses a binocular camera to acquire RGBD images of the working area, and uses a cumulative probability Hough transform (PPHT) algorithm to identify external hexagonal bolts and nuts, to obtain the position and orientation information of the center point of the bolts and nuts to be grasped in an unknown environment.
[0094] The robot arm is controlled to grasp the top of the bolt and reach above the nut. The robot's motor drives the arm downward to align with the threaded hole. After the nut and the tail surface of the bolt are in full contact, the visual tactile data changes and generates a vertically upward contact force and a tangential force. The robot arm's posture in the task coordinates is calculated using the tactile data.
[0095] The robot arm's end joint slowly rotates downwards, inputting tactile information with temporal characteristics into a screw-tightening state estimation network. This network then determines and outputs the screw-tightening state of the screw and nut, enabling the robot to autonomously perform the screw-tightening task.
[0096] The UR10 robotic arm was mounted on a fixed platform and connected to a computer via a 1.5-meter cable led from the control box. A ZED 2i camera was mounted at the end of the robotic arm. A Robotiq two-finger hand was installed at the end of the robotic arm, and two flexible tactile array sensors were attached to the fingertips of each finger. The sensors were connected to the lower-level PCB board of the data acquisition circuit via a hardware interface, and connected to the upper-level computer via a TTL serial port. The designed tactile sensor sampling frequency is 110ms, collecting 20 data points at a time, i.e., data from both sensors. Since the two sensors are mounted on the fingertips, the tactile data collected simultaneously is symmetrical; therefore, 10 data points are used as a set of twisting state data, which is sufficient to contain most of the information about the twisting state.
[0097] For example, using collected tactile data to predict the twisting state, LSTM, RNN, CNN-LSTM, and SCR-LSTM models were used for training and prediction, respectively. Comparing the four prediction algorithms, the SCR-LSTM prediction error RSME was 0.2108 and MAE was 0.044. The results show that the SCR-LSTM prediction curve is closest to the actual twisting state and performs better than CNN-LSTM. This indicates that the SCR-LSTM prediction model improves prediction accuracy by introducing an attention mechanism to enhance important features and dynamically selecting features highly correlated with the twisting state. This verifies the effectiveness of the method provided in this embodiment.
[0098] The confusion matrix of the four models using cross-validation is as follows: Figure 4 (a)~ Figure 4 As shown in (d), in the confusion matrix, each column represents the predicted class, and the total number of columns represents the number of data predicted as that class (class 0 or class 1); each row represents the true class, and the total number of rows represents the number of true data in that class. Figure 4 In (a), the number of true data points for class 0 and class 1 are 72 and 28, respectively. The top left cell shows 68, indicating that the model correctly predicted class 0 as class 0 68 times. The top right cell shows 4, indicating that the model incorrectly predicted class 1 as class 0 4 times, and so on. Based on the data in the figure, the accuracy of the SCR-LSTM model can be calculated to be 96%, which is significantly higher than other neural network models, and the mean squared error of classification accuracy is the lowest at 4.4%. The CNN-LSTM model has an 8% higher accuracy than the LSTM model. It can be seen that compared to using LSTM layers alone for prediction, adding CNN before LSTM layers can extract and expand clearer temporal features, thereby improving the model's prediction accuracy.
[0099] The application effect of the screwing execution state estimation method in actual experiments shows that the success rate of actual screwing and tightening judgment reaches 90%, and on this basis, the success rate of controlling the robot to complete the bolt and nut tightening task reaches 80%.
[0100] The technical solutions of the embodiments of the present invention have the following beneficial effects:
[0101] A flexible tactile sensor is created, and an algorithm for eliminating artifact noise in the sensor data is determined. Based on the flexible tactile sensor and the elimination algorithm, a mapping relationship between tactile data and twisting state is established. Then, based on the temporal characteristics of the tactile data, a temporal tactile data processing module is constructed. Based on the mapping relationship between tactile data and twisting state, the temporal tactile data processing module is invoked to construct a twisting state estimation network. Based on the twisting state estimation network and the tactile data, the robot performs autonomous assembly operations. Thanks to the design of a new temporal tactile data processing module and the addition of convolution to the twisting state estimation network, this invention achieves effective extraction of tactile sensing information features with temporal characteristics. The final estimated twisting execution state detection accuracy and mean square error performance are superior to other networks.
[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0103] The contents not described in detail in this specification are common knowledge to those skilled in the art.
Claims
1. A method for detecting the turning state of a robot, characterized in that, The method includes: Create a flexible tactile sensor, determine an algorithm for eliminating sensor data artifact noise, and establish a mapping relationship between tactile data and twisting state based on the flexible tactile sensor and the elimination algorithm. Based on the temporal characteristics of tactile data, a temporal tactile data processing module is constructed; Based on the mapping relationship between tactile data and twisting state, the temporal tactile data processing module is invoked to construct a twisting state estimation network; Based on the aforementioned screw-tightening state estimation network and tactile data, the robot performs autonomous assembly operations. Specifically, it acquires the position and orientation information of the center points of the bolts and nuts through a binocular camera and the PPHT algorithm, controls the robotic arm to complete the grasping and alignment, calculates the posture of the robotic arm based on the tactile data, and inputs the temporal tactile information into the screw-tightening state estimation network to output the screw-tightening state of the screws and nuts, thereby realizing the autonomous screw-tightening task.
2. The method according to claim 1, characterized in that, The creation of the flexible tactile sensor includes: The flexible tactile sensor has a flexible tactile sensing array, which consists of an upper electrode layer, an intermediate dielectric layer, and a lower electrode layer. The upper and lower electrode layers both use polyimide films as flexible substrates for the sensor array. A 200 nm thick copper layer is deposited on the 150 μm thick polyimide film as a conductive material layer, and a 1.2 mm thick silicone rubber PDMS is used as the intermediate dielectric layer. Laser cutting technology is used to fill the gap between the upper and lower electrode layers with air as a dielectric. A PDMS frustum with a height of 2mm is used as the force-bearing unit of the flexible tactile sensor unit; Each three-dimensional sensor unit in the flexible tactile sensor has five capacitive sensing elements, which are covered by PDMS prisms.
3. The method according to claim 1, characterized in that, Determine algorithms for eliminating artifact noise in sensor data, including: The pressure-time curves recorded for each electrode were filtered using a mean filter with a window size of 3 to eliminate artifact noise in the sensor data. in, This represents the filtered tactile data. This represents the input tactile data, and the size of the filter window. M represents the position of the filter window on the input tactile data, and M represents the size of the filter window.
4. The method according to claim 1, wherein the tactile sensor comprises two three-dimensional sensor units, and the screwing state includes a tightened state and a screwing state; The mapping relationship between tactile data and twisting state is represented as follows: in, This represents the input tactile data, and the label is the tag for the tightening state, indicating either a tightened state or a state in which it is being tightened.
5. The method according to claim 1, characterized in that, The temporal tactile data processing module, constructed based on the temporal characteristics of tactile data, includes: Based on the temporal characteristics of tactile data, a temporal tactile data processing module is constructed using long short-term memory; Among them, the tactile data processing module based on long short-term memory contains two layers of long short-term memory network with a neuron size of 3, the input data length is 64, and the number of hidden layer neurons is 128. The single-layer long short-term memory network layer consists of multiple long short-term memory units. Each long short-term memory unit contains a cell state and three gates, namely: an input gate, a forget gate, and an output gate. The information stored in the state of the input gate determines the information including two parameters. and Obtained by the sigmoid activation function The value used to determine the update is obtained using the tanh activation function. The new candidate vector is represented by the following expression: in, Weighting for new information updates and The weights and biases of the input gate, and Weights and biases of memory cells; The forget gate determines whether information about the cell state is forgotten and updates the information, including the hidden state from the previous time step. Input with current state The forgetting probability is obtained by feeding the signal into the sigmoid activation function. As shown in the following formula: in, and The weights and biases of the input gate; Output gate outputs memory unit information: in, This is the hidden state of the next layer. This is the output content.
6. The method according to claim 1, characterized in that, Based on the mapping relationship between tactile data and twisting state, the temporal tactile data processing module is invoked to construct a twisting state estimation network, including: Before the temporal tactile data processing module, an attention-based convolutional layer and pooling layer are added, with 3 convolutional layers and 3 pooling layers. The input of the convolutional layer is tactile data of length 10×1. Each convolutional layer has a kernel size of 3×3. The convolutional layer extracts tactile data features from the input 1-dimensional tactile data and maps the 1-dimensional tactile data to 64-dimensional tactile data features. Pooling layers are used to reduce the dimensionality of the tactile data features output by the convolutional layers.
7. The method according to claim 6, characterized in that, The tactile data features extracted by the convolutional layer are characterized as follows: in, This represents the tactile data features after the convolution operation. Indicates the first Layer Tactile data feature mapping of individual neurons Represents the activation function ReLU. For bias; Indicates the first The number of feature maps on the tactile data feature map of the layer. Indicates the first The first layer The feature map and the first feature map The weight matrix between neurons.
8. The method according to claim 6, characterized in that, The dimensionality reduction of the tactile data features output from the convolutional layer is performed using pooling layers, including: The pooling layer uses the following formula to reduce the dimensionality of the tactile data features output by the convolutional layer: in, Representing tactile data features, Indicates the index position on the tactile data feature. Indicates the position within the pooling window. Indicates the size of the pooling window. Step size, For the number of fill layers, when When the value is infinite, the pooling layer takes the maximum value.
9. The method according to claim 1, characterized in that, Based on the aforementioned twisting state estimation network and tactile data, the robot performs autonomous assembly operations, including: The system uses a binocular camera to acquire RGBD images of the working area, and uses a cumulative probability Hough transform (PPHT) algorithm to identify external hexagonal bolts and nuts, to obtain the position and orientation information of the center point of the bolts and nuts to be grasped in an unknown environment. The robot arm is controlled to grasp the top of the bolt and reach above the nut. The robot's motor drives the arm downward to align with the threaded hole. After the nut and the tail surface of the bolt are in full contact, the visual tactile data changes and generates a vertically upward contact force and a tangential force. The robot arm's posture in the task coordinates is calculated using the tactile data. The robot arm's end joint slowly rotates downwards, inputting tactile information with temporal characteristics into a screw-tightening state estimation network. This network then determines and outputs the screw-tightening state of the screw and nut, enabling the robot to autonomously perform the screw-tightening task.