Industrial robot product assembly system and method

Abstract

The invention discloses an industrial robot product assembling system which comprises a work station and a visual tactile sensor, the visual tactile sensor is arranged on an end effector of a mechanical arm of an industrial robot, and the visual tactile sensor is coupled with the work station; a camera is arranged in the visual tactile sensor, the visual tactile sensor is provided with a flexible deformation sensing layer, and the flexible deformation sensing layer makes contact with an assembled product when the industrial robot conducts product assembling. Mark points are arranged on the surface of the flexible deformation sensing layer. When an assembled product makes contact with the flexible deformation sensing layer of the visual tactile sensor, the sensing layer deforms, and meanwhile a camera collects a deformation image, namely visual and tactile original data collected by the visual tactile sensor. According to the invention, the high-resolution visual tactile sensor is utilized to realize precise assembly of the robot.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of industrial robots, and particularly relates to a system and method for automatic assembly of products using an industrial robot. BACKGROUND

[0002] With the continuous progress of science and technology, industrial automation has become the core of the transformation of manufacturing industry. Robot technology plays a crucial role in this process, and through industrial robots, the automatic assembly of products is realized. Due to the flexibility and accuracy of industrial robots, efficient assembly of complex components is achieved. This automatic assembly realized by industrial robots not only reduces the dependence on manual labor, but also reduces human errors, improves the consistency and reliability of the production process.

[0003] Industrial robot assembly systems usually integrate advanced sensors and artificial intelligence algorithms, enabling them to identify different components, perform precise positioning and operations. In addition, these systems can also adjust themselves according to different production needs to adapt to new assembly tasks or product changes. In the fields of automobile manufacturing, electronic device assembly, medical device production, etc., robot automatic assembly has become a key factor in improving productivity and competitiveness.

[0004] Currently, industrial robot assembly systems mostly use visual information or force / torque information to obtain contact modalities to guide assembly tasks, but these sensing technologies still have obvious limitations. For example, visual sensors are easily affected by changes in light and occlusion, and it is difficult to accurately model physical contact, limiting their application in assembly tasks that rely on touch; force / torque sensors have limited ability in accurately tracking object motion and detecting slip; inertial sensors are often disturbed by noise due to the influence of embedded drivers; and curvature sensors have hysteresis and drift problems, affecting their accuracy in dynamic assembly. Therefore, developing a system that utilizes high-resolution visual-tactile sensors to provide fine tactile feedback is of great significance to improving the precision and reliability of robot assembly. SUMMARY

[0005] One of the embodiments of the present disclosure is a robot assembly system based on tactile positioning, which includes a work station, and a visual-tactile sensor electrically coupled to the work station. The visual-tactile sensor is arranged at the end effector of the industrial robot manipulator. The visual-tactile sensor has a built-in camera and a flexible deformation sensing layer that comes into contact with the product being assembled when the industrial robot performs product assembly. The flexible deformation sensing layer has a marker point on its surface.

[0006] When the assembled product contacts the flexible deformation sensing layer of the visual-tactile sensor, the sensing layer deforms, and the camera captures the deformed image, which is the visual and tactile raw data collected by the visual-tactile sensor.

[0007] The product assembly step of the system industrial robot includes:

[0008] The contact state between the end effector of the mechanical arm and the target product is monitored in real time by the visual-tactile sensor.

[0009] The target product is delivered to the end effector of the mechanical arm in a random spatial pose.

[0010] The visual-tactile sensor detects the tactile image, and the end effector of the mechanical arm automatically performs a clamping action to stabilize the product.

[0011] The tactile image after the end effector stabilizes the clamped product is recorded as input data for the neural network.

[0012] The relative pose of the product relative to the end effector is estimated by neural network prediction.

[0013] The pose of the product is adjusted by the mechanical arm according to the output of the neural network, and the assembly task is completed.

[0014] Since robot assembly is a basic process of industrial automation, it is characterized by rich contact modalities and small assembly tolerances, requiring high-precision control. Inspired by the human hand's reliance on tactile sensation for precision assembly, the present embodiment provides a robot assembly system that utilizes a high-resolution visual-tactile sensor to achieve precision assembly. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and other objects, features and advantages of the present exemplary embodiments will become more apparent from the following detailed description read in conjunction with the accompanying drawings. In the drawings, several embodiments of the present application are illustrated by way of example and not limitation in which:

[0016] Figure 1 Schematic diagram of the working principle of a visual-tactile sensor according to one of the embodiments of the present application.

[0017] Figure 2 Schematic diagram of a robot assembling a screw and nut according to one of the embodiments of the present application.

[0018] Figure 3 Schematic diagram of an industrial robot assembling a screw and nut according to one of the embodiments of the present application.

[0019] Figure 4 Schematic diagram of an irregular plastic part assembly according to one of the embodiments of the present application.

[0020] Figure 5 Irregular plastic part assembly diagram according to one of the embodiments of the present application.

[0021] Figure 6 Industrial robot assembly method flow chart according to one of the embodiments of the present application.

[0022] Figure 7 Industrial robot assembly system diagram according to one of the embodiments of the present application.

[0023] 1 - mechanical arm, 2 - end gripper (clamping surface is a flexible sensing layer), 3 - assembly part, 4 - assembly hole, 5 - nut,

[0024] 6 - screw, 7 - first half of the first irregular plastic part (insertion end with three spring buckles), 8 - second half of the first irregular plastic part (corresponding insertion hole), 9 - first component of the second irregular plastic part (insertion with four spring buckles), 10 - second component of the second irregular plastic part (corresponding insertion hole). DETAILED DESCRIPTION

[0025] Currently, there have been studies on the problem of using robots to quickly assemble products. However, most of the schemes lack active sensing and decision-making ability, which easily leads to assembly failure. In addition, the limitations of pre-programmed control easily lead to part damage, such as tilt error when screwing. Therefore, it is necessary to introduce a closed-loop control strategy based on tactile sensing in industrial assembly. In this closed-loop control strategy, the problem of robot sensors needs to be solved first. In existing industrial robot assembly systems, the problems in the aspect of sensors include the following aspects:

[0026] 1. Low density of tactile information.

[0027] The tactile information widely used in current industry is mainly arrayed (0, 1) contact determination information, which can only simply determine whether a certain point is in contact with an object, and cannot provide information about the force or displacement of the contact point. This low-density tactile information leads to low accuracy of pose estimation, limiting the application of the system in precision assembly.

[0028] 2. Susceptible to environmental influences, low positioning accuracy.

[0029] Traditional external vision sensors are susceptible to changes in light and occlusions, especially during assembly, where assembly parts often occlude assembly holes, leading to unsatisfactory assembly accuracy. The positioning accuracy based on vision is usually about 2mm.

[0030] 3. High economic cost.

[0031] Array tactile system requires independent force or magnetic sensors at each contact point, accompanied by complex hardware design and wiring requirements. When performing complex tasks, the number of sensors is large and the process is complex. In addition, high-density arrays require expensive analog-to-digital converters, resulting in high overall cost.

[0032] 4. Low generalization ability

[0033] The control strategy based on array tactile information is often only applicable to specific structured scenarios. Because low-density information limits the decision space, it is difficult to adapt to other environments or task changes, resulting in task failure.

[0034] 5. High labor cost.

[0035] Imitation learning in robot operation training, as an intuitive and simple reinforcement learning method, usually requires experimenters to repeatedly operate the robot arm in each task environment to collect imitation data, and the performance quality is directly related to the amount of data. An efficient imitation strategy requires a large amount of manual data operation.

[0036] According to one or more embodiments, a robot automatic assembly system uses a high-resolution visual tactile sensor to achieve precise assembly. The visual tactile sensor integrates visual and tactile functions, enabling the robot to interact with the environment more finely and naturally. The visual tactile sensor mainly consists of a deformable flexible sensing layer with marker points on the surface and a built-in miniature camera. When an object contacts the flexible sensing layer of the visual tactile sensor, the sensing layer deforms, and the camera captures the deformation image. This is the raw data collected by the visual tactile sensor. An example of this process is shown in FIG. 1. Figure 1

[0037] The information processing process of the visual tactile sensor is as follows: after the visual tactile sensor collects the raw image, the three-dimensional deformation field of the flexible sensing layer surface is obtained based on the photometric stereo method and image tracking algorithm, and then the three-dimensional contact force field is obtained through the inverse finite element method. This step includes:

[0038] Establish the sensing link of the visual tactile sensor three-dimensional contact force field to obtain the simplified physical model and boundary conditions;

[0039] Discretize the visual tactile sensor unit using high-performance models and parameters;

[0040] According to the linear finite element theory, the forward mapping relationship between the three-dimensional contact force field and the three-dimensional deformation field of the flexible sensing layer is constructed;

[0041] Through comprehensive analysis, the overall stiffness matrix is obtained;

[0042] The three-dimensional deformation field of the flexible sensing layer surface is obtained using the photometric stereo method and image tracking algorithm;​

[0043] The transfer function from the three-dimensional deformation field to the three-dimensional contact force field is obtained by multi-layer inverse finite element analysis, and the three-dimensional contact force field is calculated.

[0044] The visual and tactile information obtained by the visual-tactile sensor is fused and positioned, and the pose estimation of the object operated by the robot is realized by fusing the visual and tactile information. The visual information is usually obtained by an externally arranged camera, and the tactile information comes from the force or magnetic sensor on the gripper of the manipulator. These sensors are generally distributed in an array (m*n) and return contact (1) or non-contact (0) information. The method includes the steps of:

[0045] Obtain visual data of the object through the image sensor, and determine the position and direction of the object in space;

[0046] Capture the precise coordinates of the contact with the object using force / tactile sensors;

[0047] Convert the coordinates of the tactile sensor to the 3D model coordinate system of the object;

[0048] Find the nearest triangular patch for each contact point and map the tactile data to the texture plane of the 3D model.

[0049] Due to the complexity of the robot assembly task, the high degree of freedom of the robot joints and the diversity of their types, physical modeling becomes difficult. Therefore, machine learning can be used to obtain an end-to-end control strategy to achieve automation in a specific assembly environment. Imitation learning is often used as a basic control strategy due to its intuitiveness and wide applicability, so the robot imitation learning method can include the following steps:

[0050] Use the Faster R-CNN model for object detection, and determine the object motion state through pixel displacement and difference analysis;

[0051] Construct a rigid kinematics model of the robot arm and a dynamics or elastic model of the target object;

[0052] Adopt a generative adversarial network and a reinforcement learning algorithm to train the imitation learning model;

[0053] Introduce model optimization through gradient descent and fitness function.

[0054] In the visual tactile sensor in the embodiments of the present disclosure, the general shape of the contact object can be first identified from the tactile information, the deformation depth field corresponding to the entire contact area can be obtained according to the photometric stereo method, and the planar displacement field of the contact area can be obtained through the optical flow method combined with the marker points on the surface of the flexible sensing layer, and finally the corresponding continuous contact force field can be obtained through the inverse finite element method. The extremely high-density tactile information obtained by the visual tactile sensor in the embodiments of the present disclosure is of decisive significance for the success or failure of the robot manipulator assembly task.

[0055] In the embodiments of the present disclosure, the visual tactile sensor is built-in with a miniature camera to record tactile information, which is not affected by the external environment, and a complex neural network structure for estimating the relative pose of the assembly part, which has a large amount of calculation parameters, can make the positioning accuracy of the assembly part reach 0.1 mm, and can meet most industrial assembly tasks. The visual tactile sensor combined with the neural network can predict and estimate the pose between the robot manipulator and the part, and correct the assembly error on the basis of closed-loop control, so as to complete the precision assembly task.

[0056] According to one or more embodiments, as shown in Figure 6 A robot manipulator assembly method based on a visual tactile sensor and a neural network includes the following steps:

[0057] Step 101, real-time contact information detection, using a visual tactile sensor to monitor the contact state between the manipulator gripper and the target object in real time;

[0058] Step 102, object pose randomization, delivering the target object to the gripper of the manipulator in a random spatial pose;

[0059] Step 103, automatic clamping response, once the visual tactile sensor detects the tactile image, the manipulator gripper automatically performs a clamping action to stabilize the object;

[0060] Step 104, tactile image recording and preprocessing, recording the tactile image after the gripper stably clamps the object, and performing necessary preprocessing to serve as input data for the neural network;

[0061] Step 105, neural network prediction, inputting the preprocessed tactile image into the neural network, and using a deep learning model to predict and estimate the accurate pose of the object relative to the gripper;

[0062] Step 106, pose adjustment and assembly completion, the manipulator adjusts the pose of the object according to the output of the neural network to ensure that it meets the predetermined assembly requirements and completes the assembly task.

[0063] The embodiment of the present disclosure adopts a visual-tactile sensor integrating visual and tactile functions to accurately capture contact information between a robot arm and an object, and realizes rapid and stable clamping of the object based on automatic control logic fed back by the sensor. The visual-tactile sensor includes preprocessing operations such as noise removal and contrast enhancement on a tactile image to improve the prediction accuracy of a neural network, and an advanced neural network algorithm such as a convolutional neural network (CNN) is used for image feature extraction and pose prediction. The robot arm control system adjusts an actuator in real time according to the prediction result of the neural network to compensate for pose deviation.

[0064] The embodiment of the present disclosure has the beneficial effect of realizing high-precision detection and adjustment of the object pose through the combination of the visual-tactile sensor and the neural network. Not only does it reduce manual intervention and improve the automation level of the assembly process, but it is also suitable for processing objects with random poses, enhances the adaptability of the system, and quickly responds to meet the real-time requirements in a dynamic assembly environment.

[0065] The embodiment of the present disclosure is suitable for assembly tasks that require high precision and high automation level, such as precision assembly of electronic components, automated assembly of automobile parts, and other industrial scenarios. Through intelligent detection and adjustment, the accuracy and efficiency of the assembly process are ensured.

[0066] According to one or more embodiments, a robot assembly method based on a visual-tactile sensor positioning algorithm includes the following steps:

[0067] S201. The assembly strategy reads the tactile information collected by the visual-tactile sensor on the gripper in real time.

[0068] At the beginning of the assembly program, the control strategy of the assembly system immediately starts to monitor the visual-tactile sensor on the gripper in real time. The sensor is specially designed to capture tactile information generated when it comes into contact with the assembly part. These information contains key information about the position, shape and contact pressure of the assembly part, which is necessary for precise assembly.

[0069] S202. Contact detection algorithm deployment.

[0070] The assembly system analyzes the tactile information provided by the visual-tactile sensor to determine whether the assembly part has touched the gripper. This step is crucial to avoid misoperation and ensure the correct progress of the assembly process, as it allows the system to confirm that the assembly part is in the correct position and ready for subsequent grasping.

[0071] S203. The assembly part is delivered to the sensor by hand or other industrial feeding method.

[0072] Once the system confirms that the assembly touches the gripper, the assembly will be precisely delivered into the working range of the sensor by manual or other industrial automatic feeding methods (such as parallel conveyor belts, screw conveyors, etc.). This step ensures that the assembly can be continuously and stably supplied, supporting efficient assembly.

[0073] S204. The tactile information changes while the gripper clamps the assembly.

[0074] When the assembly reaches the gripper, the tactile information will change significantly. The system automatically instructs the gripper to close and firmly clamp the assembly through the detected image changes. During this process, the closing force and speed of the gripper are precisely controlled to adapt to different types and fragility of assemblies.

[0075] S205. After obtaining stable tactile information by clamping the assembly, the neural network predicts the relative pose between the assembly and the sensor.

[0076] After clamping the assembly and obtaining stable tactile information, the neural network in the system begins to work, predicting the relative pose between the assembly and the sensor. The neural network uses deep learning algorithms to process the input tactile information, calculating the exact orientation and angle of the assembly to provide guidance for the next precise placement.

[0077] S206. Adjust the pose of the robotic arm to insert into the corresponding assembly position.

[0078] Based on the pose prediction results provided by the neural network, the robotic arm makes corresponding adjustments to precisely insert the assembly into the predetermined assembly position. This step requires the robotic arm control system to have high flexibility and precision to ensure the quality and efficiency of assembly.

[0079] S207. The execution program decides whether to adjust the strategy based on error conditions.

[0080] Finally, the execution program will decide whether to adjust the assembly strategy based on any errors or deviations that may occur during implementation. The system has adaptive capabilities to optimize operation steps according to actual conditions, ensuring the success rate and efficiency of the entire assembly process.

[0081] According to one or more embodiments, a robotic automatic assembly system includes:

[0082] A workstation (Workstation) connected to a visuo-tactile sensor, serving as the control center of the system, running control software, responsible for issuing instructions and managing the operation of the entire assembly system;

[0083] A robotic arm (Arm) and a mechanical hand (Hand) or robotic end effector connected to it, used to grasp, transport or manipulate objects.

[0084] The workstation sends control instructions to the robotic arm system according to preset programs or operator input. After receiving the instructions, the robotic arm performs tasks through the coordinated action of its arm and hand. During the execution process, the system continuously adjusts the control to adapt to task requirements and environmental changes.

[0085] The operating state of the robotic arm and hand is fed back to the workstation PC in real time for monitoring and necessary intervention. The control instructions sent by the workstation PC guide the robotic arm and hand to perform specific tasks.

[0086] The workstation PC communicates with the robotic arm ROS control interface through the application program interface (API) to send commands and receive states, realizing communication between the workstation and the robotic arm controller. The low-level motion control and real-time feedback are realized by embedding the robotic arm bottom control algorithm in the robotic arm controller. Ethernet is used as the communication medium to transmit data and commands between the workstation and the robotic arm controller, achieving real-time control of the robotic arm.

[0087] The workstation sends control commands to the robotic arm controller through the robotic arm control API, and exchanges data in real time through Ethernet, including commands and feedback information. After receiving the commands, the robotic arm controller executes the bottom control algorithm to realize real-time motion control of the robotic arm. The robotic arm controller feeds back the execution state and sensor data to the workstation.

[0088] The robotic automatic assembly system of the embodiments of the present disclosure integrates the workstation, the ROS control interface and the robotic arm controller to form a coordinated and consistent automatic control system. The Ethernet communication protocol is used to ensure high-speed data transmission and low delay to meet the real-time requirements of robotic arm control. High-precision motion control is realized through the robotic arm bottom control algorithm.

[0089] According to one or more embodiments, a data transmission path and control method of a robotic control system are shown as follows Figure 7 The system is particularly suitable for the working conditions set in actual industrial environments, and advanced visual-tactile sensor and robotic arm control technology is adopted. The architecture of the control system mainly includes two parts: first, the reception and control of visual-tactile sensor data; second, the operation and state monitoring of the robotic arm.

[0090] 1) Reception and control of visual-tactile sensor data:

[0091] The visual-tactile sensor is equipped with a built-in camera that continuously sends tactile information to the central processing node through a preset transmission protocol.

[0092] The host computer monitors the node to receive image data and controls the data acquisition process of the camera through a feedback mechanism.

[0093] 2) Control of the robotic arm and reception of state data:

[0094] • The top-level software operates the robotic arm by calling the underlying control algorithm, using a wired Ethernet connection to achieve precise control of each joint of the robotic arm, thereby managing the overall pose of the robotic arm.

[0095] • At the same time, sensors at each joint of the robotic arm collect state data and return it to the workstation host computer along the same transmission path for processing and feedback adjustment.

[0096] This control system design fully considers the high requirements for accuracy and reliability in industrial applications, providing a complete solution from data acquisition to mechanical operation, ensuring the efficiency and safety of robotic operation.

[0097] In the automatic tactile information acquisition process designed specifically for assembly tasks, the embodiment of the present disclosure constructs a high-complexity pose estimation network with carefully adjusted hyperparameters and a large number of network parameters. This network can accurately infer the precise pose of the part from complex tactile data, optimizing the entire process from data acquisition to processing. In order to achieve efficient integration and processing of tactile information, the embodiment of the present disclosure also constructs a comprehensive assembly system that not only integrates advanced prediction algorithms and control processes, but also improves the smoothness of operation and the response speed of the system through integrated design, ensuring the efficiency and accuracy of assembly tasks.

[0098] In the robotic assembly method of the present disclosure, the algorithm for estimating the pose of the object from the tactile information includes real-time acquisition of tactile information using visual-tactile sensors on the gripper, and calculation of the precise pose of the assembly part relative to the gripper through an advanced pose estimation network. This algorithm is superior to traditional Hough circle detection or ellipse detection methods, with higher accuracy and faster processing speed, thereby reliably guiding the precision assembly process. The assembly system of the present disclosure, based on adaptive assembly of pose estimation, integrates key technologies such as real-time transmission of tactile information, network prediction of pose, and control of the robotic arm, and can achieve high precision and high efficiency in assembly in a wide range of industrial environments. The adaptive capability of the system ensures that optimal operation performance can be maintained under varying assembly conditions.

[0099] In order to verify the embodiments of the present disclosure, three assembly conditions are designed to verify the effectiveness of the present disclosure, including high-precision assembly of screws and nuts and shaft-hole assembly of irregular plastic parts. Figures 2 to 5 By real-time acquisition and analysis of tactile information, the system can significantly improve assembly precision and efficiency. This innovative assembly system is not only suitable for complex industrial tasks, but also can be applied to medical and military fields, and its precise positioning capability provides a new technical path for realizing automated assembly, with high practical value and application potential.

[0100] Due to the strong adaptability and high generalization, in the embodiments of the present disclosure, 100% successful assembly is achieved for the three designed assembly conditions. Meanwhile, the assembly strategy of the present disclosure can also be migrated to any other similar shaft-hole assembly task. Thanks to the complexity and high parameter amount of the network in the embodiments of the present disclosure, for other tasks, only simple data collection and training are needed to achieve a high assembly success rate.

[0101] Meanwhile, due to the low economic and manpower cost and the high-density tactile information, only a single sensor arranged in an important contact area can collect sufficient effective information for pose estimation. Moreover, all the processes of network training can be programmed to realize automation without the intervention of natural persons.

[0102] In summary, the beneficial effects of the present disclosure include:

[0103] 1. The present disclosure significantly improves the density of tactile information through a high-resolution visual-tactile sensor. The visual-tactile sensor identifies the general shape of the contacted object, obtains the deformation depth field through photometric stereo method, and obtains the plane displacement field by combining the optical flow method and the marker point technology. Finally, the continuous contact force field is calculated through the inverse finite element method. The high-density contact information provides extremely high operation value and decision basis in complex industrial environments.

[0104] 2. The present disclosure uses a visual-tactile sensor with a built-in miniature camera to record tactile information, which is not affected by the external environment, and the positioning accuracy can be improved to 0.1 mm. This built-in method enhances the robustness of the sensor in dynamic and complex environments.

[0105] 3. The present disclosure uses a visual-tactile sensor to obtain high-density information, reducing the need for a large number of sensors, thereby reducing the hardware cost and maintenance cost of the system.

[0106] 4. The robot assembly system of the present disclosure expands the decision space by using high-density visual-tactile information, improves the generalization ability of the control strategy in variable environments, and enables the system to cope with complex and variable assembly tasks.

[0107] 5. The present disclosure designs an automatic tactile information collection program, so that the entire training process no longer needs manual operation of the robot arm, greatly reducing the labor cost and improving the efficiency and consistency of data collection.

[0108] It should be understood that in the embodiments of the present disclosure, the term "and / or" only describes the association relationship of the associated objects, which means that there can be three relationships. For example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0109] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0110] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. In actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0111] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0112] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0113] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An industrial robot product assembly system, characterized by, The system comprises: a workstation; a visual-haptic sensor arranged at an end effector of the industrial robot arm, and coupled to the workstation; the visual-haptic sensor is internally provided with a camera, and has a flexible deformation sensing layer which is in contact with the product being assembled when the industrial robot is assembling the product.

2. The system of claim 1, wherein, The flexible deformation sensing layer has a mark point on its surface.

3. The system of claim 1, wherein, When the end effector is a gripper, the clamping surface of the gripper is the flexible deformation sensing layer.

4. The system of claim 2, wherein, When the product being assembled contacts the flexible deformation sensing layer of the visual-haptic sensor, the sensing layer deforms, and the camera captures the deformed image, which is the visual and haptic raw data captured by the visual-haptic sensor.

5. The system of claim 4, wherein, The product assembly steps of the industrial robot of the system comprise: real-time monitoring of the contact state between the end effector of the robot arm and the target product by the visual-haptic sensor; delivering the target product to the end effector of the robot arm in a random spatial pose; detecting the haptic image by the visual-haptic sensor, and automatically performing a clamping action by the end effector of the robot arm to stabilize the product; recording the haptic image after the end effector stabilizes the clamped product as input data for the neural network; estimating the relative pose of the product relative to the end effector by neural network prediction; adjusting the pose of the product by the robot arm according to the output of the neural network to complete the assembly task.

6. The system of claim 4, wherein, The processing process of the visual and haptic raw data obtained from the visual-haptic sensor by the workstation comprises: first, identifying the general shape of the contacted object from the haptic data, second, obtaining the corresponding deformation depth field of the entire contact area according to the photometric stereo method, and at the same time obtaining the planar displacement field of the contact area by the optical flow method combined with the mark point on the surface of the flexible deformation sensing layer, and finally obtaining the corresponding continuous contact force field by the inverse finite element method.

7. An industrial robot product assembly method based on the system according to claim 1, characterized by The method comprises the following steps: real-time monitoring of the contact state between the end effector of the robot arm and the target product by the visual-haptic sensor; delivering the target product to the end effector of the robot arm in a random spatial pose; detecting the haptic image by the visual-haptic sensor, and automatically performing a clamping action by the end effector of the robot arm to stabilize the product; recording the haptic image after the end effector stabilizes the clamped product as input data for the neural network; estimating the relative pose of the product relative to the end effector by neural network prediction; adjusting the pose of the product by the robot arm according to the output of the neural network to complete the assembly task.

8. The method of claim 7, wherein, The processing process of the visual and haptic raw data obtained from the visual-haptic sensor comprises: first, identifying the general shape of the contacted object from the haptic data, second, obtaining the corresponding deformation depth field of the entire contact area according to the photometric stereo method, and at the same time obtaining the planar displacement field of the contact area by the optical flow method combined with the mark point on the surface of the flexible deformation sensing layer, and finally obtaining the corresponding continuous contact force field by the inverse finite element method.

9. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the method of any one of claims 7 to 8.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 7 to 8.

Images