Collaborative Robot Multimodal Sensing System Based on Digital Control

Abstract

The present invention discloses a multi-modal perception system for collaborative robots based on digital control, belonging to the technical field of collaborative robot control. The present invention can realize the perception of various modal information such as RGB images, depth images, and pressure sensors; and uses the maximum distance between any two points on the outer contour line of the workpiece top surface as the size characteristic parameter of the workpiece, which can effectively ensure that the pneumatic gripper at the end of the workpiece grasping robot can complete the grasping task of the current work; also through the contour detection technology combined with the principle of the depth camera, it can cleverly and accurately calculate the workpiece distance information, that is, it can control the workpiece grasping robot to drive the pneumatic gripper to descend the corresponding distance to ensure the stable progress of the subsequent grasping work; by setting a pressure sensor group in the corresponding areas of the two clamping plates of the pneumatic gripper and combining with the pre-clamping operation, it can accurately identify the workpiece deflection angle information, providing an accurate basis for the subsequent angle adjustment of the pneumatic gripper.

Description

Technical Field

[0001] The present invention relates to the technical field of collaborative robot control, and particularly to a multi-modal perception system for collaborative robots based on digital control. Background Art

[0002] A collaborative robot is a robotic system designed to work safely and efficiently with human operators in a shared workspace, aiming to improve production efficiency, quality, and flexibility through close cooperation with humans. Collaborative robots have the characteristics of high safety, strong flexibility, easy operation, and strong collaboration ability. Among them, high safety means being equipped with a variety of safety sensors and mechanisms, such as collision detection, force limit control, etc., which can stop or reduce power in time when coming into contact with humans to avoid harm to humans; strong flexibility means being able to be quickly reprogrammed and adjusted to adapt to different tasks and working environments; easy operation means having an intuitive human-machine interaction interface, without the need for professional programming knowledge, and ordinary workers can operate and program after simple training, reducing the usage threshold; strong collaboration ability means being able to perceive human actions and intentions in real time with the help of advanced sensors and algorithms, and collaborate with humans naturally and smoothly to complete complex tasks.

[0003] In the industrial production process, how to better utilize collaborative robots to help operators complete tasks such as precise grasping of workpieces and welding quality assessment is an urgent problem to be solved. For this reason, a multi-modal perception system for collaborative robots based on digital control is proposed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: how to better utilize collaborative robots to help operators complete tasks such as precise grasping of workpieces and welding quality assessment, and a multi-modal perception system for collaborative robots based on digital control is provided.

[0005] The present invention solves the above technical problems through the following technical solutions. The present invention includes a workpiece size perception module, a gripper descent distance perception module, a workpiece deflection angle perception module, a solder joint image acquisition module, and a welding quality assessment module;

[0006] The workpiece size perception module is used to identify the size of the workpiece under the workpiece grasping robot and obtain workpiece size information;

[0007] The gripper descent distance perception module is used to identify the distance that the gripper needs to descend during grasping and obtain gripper descent distance information;

[0008] The workpiece deflection angle perception module is used to identify the deflection angle of the workpiece in the horizontal plane after the pneumatic gripper descends in place and obtain workpiece deflection angle information;

[0009] The solder joint image acquisition module is used to acquire the solder joint image after the workpiece spot welding is completed, and obtain the area feature information of each solder joint according to the solder joint image;

[0010] The welding quality evaluation module is used to score the welding quality of the current workpiece according to the area feature information of each solder joint.

[0011] Furthermore, the workpiece size perception module includes a workpiece image shooting unit, a workpiece image preprocessing unit, and a workpiece image recognition unit; the workpiece image shooting unit is used to vertically downward shoot the workpiece by using a camera to obtain a top view image of the workpiece; the workpiece image preprocessing unit is used to perform noise reduction processing on the top view image of the workpiece; the workpiece image recognition unit is used to recognize the top surface of the workpiece in the top view image of the workpiece by using a trained first object detection model, and then obtain the workpiece size information, and recommend the model of the pneumatic gripper to the operator according to the workpiece size information.

[0012] Furthermore, in the workpiece image recognition unit, the specific processing process is as follows:

[0013] Step S11: Recognize the top surface of the workpiece in the top view image of the workpiece by using a trained object detection model, and obtain the detection frame of the top surface of the workpiece and the coordinates of its upper left corner point and lower right corner point in the image;

[0014] Step S12: Crop the area of the detection frame of the top surface of the workpiece from the top view image of the workpiece to obtain the detection frame image of the top surface of the workpiece;

[0015] Step S13: Use the contour detection function in OpenCV to perform external contour detection on the top surface of the workpiece, and obtain the external contour line of the top surface of the workpiece and the coordinate information of each point on the external contour line in the image;

[0016] Step S14: Calculate the distance between any two points on the external contour line, and select the distance between the two contour points with the largest distance as the size feature parameter of the workpiece, denoted as Dmax, that is, obtain the workpiece size information;

[0017] Step S15: After obtaining the workpiece size information, look up and obtain the corresponding pneumatic gripper model according to the size feature parameter Dmax in the preset size feature parameter - pneumatic gripper model database, and recommend the found pneumatic gripper model to the operator. The operator replaces the pneumatic gripper at the end of the workpiece grasping robot with the corresponding model, where the size feature parameter - pneumatic gripper model database stores the corresponding relationship between the size feature parameter values and the pneumatic gripper models.

[0018] Further, the jaw descending distance sensing module includes a depth image capturing unit, a depth image preprocessing unit, and a workpiece distance information obtaining unit; the depth image capturing unit is configured to vertically capture the workpiece downward at the same capturing position as the camera in the workpiece image capturing unit by using a depth camera, so as to obtain a top-down depth image of the workpiece; the depth image preprocessing unit is configured to perform noise reduction processing on the top-down depth image of the workpiece; the workpiece distance information obtaining unit is configured to obtain the jaw descending distance information through contour detection processing.

[0019] Further, both the top-down image of the workpiece and the top-down depth image of the workpiece include the complete workpiece. The projection point of the optical axis of the camera in the workpiece image capturing unit on the top-down image of the workpiece is located at the center point C of the top surface contour of the workpiece. The projection point of the optical axis of the depth camera in the depth image capturing unit on the top-down depth image of the workpiece is also located at the center point C of the top surface contour of the workpiece. The resolution and size of the top-down depth image of the workpiece are the same as those of the top-down image of the workpiece.

[0020] Further, in the workpiece distance information obtaining unit, the specific processing process is as follows:

[0021] Step S21: Use the contour detection function in OpenCV to identify the outer contour of the workpiece in the top-down depth image of the workpiece, and obtain the outer contour line of the workpiece and the coordinate information of each point on the outer contour line in the image;

[0022] Step S22: Read the pixel values of each point on the outer contour line of the workpiece, and record the contour point with the smallest pixel value as x, and the corresponding pixel value as Px. Px is the distance between this point and the position point S of the depth camera;

[0023] Step S23: Connect the minimum contour point x to the center point C' of the bottom surface contour of the workpiece to obtain the line segment xC'. At the same time, the line segment xC' intersects the top surface contour line of the workpiece at point J. Among them, the projection points of the center point C' of the bottom surface contour of the workpiece and the center point C of the top surface contour of the workpiece on the top-down depth image and the top-down image of the workpiece coincide;

[0024] Step S24: Calculate the length of the line segment JC in the corresponding top-down image of the workpiece according to the positions of point J and the center point C, and then transform the length of the line segment JC through the transformation relationship between the pre-calibrated pixel coordinate system and the world coordinate system to obtain the length of the line segment JC in the world coordinate system, denoted as Ljc;

[0025] Step S25: Use Ljc as the actual length of the line segment xC' between the minimum contour point x and the center point C' of the bottom surface contour of the workpiece in the world coordinate system;

[0026] Step S26: In the world coordinate system, a corresponding right triangle is formed by the depth camera position point S, the minimum contour point x, and the center point C' of the workpiece bottom surface contour. Among them, the line segment SC' is perpendicular to the line segment xC'. The length Px of the line segment Sx is known, and the length Ljc of the line segment xC' is known. The length of the line segment SC' is calculated using trigonometric functions, that is, the vertical height value between the depth camera position point S and the center point C' of the workpiece bottom surface contour is obtained, denoted as H1;

[0027] Step S27: Obtain the vertical distance value H0 between the known depth camera position point S and the lowest position point of the pneumatic gripper, and subtract H0 from H1 to obtain H2, that is, the gripper descent distance information is obtained.

[0028] Furthermore, the workpiece deflection angle sensing module includes a pre-clamping unit, a pressure detection unit, and a workpiece deflection angle acquisition unit; the pre-clamping unit is used to perform a pre-clamping operation on the workpiece after the pneumatic gripper descends in place; the pressure detection unit is used to detect and obtain the pressure value on the corresponding clamping plate using a pressure sensor after any clamping plate of the pneumatic gripper contacts and squeezes the workpiece surface; the workpiece deflection angle acquisition unit is used to obtain the workpiece deflection angle information according to the pressure value on the corresponding clamping plate, and adjust the angle of the pneumatic gripper according to the workpiece deflection angle information.

[0029] Furthermore, in the workpiece deflection angle acquisition unit, the specific processing process is as follows:

[0030] Step S31: Use a pressure sensor to obtain the pressure value of the corresponding area on the clamping plate that contacts and squeezes the workpiece surface. Among them, on the two clamping plates of the pneumatic gripper, two areas A1 and A2 are arranged on the first clamping plate. The areas A1 and A2 are respectively located at the two side edges of the first clamping plate. A plurality of pressure sensors are vertically arranged in each area. Two areas B1 and B2 are arranged on the second clamping plate. The areas B1 and B2 are respectively located at the two side edges of the second clamping plate. A plurality of pressure sensors are vertically arranged in each area;

[0031] Step S32: When the first clamping plate contacts and squeezes the workpiece surface and the second clamping plate does not contact and squeeze the workpiece surface, first calculate the arithmetic pressure means A1avg and A2avg of the pressure sensors in the two areas A1 and A2 respectively, then compare the magnitudes of the arithmetic pressure means A1avg and A2avg, and calculate the difference between the arithmetic pressure means A1avg and A2avg, denoted as the pressure difference Ac; when the second clamping plate contacts and squeezes the workpiece surface and the first clamping plate does not contact and squeeze the workpiece surface, first calculate the arithmetic pressure means B1avg and B2avg of the pressure sensors in the two areas B1 and B2 respectively, then compare the magnitudes of the arithmetic pressure means B1avg and B2avg, and calculate the difference between the arithmetic pressure means B1avg and B2avg, denoted as the pressure difference Bc;

[0032] Step S33: Look up and obtain the corresponding workpiece deflection angle θ according to the pressure difference Ac or the pressure difference Bc in the preset pressure difference - deflection angle database, that is, obtain the workpiece deflection angle information, where the pressure difference - deflection angle database stores the corresponding relationship between the pressure difference and the deflection angle;

[0033] Step S34: When A1avg is greater than A2avg or B1avg is greater than B2avg, drive the pneumatic gripper to rotate counterclockwise by an angle of θ; when A1avg is less than A2avg or B1avg is less than B2avg, drive the pneumatic gripper to rotate clockwise by an angle of θ.

[0034] Furthermore, the specific processing process of the solder joint image acquisition module is as follows:

[0035] Step S41: Use an industrial camera to photograph the welding area to obtain a solder joint image, and the solder joint image includes multiple independent solder joints;

[0036] Step S42: Use the trained second object detection model to identify each solder joint in the solder joint image, and obtain each solder joint detection frame and the coordinates of its upper left corner point and lower right corner point in the solder joint image;

[0037] Step S43: Use the contour detection function in OpenCV to perform outer contour detection on the solder joints in each solder joint detection frame, and obtain the coordinates of each solder joint contour line and each point on the outer contour line in the image;

[0038] Step S44: Read the pixel area of the internal region of each solder joint contour line as the area feature information of each solder joint, denoted as Ri, where i represents the i-th solder joint.

[0039] Furthermore, the specific processing process of the welding quality evaluation module is as follows:

[0040] Step S51: Calculate the difference between each solder joint area feature information Ri and the standard solder joint area feature value R0, denoted as Rci;

[0041] Step S52: Calculate the average value of the difference Rci, denoted as the average value Rcavg;

[0042] Step S53: Obtain the corresponding welding quality score T according to the average value Rcavg in the preset scoring database, where the scoring database stores the corresponding relationship between the average value Rcavg and the welding quality score T.

[0043] The present invention has the following advantages compared with the prior art: The collaborative robot multi-modal perception system based on digital control can realize the perception of various modal information such as RGB images, depth images, and pressure sensors through the workpiece size perception module, gripper descent distance perception module, workpiece deflection angle perception module, and solder joint image acquisition module set; By using the maximum distance between any two points on the outer contour line of the workpiece top surface as the size characteristic parameter of the workpiece, and then obtaining the corresponding pneumatic gripper model, it can effectively ensure that the pneumatic gripper at the end of the workpiece grasping robot can complete the grasping task of the current work, greatly improving the accuracy and efficiency of the pneumatic gripper replacement work; Through the contour detection technology combined with the principle of the depth camera, the workpiece distance information can be calculated skillfully and accurately. After obtaining the workpiece distance information, the workpiece grasping robot can be controlled to drive the pneumatic gripper to descend the corresponding distance to ensure the stable progress of the subsequent grasping work; By setting a pressure sensor group in the corresponding areas of the two clamping plates of the pneumatic gripper and combining the pre-clamping operation, the workpiece deflection angle information can be accurately identified, providing an accurate basis for the subsequent angle adjustment of the pneumatic gripper, and avoiding unnecessary wear on the gripper caused by too large a workpiece deflection angle during subsequent grasping; Using the pixel area of the internal area of the solder joint contour line as the area characteristic information of each solder joint, the accurate scoring of the welding quality of the current workpiece can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic block diagram of the structure of the collaborative robot multi-modal perception system based on digital control in an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of the positions of each point in the top-down depth image of the workpiece in an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of the positions of the clamping plate and the workpiece during the calculation of the workpiece deflection angle in an embodiment of the present invention (top view). DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The embodiments of the present invention will be described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0048] As Figures 1 to 3 shown, this embodiment provides a technical solution: A collaborative robot multi-modal perception system based on digital control, mainly used for the grasping of precision workpieces and the evaluation of welding quality, includes a workpiece size perception module, a gripper descent distance perception module, a workpiece deflection angle perception module, a solder joint image acquisition module, and a welding quality evaluation module;

[0049] In this embodiment, the workpiece size perception module, the gripper descent distance perception module, and the workpiece deflection angle perception module are arranged on the workpiece grasping robot; the solder joint image acquisition module is arranged on the vision detection robot, and the welding quality evaluation module is arranged in the background processing center.

[0050] It should be noted that in this embodiment, the workpiece grasping robot and the vision detection robot are both four-axis manipulators, mainly with four axes of X, Y, Z, and R. Among them, the X-axis is the moving axis in the horizontal direction; the Y-axis is the moving axis in the horizontal direction, which cooperates with the X-axis to control the position movement of the tool in the horizontal plane; the Z-axis is the moving axis in the vertical direction, mainly controlling the position movement of the tool at the end of the manipulator in the vertical direction; the R-axis is the rotation axis, also known as the rotating wrist, which is responsible for controlling the posture of the tool at the end of the manipulator to achieve more complex tasks. A pneumatic gripper (tool) that can be quickly disassembled and replaced is installed at the end of the workpiece grasping robot, and an industrial camera (tool) for taking solder joint images is installed at the end of the vision detection robot.

[0051] In this embodiment, the workpiece size perception module is used to identify the size of the workpiece under the workpiece grasping robot and obtain workpiece size information.

[0052] More specifically, the workpiece size perception module includes a workpiece image shooting unit, a workpiece image preprocessing unit, and a workpiece image recognition unit; the workpiece image shooting unit is used to vertically shoot the workpiece downward by using a camera (installed at the end of the workpiece grasping robot) to obtain a top view image (RGB image) of the workpiece; the workpiece image preprocessing unit is used to perform noise reduction processing on the top view image of the workpiece to improve the quality of the top view image of the workpiece; the workpiece image recognition unit is used to use the trained first target detection model to identify the top surface of the workpiece in the top view image of the workpiece, and then obtain the workpiece size information, and recommend the model of the pneumatic gripper to the operator according to the workpiece size information, and the operator replaces the pneumatic gripper at the end of the workpiece grasping robot with the corresponding model.

[0053] It should be noted that in this embodiment, the workpiece is a regular hexahedron (cuboid) structure, horizontally placed on the placement platform, and will be used as the housing of the car navigator module after subsequent welding and subsequent processing. The sizes of different workpieces are different. The top view image of the workpiece includes the complete workpiece. The projection point of the optical axis of the camera in the workpiece image shooting unit on the top view image of the workpiece is located at the center point of the top surface contour of the workpiece.

[0054] More specifically, the first target detection model is trained based on the YOLO V3s detection network. The YOLOV3s detection network has the ability of multi-target recognition and can accurately recognize different targets through the training of different sample images when needed.

[0055] More specifically, in the workpiece image recognition unit, the specific processing procedure is as follows:

[0056] Step S11: Use the trained object detection model to recognize the top surface of the workpiece in the top-down image of the workpiece, and obtain the detection frame of the top surface of the workpiece and the coordinates of the upper left corner point and the lower right corner point in the image;

[0057] Step S12: Crop the area of the detection frame of the top surface of the workpiece from the top-down image of the workpiece to obtain the image of the detection frame of the top surface of the workpiece;

[0058] Step S13: Use the contour detection function in OpenCV to detect the outer contour of the top surface of the workpiece, and obtain the outer contour line of the top surface of the workpiece and the coordinate information of each point on the outer contour line in the image;

[0059] Step S14: Calculate the distance between any two points on the outer contour line, and select the distance between the two contour points with the largest distance as the dimensional feature parameter of the workpiece, denoted as Dmax, that is, obtain the dimensional information of the workpiece;

[0060] Step S15: After obtaining the dimensional information of the workpiece, look up the corresponding pneumatic gripper model in the preset dimensional feature parameter - pneumatic gripper model database according to the dimensional feature parameter Dmax, recommend the found pneumatic gripper model to the operator, and the operator replaces the pneumatic gripper at the end of the workpiece gripping robot with the corresponding model.

[0061] Using the maximum distance between any two points on the outer contour line of the top surface of the workpiece as the dimensional feature parameter of the workpiece, and then obtaining the corresponding pneumatic gripper model can effectively ensure that the pneumatic gripper at the end of the workpiece gripping robot can complete the gripping task of the current work, and greatly improve the accuracy and efficiency of the pneumatic gripper replacement work.

[0062] More specifically, in the step S15, the dimensional feature parameter - pneumatic gripper model database stores the corresponding relationship between the dimensional feature parameter values and the pneumatic gripper models.

[0063] In this embodiment, the gripper descending distance sensing module is used to recognize the distance that the gripper needs to descend during gripping, and obtain the gripper descending distance information.

[0064] More specifically, the jaw descending distance sensing module includes a depth image capturing unit, a depth image preprocessing unit, and a workpiece distance information obtaining unit. The depth image capturing unit is configured to vertically capture a workpiece downward at the same capturing position as the camera in the workpiece image capturing unit by using a depth camera (installed at the end of the workpiece grasping robot), so as to obtain a top-down depth image of the workpiece. The depth image preprocessing unit is configured to perform noise reduction processing on the top-down depth image of the workpiece to improve the quality of the top-down depth image of the workpiece. The workpiece distance information obtaining unit is configured to obtain the jaw descending distance information through contour detection processing.

[0065] It should be noted that the top-down depth image of the workpiece also includes the complete workpiece, and the resolution and size of the top-down depth image of the workpiece are the same as those of the top-down image of the workpiece. See Figure 2 , and the projection point of the optical axis of the depth camera in the depth image capturing unit on the top-down depth image of the workpiece is located at the center point C of the top surface contour of the workpiece.

[0066] More specifically, in the workpiece distance information obtaining unit, the specific processing process is as follows:

[0067] Step S21: Use the contour detection function in OpenCV to identify the outer contour of the workpiece in the top-down depth image of the workpiece, and obtain the outer contour line of the workpiece and the coordinate information of each point on the outer contour line in the image.

[0068] Step S22: Read the pixel values of each point on the outer contour line of the workpiece, and record the contour point with the smallest pixel value as x, and the corresponding pixel value as Px. Px is the distance between this point and the position point S of the depth camera.

[0069] Step S23: Connect the smallest contour point x to the center point C' of the bottom surface contour of the workpiece to obtain a line segment xC'. At the same time, the line segment xC' intersects the top surface contour line of the workpiece at point J. Among them, the projection points of the center point C' of the bottom surface contour of the workpiece and the center point C of the top surface contour of the workpiece on the top-down depth image and the top-down image of the workpiece coincide.

[0070] Step S24: Calculate the length of the line segment JC in the corresponding top-down image of the workpiece according to the positions of point J and the center point C, and then transform the length of the line segment JC through the transformation relationship between the pre-calibrated pixel coordinate system and the world coordinate system to obtain the length of the line segment JC in the world coordinate system, denoted as Ljc.

[0071] Step S25: Since the workpiece is a regular hexahedron structure, that is, the smallest contour point x is on the vertical plane, take Ljc as the actual length of the line segment xC' between the smallest contour point x and the center point C' of the bottom surface contour of the workpiece in the world coordinate system.

[0072] Step S26: In the world coordinate system, a corresponding right triangle is formed by the depth camera position point S, the minimum contour point x, and the center point C' of the bottom contour of the workpiece. Among them, the line segment SC' is perpendicular to the line segment xC'. The length Px of the line segment Sx is known, and the length Ljc of the line segment xC' is known. The length of the line segment SC' is calculated using trigonometric functions, that is, the vertical height value between the depth camera position point S and the center point C' of the bottom contour of the workpiece is obtained, denoted as H1;

[0073] Step S27: Obtain the known vertical distance value H0 between the depth camera position point S and the lowest position point of the pneumatic gripper, and subtract H0 from H1 to obtain H2, that is, obtain the gripper descent distance information.

[0074] In the present invention, through the contour detection technology combined with the principle of the depth camera, the workpiece distance information can be calculated skillfully and accurately. After obtaining the workpiece distance information, the workpiece grasping robot can be controlled to drive the pneumatic gripper to descend the corresponding distance (H2) to ensure the stable progress of the subsequent grasping work.

[0075] In this embodiment, the workpiece deflection angle sensing module is used to identify the deflection angle of the workpiece in the horizontal plane after the pneumatic gripper descends in place, and obtain the workpiece deflection angle information.

[0076] More specifically, the workpiece deflection angle sensing module includes a pre-clamping unit, a pressure detection unit, and a workpiece deflection angle acquisition unit; the pre-clamping unit is used to perform a pre-clamping operation on the workpiece after the pneumatic gripper descends in place; the pressure detection unit is used to detect and obtain the pressure value on the corresponding clamping plate after any clamping plate of the pneumatic gripper contacts and presses the workpiece surface using a pressure sensor; the workpiece deflection angle acquisition unit is used to obtain the workpiece deflection angle information according to the pressure value on the corresponding clamping plate, and adjust the angle of the pneumatic gripper according to the workpiece deflection angle information (the angle adjustment is achieved by rotating the R axis of the pneumatic gripper).

[0077] It should be noted that during the pre-clamping operation, the workpiece will not move on the lower placement platform.

[0078] More specifically, in the workpiece deflection angle acquisition unit, the specific processing process is as follows:

[0079] Step S31: Use the pressure sensor 3 to obtain the pressure value of the corresponding area on the clamping plate that contacts and presses the workpiece surface. On the two clamping plates of the pneumatic gripper, two areas A1 and A2 are set on the first clamping plate 1. The areas A1 and A2 are respectively located at the two side edges of the first clamping plate 1. A plurality of pressure sensors 3 are vertically arranged in each area. Two areas B1 and B2 are set on the second clamping plate 2. The areas B1 and B2 are respectively located at the two side edges of the second clamping plate 2. A plurality of pressure sensors 3 are vertically arranged in each area. SeeFigure 3 ;

[0080] Step S32: When the first clamping plate 1 contacts and presses the workpiece surface and the second clamping plate 2 does not contact and press the workpiece surface, first calculate the arithmetic pressure means A1avg and A2avg of the pressure sensors in the two regions A1 and A2 respectively, then compare the magnitudes of the arithmetic pressure means A1avg and A2avg, and calculate the difference between the arithmetic pressure means A1avg and A2avg, denoted as the pressure difference Ac; when the second clamping plate 2 contacts and presses the workpiece surface and the first clamping plate 1 does not contact and press the workpiece surface, first calculate the arithmetic pressure means B1avg and B2avg of the pressure sensors in the two regions B1 and B2 respectively, then compare the magnitudes of the arithmetic pressure means B1avg and B2avg, and calculate the difference between the arithmetic pressure means B1avg and B2avg, denoted as the pressure difference Bc;

[0081] Step S33: Look up and obtain the corresponding workpiece deflection angle θ in the preset pressure difference - deflection angle database according to the pressure difference Ac or the pressure difference Bc, that is, obtain the workpiece deflection angle information;

[0082] Step S34: When A1avg is greater than A2avg or B1avg is greater than B2avg, drive the pneumatic gripper to rotate counterclockwise by an angle of θ; when A1avg is less than A2avg or B1avg is less than B2avg, drive the pneumatic gripper to rotate clockwise by an angle of θ.

[0083] In this embodiment, in the step S33, the pressure difference - deflection angle database stores the corresponding relationship between the pressure difference and the deflection angle.

[0084] In the present invention, by arranging a pressure sensor group in the corresponding regions of the two clamping plates of the pneumatic gripper and combining with the pre - clamping operation, the workpiece deflection angle information can be accurately identified, providing an accurate basis for the subsequent angle adjustment of the pneumatic gripper, and avoiding unnecessary wear of the gripper caused by excessive workpiece deflection angle during subsequent grasping.

[0085] In this embodiment, the solder joint image acquisition module is used to acquire a solder joint image (RGB image) after the workpiece is spot - welded and obtain the area feature information of each solder joint according to the solder joint image.

[0086] More specifically, the specific processing process of the solder joint image acquisition module is as follows:

[0087] Step S41: Use an industrial camera to photograph the welding area to obtain a solder joint image, and the solder joint image includes a plurality of independent solder joints;

[0088] Step S42: Use the trained second object detection model to identify each solder joint in the solder joint image, and obtain each solder joint detection box and the coordinates of its upper left corner point and lower right corner point in the solder joint image;

[0089] Step S43: Use the contour detection function in OpenCV to perform external contour detection on the solder joints in each solder joint detection box, and obtain the coordinates of each point on the contour line of each solder joint and the external contour line in the image;

[0090] Step S44: Read the pixel area of the internal region of each solder joint contour line as the area feature information of each solder joint, denoted as Ri, where i represents the i-th solder joint.

[0091] In the present invention, the pixel area of the internal region of the solder joint contour line is used as the area feature information of each solder joint, thereby realizing the accurate scoring work of the welding quality of the current workpiece.

[0092] In this embodiment, the welding quality evaluation module is used to score the welding quality of the current workpiece according to the area feature information of each solder joint.

[0093] More specifically, the specific processing process of the welding quality evaluation module is as follows:

[0094] Step S51: Calculate the difference between the area feature information Ri of each solder joint and the standard solder joint area feature value R0, denoted as Rci;

[0095] Step S52: Calculate the average value of the difference Rci, denoted as the average value Rcavg;

[0096] Step S53: Obtain the corresponding welding quality score T in the preset scoring database according to the average value Rcavg.

[0097] In this embodiment, the scoring database stores the corresponding relationship between the average value and the welding quality score T.

[0098] In summary, the collaborative robot multi-modal perception system based on digital control in the above embodiments can achieve the perception of various modal information such as RGB images, depth images, and pressure sensors through the workpiece size perception module, gripper descent distance perception module, workpiece deflection angle perception module, and solder joint image acquisition module set. By using the maximum distance between any two points on the outer contour line of the workpiece top surface as the size characteristic parameter of the workpiece, and then obtaining the corresponding pneumatic gripper model, it can effectively ensure that the pneumatic gripper at the end of the workpiece grasping robot can complete the grasping task of the current work, greatly improving the accuracy and efficiency of the pneumatic gripper replacement work. Through the contour detection technology combined with the principle of the depth camera, the workpiece distance information can be calculated skillfully and accurately. After obtaining the workpiece distance information, the workpiece grasping robot can be controlled to drive the pneumatic gripper to descend the corresponding distance to ensure the stable progress of the subsequent grasping work. By setting a pressure sensor group in the corresponding areas of the two clamping plates of the pneumatic gripper and combining with the pre-clamping operation, the workpiece deflection angle information can be accurately identified, providing an accurate basis for the subsequent angle adjustment of the pneumatic gripper and avoiding unnecessary wear on the gripper caused by excessive workpiece deflection angle during subsequent grasping. Using the pixel area of the internal area of the solder joint contour line as the area characteristic information of each solder joint, the accurate scoring of the welding quality of the current workpiece can be realized.

[0099] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0100] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0101] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A collaborative robot multimodal perception system based on digital control, characterized in that: It includes workpiece size sensing module, gripper descending distance sensing module, workpiece deflection angle sensing module, welding spot image acquisition module and welding quality assessment module; The workpiece size sensing module is used to identify the size of the workpiece under the workpiece grasping robot and obtain the workpiece size information; The gripper descending distance sensing module is used to identify the distance the gripper needs to descend during grasping and obtain gripper descending distance information; The workpiece deflection angle sensing module is used to identify the deflection angle of the workpiece on the horizontal plane after the pneumatic gripper is lowered into place, and obtain the workpiece deflection angle information; The welding spot image acquisition module is used to acquire welding spot images after the workpiece spot welding is completed, and acquire area feature information of each welding spot according to the welding spot images; The welding quality assessment module is used to score the welding quality of the current workpiece according to the area characteristic information of each welding point; The workpiece deflection angle sensing module includes a pre-clamping unit, a pressure detection unit and a workpiece deflection angle acquisition unit; the pre-clamping unit is used to perform a pre-clamping operation on the workpiece after the pneumatic clamp is lowered into place; the pressure detection unit is used to detect and obtain the pressure value on the corresponding clamping plate by using a pressure sensor after any clamping plate of the pneumatic clamp contacts and squeezes the surface of the workpiece; the workpiece deflection angle acquisition unit is used to obtain the workpiece deflection angle information according to the pressure value on the corresponding clamping plate, and adjust the angle of the pneumatic clamp according to the workpiece deflection angle information; In the workpiece deflection angle acquisition unit, the specific processing process is as follows: Step S31: using a pressure sensor to obtain the pressure value of the corresponding area on the clamping plate that contacts and squeezes the surface of the workpiece, wherein on the two clamping plates of the pneumatic clamping jaws, the first clamping plate is provided with two areas A1 and A2, the two areas A1 and A2 are respectively located on the two side edges of the first clamping plate, and a plurality of pressure sensors are vertically arranged in each area, and the second clamping plate is provided with two areas B1 and B2, the two areas B1 and B2 are respectively located on the two side edges of the second clamping plate, and a plurality of pressure sensors are vertically arranged in each area; Step S32: when the first clamping plate is in contact with and pressed against the surface of the workpiece, and the second clamping plate is not in contact with and pressed against the surface of the workpiece, firstly, the arithmetic pressure averages A1avg and A2avg of the pressure sensors in the two regions A1 and A2 are calculated respectively, then the arithmetic pressure averages A1avg and A2avg are compared, and the difference between the arithmetic pressure averages A1avg and A2avg is calculated, which is recorded as the pressure difference Ac; when the second clamping plate is in contact with and pressed against the surface of the workpiece, and the second clamping plate is not in contact with and pressed against the surface of the workpiece, firstly, the arithmetic pressure averages B1avg and B2avg of the pressure sensors in the two regions B1 and B2 are calculated respectively, then the arithmetic pressure averages B1avg and B2avg are compared, and the difference between the arithmetic pressure averages B1avg and B2avg is calculated, which is recorded as the pressure difference Bc; Step S33: searching and obtaining the corresponding workpiece deflection angle θ in a preset pressure difference value-deflection angle database according to the pressure difference value Ac or the pressure difference value Bc, that is, obtaining the workpiece deflection angle information, wherein the pressure difference value-deflection angle database stores the corresponding relationship between the pressure difference value and the deflection angle; Step S34: When A1avg is greater than A2avg or B1avg is greater than B2avg, the pneumatic clamp is driven to rotate counterclockwise at an angle of θ; when A1avg is less than A2avg or B1avg is less than B2avg, the pneumatic clamp is driven to rotate clockwise at an angle of θ.

2. The collaborative robot multimodal perception system based on digital control according to claim 1, characterized in that: The workpiece size sensing module includes a workpiece image capturing unit, a workpiece image preprocessing unit and a workpiece image recognition unit; the workpiece image capturing unit is used to use a camera to vertically shoot the workpiece downward to obtain an overhead image of the workpiece; the workpiece image preprocessing unit is used to perform noise reduction processing on the overhead image of the workpiece; The workpiece image recognition unit is used to recognize the top surface of the workpiece in the overhead image of the workpiece using the trained first target detection model, thereby obtaining workpiece size information, and recommending the model of the pneumatic gripper to the operator based on the workpiece size information.

3. The collaborative robot multimodal perception system based on digital control according to claim 2 is characterized in that: In the workpiece image recognition unit, the specific processing process is as follows: Step S11: using the trained target detection model to identify the top surface of the workpiece in the overhead image of the workpiece, and obtaining a detection frame of the top surface of the workpiece and the coordinates of its upper left corner and lower right corner in the image; Step S12: cutting out the workpiece top surface detection frame area from the workpiece overhead image to obtain a workpiece top surface detection frame image; Step S13: Use the contour detection function in OpenCV to detect the outer contour of the top surface of the workpiece, and obtain the outer contour line of the top surface of the workpiece and the coordinate information of each point on the outer contour line in the image; Step S14: Calculate the distance between any two points on the outer contour line, select the distance between the two contour points with the largest distance as the size characteristic parameter of the workpiece, denoted as Dmax, and obtain the size information of the workpiece; Step S15: After obtaining the workpiece size information, the corresponding pneumatic gripper model is searched and obtained in the preset size characteristic parameter-pneumatic gripper model database according to the size characteristic parameter Dmax, and the found pneumatic gripper model is recommended to the operator, who replaces the pneumatic gripper at the end of the workpiece grasping robot with the corresponding model, wherein the size characteristic parameter-pneumatic gripper model database stores the correspondence between the size characteristic parameter values ​​and the pneumatic gripper models.

4. The collaborative robot multimodal perception system based on digital control according to claim 2, characterized in that: The gripper descent distance sensing module includes a depth image capturing unit, a depth image preprocessing unit and a workpiece distance information acquisition unit; the depth image capturing unit is used to use a depth camera to vertically downwardly shoot the workpiece at the same shooting position as the camera in the workpiece image capturing unit to obtain a workpiece overhead depth image; the depth image preprocessing unit is used to perform noise reduction processing on the workpiece overhead depth image; the workpiece distance information acquisition unit is used to obtain the gripper descent distance information through contour detection processing.

5. The collaborative robot multimodal perception system based on digital control according to claim 4 is characterized in that: The workpiece overhead image and the workpiece overhead depth image both include a complete workpiece. The projection point of the optical axis of the camera in the workpiece image capturing unit on the workpiece overhead image is located at the center point C of the top surface contour of the workpiece. The projection point of the optical axis of the depth camera in the depth image capturing unit on the workpiece overhead depth image is also located at the center point C of the top surface contour of the workpiece. The resolution and size of the workpiece overhead depth image are the same as those of the workpiece overhead image.

6. The collaborative robot multimodal perception system based on digital control according to claim 5, characterized in that: In the workpiece distance information acquisition unit, the specific processing process is as follows: Step S21: using the contour detection function in OpenCV to identify the outer contour of the workpiece in the overhead depth image of the workpiece, and obtaining the outer contour line of the workpiece and the coordinate information of each point on the outer contour line in the image; Step S22: Read the pixel values ​​of each point on the outer contour line of the workpiece, record the contour point with the smallest pixel value as x, and record the corresponding pixel value as Px, where Px is the distance between the point and the depth camera position point S; Step S23: Connect the minimum contour point x and the center point C' of the bottom contour of the workpiece to obtain a line segment xC', and the line segment xC' intersects the top contour line of the workpiece at point J, wherein the projection points of the center point C' of the bottom contour of the workpiece and the center point C of the top contour of the workpiece on the overhead depth image of the workpiece and the overhead image of the workpiece coincide with each other; Step S24: Calculate the length of the line segment JC in the corresponding overhead image of the workpiece according to the positions of the point J and the center point C, and then transform the length of the line segment JC through the transformation relationship between the pre-calibrated pixel coordinate system and the world coordinate system to obtain the length of the line segment JC in the world coordinate system, which is recorded as Ljc; Step S25: Ljc is taken as the actual length of the line segment xC' between the minimum contour point x and the center point C' of the bottom contour of the workpiece in the world coordinate system; Step S26: In the world coordinate system, a corresponding right triangle is formed by the depth camera position point S, the minimum contour point x, and the center point C' of the bottom contour of the workpiece, wherein the line segment SC' is perpendicular to xC', the length Px of the line segment Sx is known, and the length Ljc of the line segment xC' is known. The length of the line segment SC' is calculated using trigonometric functions, that is, the vertical height value between the depth camera position point S and the center point C' of the bottom contour of the workpiece is obtained, which is recorded as H1; Step S27: Obtain the vertical distance value H0 between the known depth camera position point S and the lowest position point of the pneumatic gripper, and subtract H0 from H1 to obtain H2, that is, obtain the gripper descent distance information.

7. The collaborative robot multimodal perception system based on digital control according to claim 1, characterized in that: The specific processing process of the solder joint image acquisition module is as follows: Step S41: photographing the welding area using an industrial camera to obtain a welding spot image, wherein the welding spot image includes a plurality of independent welding spots; Step S42: using the trained second target detection model to identify each solder joint in the solder joint image, and obtaining each solder joint detection frame and the coordinates of its upper left corner point and lower right corner point in the solder joint image; Step S43: using the contour detection function in OpenCV to perform outer contour detection on the solder joints in each solder joint detection frame, and obtaining the contour line of each solder joint and the coordinate information of each point on the outer contour line in the image; Step S44: reading the pixel area of ​​the inner region of each solder joint contour line as the area feature information of each solder joint, denoted as Ri, where i represents the i-th solder joint.

8. The collaborative robot multimodal perception system based on digital control according to claim 7, characterized in that: The specific processing process of the welding quality assessment module is as follows: Step S51: Calculate the difference between each solder joint area characteristic information Ri and the standard solder joint area characteristic value R0, recorded as Rci; Step S52: Calculate the average value of the difference values ​​Rci, recorded as the average value Rcavg; Step S53: Obtain the corresponding welding quality score T in a preset scoring database according to the average value Rcavg, wherein the corresponding relationship between the average value Rcavg and the welding quality score T is stored in the scoring database.

Images