A servo control method and system based on robot image processing

Abstract

This invention provides a servo control method and system based on robot image processing. It analyzes two types of cylindrical workpieces to obtain three superposition scenarios. Then, it establishes a robot kinematic model, uses edge fitting to determine the edge point set of the workpieces, solves for the centroid coordinates of the workpieces in the robot's base coordinate system using inverse kinematics, performs trajectory planning, and generates motion control commands to achieve workpiece grasping. The system enables real-time wired or wireless control of the robot and is compatible with robots using communication protocols such as Modbus / EtherCAT / CAN. Furthermore, the first host computer part uses MATLAB / Simulink graphical programming, overcoming the closed nature of previous robot control systems. The robot control system in this invention can also perform digital twin functions, enabling real-time synchronous display of the virtual robot and the physical robot.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and in particular relates to a servo control method and system based on robot image processing. Background Technology

[0002] In industrial manufacturing and computer vision, image processing technology has become an indispensable key component of automated production. These technologies provide powerful tools for monitoring and controlling production processes through the acquisition, analysis, and interpretation of visual data. However, image processing still faces several challenges, particularly when dealing with images containing overlapping workpieces and edge defects. Overlapping workpieces are a common but complex problem because they make accurate centroid positioning difficult. In many manufacturing processes, workpieces may overlap, stack, or partially cover each other, resulting in unclear outlines of different workpieces in the image, making it challenging to accurately determine the centroid of each workpiece. Simultaneously, edge defects in the image also pose a challenge to image processing. Edge defects may be caused by physical changes, damage, or wear during the workpiece manufacturing process, leading to incomplete or irregular workpiece outlines. These defects can interfere with the accuracy of algorithms during edge detection and center point determination, resulting in misjudgments or inaccurate results. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a servo control method and system based on robot image processing, which can better improve the accuracy of image recognition and achieve more precise control of the robot.

[0004] To address the above technical problems, the present invention provides the following technical solution: a servo control method based on robot image processing, comprising the following steps:

[0005] S1. Use the hand-eye calibration method to determine the conversion relationship between the robot arm and the image acquisition device. For two types of cylindrical workpieces, use the image acquisition device to acquire workpiece images. Combine the depth of the worktable and the depth of the workpiece to obtain the workpiece height. Determine whether the workpieces are superimposed based on the workpiece height. If so, proceed to step S2; otherwise, determine that the workpieces are not superimposed.

[0006] S2. Calculate the number of pixels for the two workpiece colors using Blob analysis, obtain the area of ​​the workpiece corresponding to the two colors, and determine the three superposition scenarios of the two workpieces.

[0007] S3. Establish a robot kinematic model, use the edge fitting method to determine the edge point set of the workpiece, obtain the coordinates of the workpiece in the pixel coordinate system, solve the centroid coordinates of the workpiece in the robot base coordinate system through inverse kinematics, perform trajectory planning, obtain the angle values ​​of each joint of the robot, and generate motion control commands. Based on the three superposition cases of the workpiece, control the robot through motion control commands to realize the grasping of the workpiece.

[0008] Furthermore, the hand-eye calibration method in step S1 described above is specifically as follows:

[0009] S101. Fix the calibration plate to the robot end effector. The transformation matrix between the calibration plate and the robot end effector... Represented as:

[0010]

[0011] in, The transformation matrix representing the robot's base coordinate system relative to the end effector is obtained through robot forward kinematics. This represents the transformation matrix between the coordinate system of the image acquisition device and the robot's base coordinate system. This represents the transformation matrix of the calibration plate relative to the camera coordinate system, which is obtained through camera calibration;

[0012] S102. Perform the hand-eye calibration process. Keeping the position unchanged, control the robot to move to any two poses to obtain the formula:

[0013]

[0014] The above equation, after shifting terms left and right:

[0015]

[0016] make The above expression can be represented as:

[0017] AX = XB

[0018] The hand-eye calibration result is calculated using the above formula.

[0019] Further, in the aforementioned step S1, the determination of whether the workpieces are superimposed is made as follows: calculate the height of the workpiece: the depth of the worktable - the depth of the workpiece = the height of the workpiece. If the workpiece height is H+H or H+h or h+h, then the workpieces are considered to be superimposed, where H is the height of the first cylinder and h is the height of the second cylinder.

[0020] Furthermore, the aforementioned S2 includes the following sub-steps:

[0021] S1.1. Identify the color of the workpiece through color recognition, perform binarization on the workpiece color, and then obtain the number of pixels of the workpiece color through Blob analysis. Calculate the area of ​​the workpiece corresponding to the two colors. S1.2. Based on the area corresponding to the two colors, obtain three superposition situations of the workpiece: (a) the two workpieces are completely superimposed, (b) the two workpieces are superimposed with misalignment, and (c) the same type of workpieces are superimposed with misalignment.

[0022] Furthermore, in the aforementioned step S1, color recognition involves converting the RGB color space to the HSV color space, specifically as follows:

[0023] The conversion formula between RGB color space and HSV color space is as follows:

[0024]

[0025] V = max.

[0026] Furthermore, the aforementioned step S1.2 specifically includes:

[0027] 1) If the area of ​​one of the colors is approximately equal to the area of ​​the whole circle: πR 2 or πr 2 R is the cross-sectional radius of the first cylindrical workpiece, and r is the cross-sectional radius of the second cylindrical workpiece. The corresponding cylindrical workpiece is on top. If the area of ​​the workpiece of another color is zero, it is the superposition case (a).

[0028] 2) If the area of ​​the workpiece of another color is not zero, then it is the superposition case (b);

[0029] 3) If the area of ​​a certain color is greater than the area of ​​the whole circle: πR 2 or πr 2 This is the case of superposition (c).

[0030] Further, in the aforementioned step S3, determining the edge point set of the workpiece using the edge fitting method for the superimposed cases (b) and (c) includes the following sub-step S301: using Canny edge detection, the edge point set of the workpiece above is obtained as (x i ,y i Then determine the initial point set of the workpiece below as (X). i ,Y i According to the formula (X) i ,Y i )-(x i ,y i )∩(X i ,Y i ), remove the point set at the intersection of the edges to obtain the edge point set of the workpiece below;

[0031] S302. Given a circle with radius R and center (A, B), the expression for the circle is:

[0032] (xA) 2 +(yB) 2 =R 2 (1-1)

[0033] Expanding equation (1-1) above, we get the following expression:

[0034] x 2 -2Ax+A 2 +y 2 -2By+B 2 =R 2 (1-2)

[0035] S303. After Canny edge detection, the point set of the Blob block edge is (X... n ,Y n If it is the workpiece above, then (X) n ,Y n )=(X i ,Y i If it is the workpiece below, then (X) n ,Y n )=(X i ,Y i )-(x i ,y i )∩(X i ,Y i );

[0036] S304, Point Set (X) n ,Y n The distance d from the center of the circle n for:

[0037]

[0038] Point set (X) n ,Y n The difference between the square of the distance to the center and the square of the radius is:

[0039] σ n =d n 2 -R 2 =X n 2 +Y n 2 -2AX n -2BY n +A 2 +B 2 -R 2 (1-4)

[0040] Let Q(A,B,R) be σ n The sum of squares is expressed as:

[0041] Q(A,B,R)=∑σ n 2 =∑(X n 2 +Y n 2 -2AX n -2BY n +A 2 +B 2 -R 2 ) 2 (1-5)

[0042] When Q(A,B,R) reaches its minimum value, the parameters A, B, and R can be obtained. By fitting the equation of the circle, the center of the circle is (A,B), which is the coordinate in the pixel coordinate system.

[0043] Furthermore, in the aforementioned step S3, solving the centroid coordinates of the workpiece in the robot's base coordinates using inverse kinematics includes the following sub-steps:

[0044] S3.1. Based on the target's coordinates (u1, v1) in the pixel coordinate system, calculate its coordinates P in the camera coordinate system using the intrinsic parameter matrix M obtained from camera calibration. c (X c ,Y c Z c )for:

[0045]

[0046] Where M is the camera intrinsic parameter matrix, Z c The depth of the object in the camera coordinate system is obtained through the camera depth sensor;

[0047] S3.2. Based on the hand-eye calibration results, calculate the spatial coordinates of the target workpiece in the robot's base coordinate system:

[0048]

[0049] in, The transformation matrix is ​​the ratio of the camera coordinate system to the robot base coordinate system; (X) w Y w Z w ( ) represents the spatial coordinates of the centroid of the target workpiece in the robot's base coordinate system.

[0050] S3.3. Based on the rotation direction of the target workpiece on the conveyor belt, this rotation direction is defined as the positive direction of the X-axis. The time from image acquisition to the robot's first grasp is t1, and the conveyor belt speed is v. During this time, the centroid coordinates of the target workpiece change to (X... w +t1v, Y w Z w ); The time from the robot's first grasp to the second grasp is t2, and the centroid coordinates of the target object become (X... w +(t1+t2)v,Y w Z w ).

[0051] Furthermore, the aforementioned servo control method based on robot image processing also includes controlling the robot through a digital twin method, specifically:

[0052] S401. Robot 3D Model Drawing: In CAD software, a 3D model is drawn based on the actual shape and structure of the robot, including drawing the robot's various components, parts, joints, and transmission system.

[0053] S402, URDF file generation: Convert the robot's 3D model into a URDF file, defining the robot's connection relationships, joint types, joint constraints, mass, and inertial parameters.

[0054] S403, Simscape Physical Model Generation: Import the robot's 3D model and URDF file into MATLAB / Simulink. Use the Simscape toolbox in MATLAB to create a physical model of the virtual robot. In Simulink, use the Simscape Multibody library to build the robot's kinematics and dynamics model, and define joint constraints, transmission systems, and sensors.

[0055] S404. Simulation Environment Configuration: Set simulation parameters and environmental conditions in Simulink;

[0056] S405, Control Algorithm Development: Using the Simscape physical model and simulation environment, develop robot control algorithms in Simulink. Implement autonomous robot movement using path planning and trajectory tracking algorithms, and optimize robot performance and behavior by adjusting control parameters and algorithm logic.

[0057] S406, Digital Twin: Run a digital twin platform in Simulink to observe and analyze the robot's motion, mechanical response, and sensor output. View the robot's real-time animation, plot curves of key indicators, and perform data collection and analysis through a graphical interface.

[0058] Another aspect of the present invention proposes a servo control system based on robot image processing, characterized in that it includes: a first host computer, an image acquisition device, a data transmission module, a robot body, a slave computer, and a digital twin platform;

[0059] The image acquisition device is connected to the first host computer and outputs the image to the host computer.

[0060] The first host computer is equipped with programming software and a real-time kernel. It uses hand-eye calibration to determine the conversion relationship between the robot arm and the image acquisition device. For two types of cylindrical workpieces, the image acquisition device acquires workpiece images. Combining the depth of the worktable and the depth of the workpiece, the workpiece height is obtained. Based on the workpiece height, it determines whether the workpieces overlap. If so, Blob analysis is used to calculate the number of pixels for each of the two workpiece colors, obtaining the area of ​​the workpiece corresponding to each color, and determining three overlapping scenarios for the two workpieces. Otherwise, it is determined that the workpieces do not overlap. Then, a robot kinematic model is established, and edge fitting methods are used to determine the edge point set of the workpiece, obtaining the workpiece coordinates in the pixel coordinate system. Inverse kinematics is used to solve for the centroid coordinates of the workpiece in the robot's base coordinate system, performing trajectory planning to obtain the angle values ​​of each joint of the robot, and generating motion control commands.

[0061] The input terminal of the data transmission module is connected to the output terminal of the first host computer to receive motion control commands output by the first host computer. The output terminal of the data transmission module is connected to the input terminal of the slave computer to transmit the motion control commands to the slave computer. The microcontroller of the slave computer performs communication protocol conversion and outputs the commands to the connected robot. Based on the three superimposed conditions of the workpiece, the robot is controlled by the motion control commands to grasp the workpiece. The robot synchronously feeds back the real-time running data to the host computer for real-time data display.

[0062] The first host computer is connected to the second host computer. A digital twin platform is built in the second host computer to feed back the real-time running status of the robot to the virtual robot, so as to realize the synchronous display of the virtual model and the physical robot.

[0063] The data transmission module includes: an Ethernet switch, a CAN communication module, and a wireless transceiver module.

[0064] The microcontroller is configured to perform the following actions:

[0065] S501: Receive data from the data transmission module, parse it, and extract data fields and related information according to the specifications of the corresponding protocol;

[0066] S502. Based on the communication protocol used by the target robot, repackage the parsed data fields into the format of the target protocol; including data type conversion, format adjustment, and protocol-specific data packet assembly.

[0067] S503. Use the target protocol to transmit the encapsulated data to the target robot via wired or wireless transmission.

[0068] Compared to existing technologies, the beneficial technical effects of the present invention using the above technical solution are as follows: The present invention provides a servo control method and system based on robot image processing, realizing functions such as image processing, motion control, wired / wireless transmission, communication protocol conversion, and digital twin. Through the recognition and positioning technology of superimposed objects, the present invention can accurately distinguish and locate superimposed workpieces, avoiding the influence of fuzzy centroid positions and workpiece edge defects, ensuring that the position and boundary information of each workpiece are accurately captured. The automated processing of superimposed objects enables the robot system to automatically process superimposed workpieces, achieving precise grasping and positioning of each workpiece, thereby improving the automation level and production efficiency of the production line.

[0069] This invention acquires image data from a binocular camera via a first host computer, transmits the image data to the host computer for image processing, and obtains the center point coordinates of the target object. Based on the obtained workpiece center point coordinates, a smooth and safe robot motion trajectory is generated through a path planning algorithm. The first host computer transmits control commands to a lower-level computer via wired / wireless transmission. The lower-level computer receives and parses the control commands, performs communication protocol conversion as needed, and transmits the commands to the robot with the corresponding communication protocol. The robot precisely controls its body according to the received control commands to complete the workpiece grasping action. The lower-level computer performs corresponding communication protocol conversion on the feedback data and uploads the data to the first host computer. The first host computer transmits real-time data to a digital twin platform, which is synchronized with the robot model in real time, displaying the robot's operating status and workpiece grasping process in a visual manner. This invention uses a modular approach to build the robot's motion control model, realizing functions such as image processing, trajectory planning, and digital twin, which can shorten the development time of the robot system. Simulink provides a powerful development environment, enabling beginners to build and analyze robot control systems through graphical programming. Its visualization features, rich libraries and modules, and integration capabilities with hardware systems enable beginners to learn and practice robot control more efficiently. In this embodiment, a common PC is used as the first host computer to implement the robot control system, eliminating the need for a dedicated industrial PC, simplifying maintenance, and reducing development costs. The system not only features operation monitoring, data storage, graphical display, and digital twin functions, but also enables protocol conversion. Through protocol conversion, the first host computer can communicate seamlessly with the robot controller, achieving interoperability between different communication protocols. This means that the first host computer can communicate with various types of robot controllers, whether based on common protocols such as CAN, Ethernet, or Modbus, all can connect and interact with the first host computer. This protocol conversion function greatly improves the system's flexibility and compatibility, allowing developers to more easily integrate different types of robot controllers, reducing dependence on specific hardware, and further lowering system development costs. The first host computer's graphical display unit collects data such as the robot's position, velocity, and acceleration, displaying motion curves in real time, accelerating development speed. Furthermore, the second host computer implements digital twin functionality, connecting the physical system and the digital model, simulating and optimizing the robot system, and improving performance. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the system architecture provided in an embodiment of the present invention;

[0071] Figure 2 This is a flowchart of hand-eye calibration provided in an embodiment of the present invention;

[0072] Figure 3A color recognition flowchart provided for embodiments of the present invention;

[0073] Figure 4 A flowchart of a digital twin provided for embodiments of the present invention;

[0074] Figure 5 A flowchart illustrating the communication protocol conversion provided in an embodiment of the present invention. Detailed Implementation

[0075] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0076] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0077] This embodiment uses a six-degree-of-freedom serial robot, but it is not limited to this; other multi-degree-of-freedom or parallel robots can also be controlled, and multiple robots can be controlled simultaneously. This includes other robots that conform to communication protocols such as Modbus / EtherCAT / CAN. First, the robot state is initialized: the robot's joints are initialized, including setting the joints to their initial positions, clearing any erroneous states, calibrating the sensors, and setting the maximum speed and maximum acceleration / deceleration. This ensures that the robot is in a reliable and stable state before motion control.

[0078] Scenario Description: Imagine an automated conveyor belt carrying two types of target objects: a large yellow cylindrical workpiece and a small red cylindrical workpiece. The cylindrical workpieces need to be placed on the conveyor belt. If there is no overlap, the robot sorting will be very smooth. However, if the two types of workpieces overlap, it will greatly affect the sorting process. This patent provides a robot control system based on edge fitting, aiming to solve the problem of workpiece overlap preventing sorting.

[0079] Target workpiece:

[0080] 1. Large cylinder: Height H, radius R, area πR 2 ;

[0081] 2. Small cylinder: height h, radius r, area πr 2 .

[0082] Based on the above application scenarios, the present invention provides a servo control method based on robot image processing, comprising the following steps:

[0083] S1. Determine the conversion relationship between the robot arm and the image acquisition device using the hand-eye calibration method. For two types of cylindrical workpieces, use the image acquisition device to acquire workpiece images. Combine the depth of the worktable and the depth of the workpiece to obtain the workpiece height. Determine whether the workpieces are stacked based on the workpiece height. If so, proceed to step S2; otherwise, determine that the workpieces are not stacked. Specifically:

[0084] To determine if workpieces are stacked, calculate the workpiece height as follows: Worktable depth - workpiece depth = workpiece height. If the workpiece height is H+H, H+h, or h+h, then the workpieces are considered to be stacked, where H is the height of the first cylinder and h is the height of the second cylinder.

[0085] refer to Figure 2 In step S1, the hand-eye calibration method uses an eye-to-hand approach, specifically as follows:

[0086] In an eye-to-hand system, the calibration board is fixed to the robot's end effector, and their relative positional relationship remains constant. The transformation matrix between the calibration board and the robot's end effector... Represented as:

[0087]

[0088] in, The transformation matrix representing the robot's base coordinate system relative to the end effector can be obtained through robot forward kinematics. This represents the transformation matrix between the camera coordinate system and the robot's base coordinate system. The transformation matrix representing the calibration plate relative to the camera coordinate system can be obtained through camera calibration.

[0089] During the hand-eye alignment process, due to Keeping the position unchanged, control the robot to move to any two poses to obtain the formula:

[0090]

[0091] The above equation can be obtained by shifting terms left and right:

[0092]

[0093] make The above formula can then be expressed as:

[0094] AX = XB

[0095] The hand-eye alignment result can be calculated using the above formula.

[0096] refer to Figure 3 Target object color recognition involves converting the RGB color space to the HSV color space. Images acquired by image acquisition devices are typically RGB color images. However, the RGB color space is sensitive to lighting conditions; when lighting conditions change, the RGB values ​​change accordingly, making color recognition difficult. Since the HSV color space is more robust to changes in lighting and shadows than the RGB color space, it is more suitable for color recognition and analysis. Therefore, when performing color recognition, the RGB color space should first be converted to the HSV color space. The conversion formula between RGB and HSV color spaces is shown below.

[0097]

[0098]

[0099] V = max.

[0100] S2. Calculate the number of pixels for each of the two workpiece colors using Blob analysis, obtain the area of ​​the workpiece corresponding to each color, and determine the three possible superposition scenarios for the two workpieces, as shown in Table 1.

[0101] S1.1. Through color recognition, the color of the workpiece is identified, the workpiece color is binarized, and then the number of pixels of the workpiece color is obtained through Blob analysis. The area of ​​the workpiece corresponding to the two colors is calculated.

[0102] S1.2. Based on the areas corresponding to the two colors, obtain three superposition situations of the workpieces: (a) the two workpieces are completely superimposed, (b) the two workpieces are superimposed with misalignment, and (c) the same type of workpieces are superimposed with misalignment.

[0103]

[0104] S3. Establish a robot kinematic model, use the edge fitting method to determine the edge point set of the workpiece, obtain the coordinates of the workpiece in the pixel coordinate system, solve the centroid coordinates of the workpiece in the robot base coordinate system through inverse kinematics, perform trajectory planning, obtain the angle values ​​of each joint of the robot, and generate motion control commands. Based on the three superposition cases of the workpiece, control the robot through motion control commands to realize the grasping of the workpiece.

[0105] During control, operations such as controller parameter tuning, system calibration, and real-time data acquisition are performed to ensure the accuracy and stability of the controller.

[0106] In step S3, determining the edge point set of the workpiece using the edge fitting method for the superposition cases (b) and (c) includes the following sub-steps:

[0107] In this embodiment, the target object selected is a cylindrical object. Therefore, after using Canny edge detection, the extracted edge points are fitted using the least squares method to obtain more accurate circular contour information.

[0108] S301. Using Canny edge detection, the edge point set of the workpiece above is obtained as (x i ,y i Then determine the initial point set of the workpiece below as (X). i ,Y i According to the formula (X) i ,Y i )-(x i ,y i )∩(X i ,Y i ), remove the point set at the intersection of the edges to obtain the edge point set of the workpiece below;

[0109] S302. Given a circle with radius R and center (A, B), the expression for the circle is:

[0110] (xA) 2 +(yB) 2 =R 2 (1-1)

[0111] Expanding equation (1-1) above, we get the following expression:

[0112] x 2 -2Ax+A 2 +y 2 -2By+B 2 =R 2 (1-2)

[0113] S303. After Canny edge detection, the point set of the Blob block edge is (X... n ,Y n If it is the workpiece above, then (X) n ,Y n )=(X i ,Y i If it is the workpiece below, then (X) n ,Y n )=(X i ,Y i )-(x i ,y i )∩(X i ,Y i );

[0114] S304, Point Set (X) n ,Y n The distance d from the center of the circlen for:

[0115]

[0116] Point set (X) n ,Y n The difference between the square of the distance to the center and the square of the radius is:

[0117] σ n =d n 2 -R 2 =X n 2 +Y n 2 -2AX n -2BY n +A 2 +B 2 -R 2 (1-4)

[0118] Let Q(A,B,R) be σ n The sum of squares is expressed as:

[0119] Q(A,B,R)=∑σ n 2 =∑(X n 2 +Y n 2 -2AX n -2BY n +A 2 +B 2 -R 2 ) 2 (1-5)

[0120] When Q(A,B,R) reaches its minimum value, the parameters A, B, and R can be obtained. By fitting the equation of the circle, the center of the circle is (A,B), which is the coordinate in the pixel coordinate system.

[0121] Solving for the centroid coordinates of the workpiece in the robot's base coordinates using inverse kinematics includes the following sub-steps:

[0122] S3.1. Based on the target's coordinates (u1, v1) in the pixel coordinate system, calculate its coordinates P in the camera coordinate system using the intrinsic parameter matrix M obtained from camera calibration. c (X c ,Y c Z c )for:

[0123]

[0124] Where M is the camera intrinsic parameter matrix, Zc The depth of the object in the camera coordinate system is obtained through the camera depth sensor;

[0125] S3.2. Based on the hand-eye calibration results, calculate the spatial coordinates of the target workpiece in the robot's base coordinate system:

[0126]

[0127] in, The transformation matrix is ​​the ratio of the camera coordinate system to the robot base coordinate system; (X) w Y w Z w ( ) represents the spatial coordinates of the centroid of the target workpiece in the robot's base coordinate system.

[0128] S3.3. Based on the rotation direction of the target workpiece on the conveyor belt, this rotation direction is defined as the positive direction of the X-axis. The time from image acquisition to the robot's first grasp is t1, and the conveyor belt speed is v. During this time, the centroid coordinates of the target workpiece change to (X... w +t1v, Y w Z w ); The time from the robot's first grasp to the second grasp is t2, and the centroid coordinates of the target object become (X... w +(t1+t2)v,Y w Z w ).

[0129] refer to Figure 4 This invention achieves robot control through digital twin technology, specifically as follows:

[0130] S401. Robot 3D Model Drawing: In CAD software, a 3D model is drawn based on the actual shape and structure of the robot, including drawing the robot's various components, parts, joints, and transmission system to ensure that the model accurately reflects the robot's geometry and dimensions.

[0131] S402, URDF File Generation: URDF is an XML format file used to describe a robot's geometry, kinematics, dynamics, and sensor information. By using SolidWorks plugins (such as SW2URDF) or other tools, a robot's 3D model can be converted into a URDF file, defining the robot's connectivity, joint types, joint constraints, mass, and inertial parameters.

[0132] S403, Simscape Physical Model Generation: Import the robot's 3D model and URDF file into MATLAB / Simulink. Use the Simscape toolbox in MATLAB to create a physical model of the virtual robot. In Simulink, use the Simscape Multibody library to build the robot's kinematics and dynamics model, and define joint constraints, transmission systems, and sensors.

[0133] S404. Simulation Environment Configuration: Set simulation parameters and environmental conditions in Simulink; add various external forces, sensor inputs, control algorithms, etc. Furthermore, you can configure virtual environments, such as ground friction and gravity, to more realistically simulate robot behavior.

[0134] S405, Control Algorithm Development: Using the Simscape physical model and simulation environment, develop robot control algorithms in Simulink. Implement autonomous robot movement using path planning and trajectory tracking algorithms, and optimize robot performance and behavior by adjusting control parameters and algorithm logic.

[0135] S406, Digital Twin: Run a digital twin platform in Simulink to observe and analyze the robot's motion, mechanical response, and sensor output. View real-time animations of the robot through a graphical interface, plot curves of key indicators, and perform data acquisition and analysis. Based on the simulation results, the robot's performance can be evaluated, and system optimization and improvement can be performed.

[0136] like Figure 1 As shown, the present invention also provides a servo control system based on robot image processing, including: a first host computer, an image acquisition device, a data transmission module, a robot body, a slave computer, and a digital twin platform;

[0137] The image acquisition device is connected to the first host computer and outputs the image to the host computer.

[0138] The first host computer is equipped with programming software and a real-time kernel. It uses hand-eye calibration to determine the conversion relationship between the robot arm and the image acquisition device. For two types of cylindrical workpieces, the image acquisition device acquires workpiece images. Combining the depth of the worktable and the depth of the workpiece, the workpiece height is obtained. Based on the workpiece height, it determines whether the workpieces overlap. If so, Blob analysis is used to calculate the number of pixels for each of the two workpiece colors, obtaining the area of ​​the workpiece corresponding to each color, and determining three overlapping scenarios for the two workpieces. Otherwise, it is determined that the workpieces do not overlap. Then, a robot kinematic model is established, and edge fitting methods are used to determine the edge point set of the workpiece, obtaining the workpiece coordinates in the pixel coordinate system. Inverse kinematics is used to solve for the centroid coordinates of the workpiece in the robot's base coordinate system, performing trajectory planning to obtain the angle values ​​of each joint of the robot, and generating motion control commands.

[0139] The input terminal of the data transmission module is connected to the output terminal of the first host computer to receive motion control commands output by the first host computer. The output terminal of the data transmission module is connected to the input terminal of the slave computer to transmit the motion control commands to the slave computer. The microcontroller of the slave computer performs communication protocol conversion and outputs the commands to the connected robot. Based on the three superimposed conditions of the workpiece, the robot is controlled by the motion control commands to grasp the workpiece. The robot synchronously feeds back the real-time running data to the host computer for real-time data display.

[0140] The first host computer is connected to the second host computer. A digital twin platform is built in the second host computer to feed back the real-time operating status of the robot to the virtual robot, so as to realize the synchronous display of the virtual model and the physical robot.

[0141] The lower-level microcontroller uses an STM32 as the main control chip, primarily responsible for converting between various communication protocols. The robot's joint motors are integrated drive and control motors; the robot can be a six-degree-of-freedom serial robot, a parallel robot, or similar configuration. A digital twin platform is built on the second upper-level computer, directly connecting the motion control algorithm model to the robot's 3D model. This allows for robot control in a virtual environment, and the motion curves of the 3D model can be observed using a software oscilloscope. The real-time operating status of the virtual robot can also be visualized, enabling pre-control and algorithm verification of the robot within the digital twin platform environment.

[0142] The data feedback process involves the integrated drive and control joint motor uploading joint operation data to the lower-level machine. The lower-level machine then converts the data into corresponding data through its communication protocol and uploads it to the first upper-level machine via wired / wireless means. The first upper-level machine monitors this data in real time and can also transmit the data to the second upper-level machine for real-time control of the virtual robot, thus completing the function of a digital twin.

[0143] The data transmission module of this invention includes: an Ethernet switch, a CAN communication module, and a wireless transceiver module. The transmission from the host computer to the slave computer can be either wired or wireless.

[0144] Wired transmission methods include:

[0145] a. First method: Using a CAN card to connect the host computer and the slave computer. The CAN communication module achieves communication through differential signal lines, where CAN_H (CAN High) and CAN_L (CAN Low) form a set of differential signal lines. The voltage difference between the differential signal lines represents the logic 0 and logic 1 signals.

[0146] b. The second method: Data transmission is achieved by connecting an Ethernet switch using a network cable. This part is implemented using a W5500 router, which connects the network module to the router, allowing various network devices to communicate with the network module within the same subnet.

[0147] Wireless transmission method: Communication with the lower-level device is achieved using a wireless network card. This part is implemented using an E8266 module, and the main steps are as follows:

[0148] a. The WiFi module connects to the target hotspot, and the main mode of the WiFi module is STA.

[0149] b. Establish information communication between the network transmission module in the first host computer and the WiFi module in the lower computer.

[0150] c. When the WiFi module receives information sent by the first host computer, it communicates with the microcontroller via the serial port and stores the received data in the microcontroller.

[0151] The communication protocol conversion process includes:

[0152] Step 1, Data Reception: Receive data from the source device, including CAN bus, Modbus devices, EtherCAT devices, or TCP / IP network packets. This data is transmitted in different formats and protocols, including but not limited to CAN to Modbus communication protocol conversion, CAN to EtherCAT communication protocol conversion, TCP / IP to CAN communication protocol conversion, TCP / IP to Modbus communication protocol conversion, and TCP / IP to EtherCAT communication protocol conversion.

[0153] Step 2, Data Parsing: Parse the received data and extract the required data fields and related information according to the specifications of the corresponding protocol.

[0154] Step 3, Data Conversion: Based on the communication protocol used by the target device, the parsed data fields are repackaged into the target protocol's format. This process involves data type conversion, format adjustment, and protocol-specific data packet assembly.

[0155] Step 4: Data Transmission: Transmit the encapsulated data to the target device using the target protocol. The converted data is sent to the target device via wired or wireless means, ensuring reliable data transmission between devices.

[0156] refer to Figure 5 In the digital twin process, the following are included:

[0157] Step 1: The first host computer establishes a connection with the robot's digital twin platform using the TCP / IP communication protocol. This connection method ensures real-time and reliable data transmission and control.

[0158] Step 2: The first host computer sends control commands via wired mode. Simulink runs in a real-time kernel environment, and data can be transmitted to the digital twin platform in real time through an Ethernet switch, enabling precise control of the virtual robot.

[0159] Step 3: Simulink uses an oscilloscope to observe the feedback data of the physical robot's operating status in real time, monitoring the control situation during program execution. Through the real-time feedback from the oscilloscope, users can promptly understand changes in control data and the robot's response.

[0160] Step 4: The digital twin platform receives and executes the feedback data, controlling the operation of the virtual robot. By feeding back the robot's operating status to the digital twin platform, users can obtain real-time information about the virtual robot, such as its position, speed, and posture, and can map the physical robot in virtual space for further analysis and adjustments.

[0161] This control system has two operating modes. Mode 1: It first controls the virtual simulation and then the physical robot, automatically switching between the two. If the virtual robot simulation is correct, the physical robot is then controlled, ensuring the robot's safety. Mode 2: The virtual robot and the physical robot are displayed synchronously, achieving the effect of a digital twin. By controlling the physical robot, real-time feedback of operating data, including joint angles, joint speeds, and joint accelerations, is achieved, enabling real-time monitoring of the robot's operating status.

[0162] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A servo control method based on robot image processing, characterized in that, Includes the following steps: S1. Use the hand-eye calibration method to determine the conversion relationship between the robot arm and the image acquisition device. For two types of cylindrical workpieces, use the image acquisition device to acquire workpiece images. Combine the depth of the worktable and the depth of the workpiece to obtain the workpiece height. Determine whether the workpieces are superimposed based on the workpiece height. If so, proceed to step S2; otherwise, determine that the workpieces are not superimposed. S2. Calculate the number of pixels for each of the two workpiece colors using Blob analysis, obtain the area of ​​the workpiece corresponding to each color, and determine the three possible superposition scenarios for the two workpieces. This includes the following sub-steps: S1.

1. Through color recognition, the color of the workpiece is identified, and the workpiece color is binarized. Then, the number of pixels of the workpiece color is obtained through Blob analysis, and the area of ​​the workpiece corresponding to the two colors is calculated. S1.

2. Based on the area corresponding to the two colors, three superposition scenarios of the workpiece are obtained: (a) complete superposition of the two workpieces, (b) staggered superposition of the two workpieces, and (c) staggered superposition of the same type of workpieces; specifically: 1) If the area of ​​one of the colors is approximately equal to the area of ​​the whole circle: πR 2 or πr 2 R is the cross-sectional radius of the first cylindrical workpiece, and r is the cross-sectional radius of the second cylindrical workpiece. The corresponding cylindrical workpiece is on top. If the area of ​​the workpiece of another color is zero, it is the superposition case (a). 2) If the area of ​​the workpiece of another color is not zero, then it is a superposition case (b); 3) If the area of ​​a certain color is greater than the area of ​​the whole circle: πR 2 or πr 2 This is the case of superposition (c); S3. Establish a robot kinematic model, use the edge fitting method to determine the edge point set of the workpiece, obtain the coordinates of the workpiece in the pixel coordinate system, solve the centroid coordinates of the workpiece in the robot base coordinate system through inverse kinematics, perform trajectory planning, obtain the angle values ​​of each joint of the robot, and generate motion control commands. Based on the three superposition cases of the workpiece, control the robot through motion control commands to realize the grasping of the workpiece.

2. The servo control method based on robot image processing according to claim 1, characterized in that, The hand-eye calibration method in step S1 is as follows: S101. Fix the calibration plate to the robot end effector. The transformation matrix between the calibration plate and the robot end effector... Represented as: in, The transformation matrix representing the robot's base coordinate system relative to the end effector is obtained through robot forward kinematics. This represents the transformation matrix between the coordinate system of the image acquisition device and the robot's base coordinate system. This represents the transformation matrix of the calibration plate relative to the camera coordinate system, which is obtained through camera calibration; S102. Perform the hand-eye calibration process. Keeping the position unchanged, control the robot to move to any two poses to obtain the formula: The above equation, after shifting terms left and right: make The above expression can be represented as: AX = XB The hand-eye calibration result is calculated using the above formula.

3. The servo control method based on robot image processing according to claim 1, characterized in that, In step S1, the following method is used to determine whether the workpieces are stacked: Calculate the height of the workpiece: depth of the worktable - depth of the workpiece = height of the workpiece. If the workpiece height is H+H or H+h or h+h, then the workpieces are considered to be stacked, where H is the height of the first cylinder and h is the height of the second cylinder.

4. The servo control method based on robot image processing according to claim 3, characterized in that, In step S1, color recognition involves converting the RGB color space to the HSV color space, specifically: The conversion formula between RGB color space and HSV color space is as follows: V = max.

5. The servo control method based on robot image processing according to claim 4, characterized in that, In step S3, determining the edge point set of the workpiece using the edge fitting method for the superposition cases (b) and (c) includes the following sub-steps: S301: Using Canny edge detection, the edge point set of the workpiece above is obtained as (x i ,y i Then determine the following work. The initial point set of the component is (X) i ,Y i According to the formula (X) i ,Y i )-(x i ,y i )∩(X i ,Y i ), remove the point set at the intersection of the edges to obtain the edge point set of the workpiece below; S302. Given a circle with radius R and center (A, B), the expression for the circle is: (x-A) 2 +(y-B) 2 =R 2 (1-1) Expanding the above equation (1-1) yields the following expression: x 2 -2Ax+A 2 +y 2 -2By+B 2 =R 2 (1-2) S303. After Canny edge detection, the point set of the Blob block edge is (X... n ,Y n If it is the workpiece above, then (X) n ,Y n )=(X i ,Y i If it is the workpiece below, then (X) n ,Y n )=(X i ,Y i )-(x i ,y i )∩(X i ,Y i ); S304, Point Set (X) n ,Y n The distance d from the center of the circle n for: Point set (X) n ,Y n The difference between the square of the distance to the center and the square of the radius is: Let Q(A,B,R) be σ n The sum of squares is expressed as: When Q(A,B,R) reaches its minimum value, the parameters A, B, and R can be obtained. By fitting the equation of the circle, the center of the circle is (A,B), which is the coordinate in the pixel coordinate system.

6. The servo control method based on robot image processing according to claim 5, characterized in that, In step S3, the centroid coordinates of the workpiece in the robot base coordinates are solved by inverse kinematics, including the following sub-steps: S3.

1. Based on the target's coordinates (u1, v1) in the pixel coordinate system, calculate its coordinates P in the camera coordinate system using the intrinsic parameter matrix M obtained from camera calibration. c (X c ,Y c Z c )for: Where M is the camera intrinsic parameter matrix, Z c The depth of the object in the camera coordinate system is obtained through the camera depth sensor; S3.

2. Based on the hand-eye calibration results, calculate the spatial coordinates of the target workpiece in the robot's base coordinate system: in, The transformation matrix is ​​the ratio of the camera coordinate system to the robot base coordinate system; (X) w Y w Z w ( ) represents the spatial coordinates of the centroid of the target workpiece in the robot's base coordinate system. S3.

3. Based on the rotation direction of the target workpiece on the conveyor belt, this rotation direction is defined as the positive direction of the X-axis. The time from image acquisition to the robot's first grasp is t1, and the conveyor belt speed is v. During this time, the centroid coordinates of the target workpiece change to (X... w +t1v, Y w Z w ); The time from the robot's first grasp to the second grasp is t2, and the centroid coordinates of the target object become (X... w +(t1+t2)v,Y w Z w ).

7. The servo control method based on robot image processing according to claim 1, characterized in that, This also includes controlling robots through digital twins, specifically: S401. Robot 3D Model Drawing: In CAD software, a 3D model is drawn based on the actual shape and structure of the robot, including drawing the robot's various components, parts, joints, and transmission system. S402, URDF file generation: Convert the robot's 3D model into a URDF file, defining the robot's connection relationships, joint types, joint constraints, mass, and inertial parameters; S403, Simscape Physical Model Generation: Import the robot's 3D model and URDF file into MATLAB / Simulink. Use the Simscape toolbox in MATLAB to create a physical model of the virtual robot. In Simulink, use the Simscape Multibody library to build the robot's kinematics and dynamics model, and define joint constraints, transmission systems, and sensors. S404. Simulation Environment Configuration: Set simulation parameters and environmental conditions in Simulink; S405, Control Algorithm Development: Using the Simscape physical model and simulation environment, develop robot control algorithms in Simulink. Implement autonomous robot movement using path planning and trajectory tracking algorithms, and optimize robot performance and behavior by adjusting control parameters and algorithm logic. S406, Digital Twin: Run a digital twin platform in Simulink to observe and analyze the robot's motion, mechanical response, and sensor output. View the robot's real-time animation, plot curves of key indicators, and perform data collection and analysis through a graphical interface.

8. A servo control system based on robot image processing, characterized in that, include: First host computer, image acquisition device, data transmission module, robot body, lower computer, digital twin platform; The image acquisition device is connected to the first host computer and outputs the image to the host computer. The first host computer is equipped with programming software and a real-time kernel. It uses the hand-eye calibration method to determine the conversion relationship between the robot arm and the image acquisition device. For two types of cylindrical workpieces, the image acquisition device acquires workpiece images. Combined with the depth of the worktable and the depth of the workpiece, the workpiece height is obtained. Based on the workpiece height, it is determined whether the workpieces are superimposed. If so, Blob analysis is used to calculate the number of pixels of the two workpiece colors and obtain the area of ​​the workpiece corresponding to the two colors. The three superposition cases of the two workpieces are judged. Otherwise, it is determined that the workpieces are not superimposed. Then, a robot kinematic model is established. The edge fitting method is used to determine the edge point set of the workpiece and obtain the coordinates of the workpiece in the pixel coordinate system. The centroid coordinates of the workpiece in the robot base coordinate system are solved by inverse kinematics. Trajectory planning is performed to obtain the angle values ​​of each joint of the robot and generate motion control commands. The process involves determining three possible superposition scenarios for the two workpieces; the specific steps are as follows: S1.

1. Through color recognition, the color of the workpiece is identified, and the workpiece color is binarized. Then, the number of pixels of the workpiece color is obtained through Blob analysis, and the area of ​​the workpiece corresponding to the two colors is calculated. S1.

2. Based on the area corresponding to the two colors, three superposition scenarios of the workpiece are obtained: (a) complete superposition of the two workpieces, (b) staggered superposition of the two workpieces, and (c) staggered superposition of the same type of workpieces; specifically: 1) If the area of ​​one of the colors is approximately equal to the area of ​​the whole circle: πR 2 or πr 2 R is the cross-sectional radius of the first cylindrical workpiece, and r is the cross-sectional radius of the second cylindrical workpiece. The corresponding cylindrical workpiece is on top. If the area of ​​the workpiece of another color is zero, it is the superposition case (a). 2) If the area of ​​the workpiece of another color is not zero, then it is a superposition case (b); 3) If the area of ​​a certain color is greater than the area of ​​the whole circle: πR 2 or πr 2 This is the case of superposition (c); The input terminal of the data transmission module is connected to the output terminal of the first host computer to receive motion control commands output by the first host computer. The output terminal of the data transmission module is connected to the input terminal of the slave computer to transmit the motion control commands to the slave computer. The microcontroller of the slave computer performs communication protocol conversion and outputs the commands to the robot it is connected to. Based on the three superimposed conditions of the workpiece, the robot is controlled by motion control commands to grasp the workpiece. The robot synchronously feeds back the real-time running data to the host computer for real-time data display. The first host computer is connected to the second host computer. A digital twin platform is built in the second host computer to feed back the real-time running status of the robot to the virtual robot, so as to realize the synchronous display of the virtual model and the physical robot. The data transmission module includes: an Ethernet switch, a CAN communication module, and a wireless transceiver module. The microcontroller is configured to perform the following actions: S501: Receive data from the data transmission module, parse it, and extract data fields and related information according to the specifications of the corresponding protocol; S502. Based on the communication protocol used by the target robot, repackage the parsed data fields into the target protocol's format; this includes data type conversion, format adjustment, and protocol-specific data packet assembly. S503. Use the target protocol to transmit the encapsulated data to the target robot via wired or wireless transmission.

Images