Mechanical arm action recognition and control method based on machine vision and digital twinning technology

Abstract

The invention relates to the field of robot technology, machine vision, deep learning, computer graphics and digital twinning, in particular to a mechanical arm action recognition and control method based on the machine vision and the digital twinning technology. The motion state and environment information of the mechanical arm are captured in real time through the machine vision technology, high-precision recognition of the motion of the mechanical arm can be achieved in combination with an image processing algorithm, and a control strategy is adjusted in real time according to the recognition result; based on the digital twin model, the state change of the mechanical arm can be monitored in real time, and the mechanical arm and the virtual model are synchronously driven, so that the action of the mechanical arm is ensured to be consistent with the expected action; through intelligent identification and automatic adjustment, manual operation errors are reduced, the working efficiency and the production precision are improved, and the complexity of system maintenance is reduced; real-time optimization of the operation process of the mechanical arm can be achieved through combination of digital twinning and machine vision, potential faults are avoided through simulation and prediction, and stable operation of a production line is ensured.

Description

Technical Field

[0001] The present invention relates to the fields of robotics, machine vision, deep learning, computer graphics and digital twins, and in particular to a method for recognizing and controlling robot arm motions based on machine vision and digital twin technologies. Background Art

[0002] With the development of intelligent manufacturing and industrial automation, the importance of robotic arms in various application scenarios continues to increase. The precise operation ability of robotic arms directly affects production efficiency and product quality, and the motion trajectory and posture recognition of robotic arms are the basis for their precise control. Traditional robotic arm control methods mostly rely on predetermined path planning and fixed motion instructions, lacking adaptability to dynamic environments and room for precision improvement.

[0003] In order to solve the above problems, many studies have begun to focus on robot arm motion recognition and control technology based on computer vision. Through high-precision visual recognition technology, the movement changes of the robot arm can be captured in real time, and then the various operations of the robot arm can be accurately controlled. However, traditional robot arm visual recognition methods generally have problems such as insufficient accuracy, poor real-time performance and high computational complexity, making it difficult to cope with complex industrial environments and changing task requirements.

[0004] Existing robot arm control methods usually rely on manual programming and preset models, lacking flexibility and intelligence. With the rapid development of deep learning and artificial intelligence technology, deep learning-based visual recognition technology has gradually been introduced into the motion control of robot arms, and has made some progress. In particular, posture recognition based on the YOLO (You Only Look Once) model has achieved good application results in the field of human motion recognition, but it still faces some challenges in the motion recognition and control of robot arms. In the field of virtual simulation, digital twin technology has gradually emerged. It realizes real-time monitoring and optimized control of the physical world by establishing an accurate mapping relationship between physical entities and virtual models. Combining digital twins with the real-time motion recognition technology of the robot arm, the precise driving and scheduling of the robot arm in the virtual environment can be realized, and the autonomy and flexibility of the robot arm can be improved.

[0005] Therefore, the present invention proposes a method for accurately identifying and driving robotic arm movements based on machine vision and digital twins, which aims to calibrate the joint key points in the robotic arm image, combine the YOLOv8 Pose model for training and tracking, and use the extracted key point coordinates to drive the robotic arm model in Unity, thereby achieving precise control and real-time simulation of the robotic arm movements, and providing a more efficient and intelligent solution for intelligent manufacturing and robotics technology. Summary of the invention

[0006] The present invention relates to a robot arm motion recognition and control method based on machine vision and digital twin technology, aiming to achieve efficient control and intelligent management of the robot arm in a complex working environment by deeply integrating real-time image processing and digital twin technology.

[0007] To achieve the above purpose, the present invention provides the following technical solutions:

[0008] The robot arm motion recognition and control method based on machine vision and digital twin technology includes the following steps:

[0009] S1. Use industrial cameras to obtain motion images or videos of the robotic arm, process the image or video data through deep learning algorithms, and extract key motion features;

[0010] S2. Use image annotation tools to accurately annotate the joint points of the segmented image and generate a standardized annotation file;

[0011] S3, dividing the annotated files and their corresponding image data into a training set and a validation set according to the proportion, and training the target detection model;

[0012] S4. Use the trained target detection model to identify the action of the robot arm, connect the marked joints with skeletons, accurately calculate the angles between the skeletons during the movement of the robot arm, and present them dynamically and visually in the image;

[0013] S5. Based on the geometric structure and motion model of the robotic arm, a digital twin model consistent with the physical robotic arm is established in the virtual environment to control the robotic arm.

[0014] Further, the S1 includes:

[0015] S1.1. Arrange the position of the industrial camera according to the movement area of ​​the robot arm, adjust the resolution and frame rate of the camera according to the size and movement trajectory of the robot arm, use the industrial camera to shoot the movement process of the robot arm, and generate a continuous image frame sequence or video data;

[0016] S1.2, decoding the original video data and converting it into a temporally continuous high-quality image frame sequence;

[0017] S1.3, preprocessing each extracted image frame to meet the input requirements of the target detection model;

[0018] S1.4, using yolov8 target detection algorithm, real-time analysis of the captured images is performed to identify the key parts of the robotic arm and its current motion status.

[0019] Further, the S4 includes:

[0020] S4.1. Use the trained object detection model to detect the positions of the joints in the motion of the robot arm, build the skeleton of the robot arm according to the detection results of the joints, and dynamically connect the skeleton. In each frame, according to the detected joint coordinates, connect the adjacent joints through line segments to form a skeleton structure.

[0021] S4.2. Dynamically visualize the skeleton and joint angles of the robot arm's motion process in the image, annotate the calculated angle values ​​on the lines between the joint points in real time, update the joint point and skeleton information in each frame, and ensure that the entire robot arm motion process can be dynamically tracked and displayed.

[0022] Furthermore, the robot arm joint angle calculation includes:

[0023] Step 1: Extract the center coordinates (x i ,y i ), where i represents different joint numbers;

[0024] Step 2: Calculate the two-dimensional angle: Assume that in the two-dimensional image plane, the coordinates of the two adjacent joints of the robot are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ between joint i and joint i+1 i The inverse tangent function can be used to calculate:

[0025] θ i =tan -1 2(y i+1 -y i ,x i+1 -x i )

[0026] Among them, tan -1 2 is the inverse tangent function, which is used to process angles ranging from -π to π on a two-dimensional plane;

[0027] Step 3: Angle calculation in three-dimensional space: Using 3D position information, use the vector method to calculate the angle of the joint. Assume that the coordinates of joints i and i+1 in 3D space are (x i ,y i ,z i ) and (x i+1 ,y i+1 ,z i+1 ), the angle calculation between joints can be based on the vector dot product formula:

[0028] v i =(x i+1 -x i ,yi+1 -y i ,z i+1 -z i )

[0029] Calculate the angle θ using the dot product formula i :

[0030]

[0031] Among them, v i and v i+1 It is the vector between adjacent joints, and the result of the dot product calculation is used to get the angle through the arc cosine function.

[0032] Further, the S5 includes:

[0033] S5.1. Establish the basic structure of the digital twin of the robot through its geometric parameters and kinematic model, and obtain the specific geometric parameters of the robot, including the connecting rod length, joint type, joint limit angle and working range;

[0034] S5.2. Use forward kinematics to describe the position from the base to the end effector, define the relative position of each link and joint based on DH parameters, establish the motion model of the robot arm, and calculate the joint angles required for a given end effector position through inverse kinematics;

[0035] S5.3. Create a digital twin model consistent with the physical robotic arm in the virtual environment, including the geometric structure and motion model. Select Unity industrial simulation software, use SolidWorks to create a 3D geometric model of the robotic arm, and import it into the Unity simulation platform;

[0036] S5.4. Map the real-time movements of the physical robotic arm to the digital twin model to achieve synchronization between virtual and reality. The joint angles obtained in real time by the machine vision system are transmitted to the robotic arm joint control system in the virtual environment to update the posture of the virtual model.

[0037] The beneficial effects of the present invention are as follows: the robot arm motion recognition and control method based on machine vision and digital twin technology provided by the present invention can effectively improve the motion recognition accuracy of the robot arm, optimize the motion control performance of the robot arm, and enhance the adaptability of the robot arm in complex tasks. Through the real-time capture of the motion state and environmental information of the robot arm by machine vision technology, combined with the image processing algorithm, it is possible to achieve high-precision recognition of the robot arm motion, and adjust the control strategy in real time according to the recognition result; based on the digital twin model, it is possible to monitor the state change of the robot arm in real time, and drive it synchronously with the virtual model to ensure that the motion of the robot arm is consistent with the expected motion; through intelligent recognition and automatic adjustment, human operation errors are reduced, work efficiency and production accuracy are improved, and the complexity of system maintenance is reduced; the combination of digital twin and machine vision can achieve real-time optimization of the robot arm operation process, and avoid potential faults through simulation and prediction, ensuring the stable operation of the production line. The present invention has broad application prospects and can be widely used in industrial automation, intelligent manufacturing, logistics handling, medical equipment and other fields. In these fields, the efficient control and intelligent management of the robot arm will significantly improve production efficiency, reduce operational risks, and improve the overall effect of human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The 3D camera coordinates P = (X, Y, Z) of the present invention are converted into 2D normalized plane coordinates (x, y, 1);

[0039] Figure 2 is the relative position of the camera and the mechanical arm of the present invention;

[0040] Figure 3 It is the image data obtained after processing and segmentation;

[0041] Figure 4 It is a visualization image of the annotated skeleton connection and the angle of the robot arm during movement;

[0042] Figure 5 It is a digital twin model of the robotic arm established by the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0044] Example 1: Figure 1-5 As shown, the present invention mainly creates an accurate robot arm motion recognition and control system by combining machine vision technology and digital twin technology. The motion data of the robot arm is collected by machine vision, and the digital twin model is used to perform accurate motion comparison and recognition, thereby driving the robot arm to achieve accurate motion control. The main steps include:

[0045] S1. Use industrial cameras to obtain motion images or videos of the robotic arm, process the image or video data through deep learning algorithms, and extract key motion features, including:

[0046] S1.1. Arrange the position of the industrial camera according to the movement area of ​​the robot arm, and adjust the camera resolution and frame rate according to the size and movement trajectory of the robot arm to ensure the clarity of the image and the continuity of the data. Use the industrial camera to shoot the movement process of the robot arm to generate a continuous image frame sequence or video data. The video should capture each movement stage of the robot arm to ensure the integrity of the data.

[0047] In this embodiment, a high-resolution industrial camera is used with high-speed image capture and automatic adjustment functions to capture image data of the robot arm and its working environment in real time. The position of the camera takes into account the range of motion of the robot arm to ensure that all key operating areas are covered. To adapt to different lighting conditions and background changes, the system has automatic exposure, white balance adjustment and noise suppression functions to ensure that the clarity and quality of the captured images are relatively high. Image acquisition is not just simple image capture, but also includes real-time monitoring of dynamic scenes. In a rapidly changing environment, the system can continuously acquire image frames to ensure that every key moment is captured. These high-quality image data will provide the basis for subsequent action recognition.

[0048] S1.2. Decode the raw video data and convert it into a temporally continuous sequence of high-quality image frames. This process involves parsing the video encoding format and container format, extracting each frame in the video stream to ensure that subsequent image processing and analysis can be performed accurately at the frame level. By using professional video processing tools or libraries (OpenCV), the conversion of video to image frames can be completed efficiently.

[0049] S1.3. Preprocess each extracted image frame to meet the input requirements of the target detection model. The captured image data needs to be processed in real time, including noise removal, image enhancement, and illumination compensation, to improve image quality.

[0050] Adjust the image frame to the input size required by the deep learning model, which usually involves precise scaling or cropping operations to ensure consistency in spatial resolution and avoid degradation of model performance due to size differences. The pixel values ​​of the image are then normalized to map the pixel values ​​to a specific numerical range (between 0 and 1). This normalization operation helps to eliminate differences in brightness and contrast between different images, reduce the variance of the data, improve the stability of the training process, and promote rapid convergence of the model. In addition, normalization can prevent problems such as gradient disappearance or explosion, making the optimization process smoother and more efficient. Normalization is a crucial step in image preprocessing, which aims to adjust the pixel values ​​of the image to a uniform numerical range or distribution to improve the training efficiency and performance of the deep learning model.

[0051] Since different images may have different brightness and contrast, normalization can minimize these differences, thereby standardizing the data distribution and improving the stability and performance of model training. A uniform data distribution helps the optimization algorithm converge to the global optimal solution faster. A common method is to linearly scale the pixel value from the original range [0,255] to [0,1], that is, through the formula: This linear normalization method is simple and efficient, suitable for initial data preprocessing in most cases, and helps speed up the training process and improve the generalization ability of the model.

[0052] S1.4. In this embodiment, the YOLOv8 target detection algorithm is used to perform real-time analysis on the captured images. The model has high recognition accuracy and real-time processing capabilities, and can quickly identify the key parts of the robotic arm and its current motion state. By analyzing the targets in the image, the system can accurately determine the movement direction, speed and posture of the robotic arm.

[0053] S2. Use an image annotation tool to accurately annotate the joint points of the segmented image and generate a standardized annotation file. In this embodiment, the annotation tool Labelme is used to annotate the key joint points of the robotic arm.

[0054] S3. Divide the annotated files and their corresponding image data into training sets and validation sets in proportion to train the target detection model; divide them into 80% training sets and 20% validation sets. Ensure that the validation set can reflect the diversity of various robot arm postures in order to evaluate the performance of the model in different scenarios. Ensure the randomness of the data when dividing to avoid a large number of similar samples in the training set, which may lead to overfitting of the model. In addition, there are sufficient sample coverage of various types of robot arm postures and joint movements in the training set.

[0055] S4. The trained target detection model is used to identify the action of the robotic arm, connect the marked joints with skeletons, accurately calculate the angles between the skeletons during the movement of the robotic arm, and dynamically visualize them in the image.

[0056] S4.1. Use the trained object detection model to detect the positions of the joints in the motion of the robot arm, build the skeleton of the robot arm according to the detection results of the joints, and dynamically connect the skeleton. In each frame, according to the detected joint coordinates, connect the adjacent joints through line segments to form a skeleton structure.

[0057] S4.2. Dynamically visualize the skeleton and joint angles of the robot arm's motion process in the image, annotate the calculated angle values ​​on the lines between the joint points in real time, update the joint point and skeleton information in each frame, and ensure that the entire robot arm motion process can be dynamically tracked and displayed.

[0058] Furthermore, the robot arm joint angle calculation includes:

[0059] Step 1: Extract the center coordinates (x i ,y i ), where i represents different joint numbers;

[0060] Step 2: Calculate the two-dimensional angle: Assume that in the two-dimensional image plane, the coordinates of the two adjacent joints of the robot are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ between joint i and joint i+1 i The inverse tangent function can be used to calculate:

[0061] θ i =tan -1 2(y i+1 -y i ,x i+1 -x i )

[0062] Among them, tan -1 2 is the inverse tangent function, which is used to process angles ranging from -π to π on a two-dimensional plane;

[0063] Step 3: Angle calculation in three-dimensional space: Using 3D position information, use the vector method to calculate the angle of the joint. Assume that the coordinates of joints i and i+1 in 3D space are (x i ,y i ,z i ) and (x i+1 ,y i+1 ,zi+1 ), the angle calculation between joints can be based on the vector dot product formula:

[0064] v i =(x i+1 -x i ,y i+1 -y i ,z i+1 -z i )

[0065] Calculate the angle θ using the dot product formula i :

[0066]

[0067] Among them, v i and v i+1 It is the vector between adjacent joints, and the result of the dot product calculation is used to get the angle through the arc cosine function.

[0068] The multiple joints of a robotic arm usually form a series of joint chains, so the angle of each joint may have a certain geometric relationship with the adjacent joints. By calculating the angle of each joint, the posture information of the entire robotic arm can be obtained.

[0069] The calculated angle parameters are marked on each joint of the robot arm, and the system can display the angle changes of each joint in real time. The joint position information and calculated angle detected by the yolov8 model can provide the operator with visual joint angle feedback;

[0070] As the robotic arm moves, the yolov8 model is able to continuously capture the joint positions of each frame and calculate the angles in real time, thereby providing angle parameter support for the precise control and operation of the robotic arm.

[0071] S5. Based on the geometric structure and motion model of the robot arm, a digital twin model consistent with the physical robot arm is established in the virtual environment to control the robot arm. In the process of building the digital twin model, the system first creates a corresponding virtual model based on the motion parameters of the robot arm and the external environmental conditions. The model can receive status information from the robot arm in real time and combine it with actual operations. In this way, users can intuitively monitor the operating status of the robot arm in the virtual environment, identify potential problems in a timely manner and make adjustments.

[0072] S5.1. Establish the basic structure of the digital twin of the robot through its geometric parameters and kinematic model, and obtain the specific geometric parameters of the robot, including the connecting rod length, joint type, joint limit angle and working range;

[0073] S5.2. Use forward kinematics to describe the position from the base to the end effector, define the relative position of each link and joint based on DH parameters, establish the motion model of the robot arm, and calculate the joint angles required for a given end effector position through inverse kinematics;

[0074] S5.3. Create a digital twin model consistent with the physical robotic arm in the virtual environment, including the geometric structure and motion model. Select Unity industrial simulation software, use SolidWorks to create a 3D geometric model of the robotic arm, and import it into the Unity simulation platform;

[0075] S5.4. Map the real-time movements of the physical robotic arm to the digital twin model to achieve synchronization between virtual and reality. The joint angles obtained in real time by the machine vision system are transmitted to the robotic arm joint control system in the virtual environment to update the posture of the virtual model.

[0076] Once the motion recognition and digital twin model construction are completed, the system enters the motion drive and control stage. In this stage, the system dynamically adjusts the motion instructions of the robot arm according to the status of the digital twin model. Through real-time data analysis of the robot arm, the system can identify and handle abnormal situations to ensure that the robot arm can still operate stably in complex environments.

[0077] The control strategy of the present invention adopts a closed-loop feedback mechanism, which enables the robot arm to adjust itself according to the actual operation situation. For example, when an obstacle is identified, the system can immediately change the movement path of the robot arm to avoid collision. At the same time, the system can also continuously optimize the operating efficiency of the robot arm through real-time feedback and achieve more refined control. The specific steps are:

[0078] Step 1: Generate accurate motion instructions based on the state and motion recognition results of the digital twin model to ensure that the robot arm's movements meet expectations;

[0079] Step 2: The system can monitor the operating status of the robot arm in real time and dynamically adjust the motion strategy according to environmental changes and task requirements;

[0080] Step 3: When obstacles or abnormal situations are identified, the system can respond quickly, adjust the path or change the operation strategy to avoid collision and ensure safety.

[0081] In order to improve the intelligence level of the system, the present invention also introduces a self-learning and optimization mechanism. By collecting and analyzing historical data, the system can use machine learning algorithms to continuously optimize motion recognition and control strategies. This process not only includes fine-tuning of existing actions, but also adaptive learning in new environments, thereby continuously improving overall performance. For example, the system can identify which action combinations are more efficient and which environmental factors have a greater impact on operational accuracy based on past operating data, thereby automatically adjusting strategies in subsequent tasks. This self-learning ability makes the robotic arm more flexible and efficient in complex tasks, and can adapt to different working conditions and task requirements. The specific steps are:

[0082] Step 1: The system collects historical data for each operation, including information such as the success rate, efficiency, and error types of the action execution;

[0083] Step 2: Analyze historical data and use machine learning algorithms to continuously optimize the motion recognition model and control strategy to improve the overall performance of the system;

[0084] Step 3: Obtain feedback from actual operations, form a closed loop of continuous learning, and improve the ability to adapt to new environments.

[0085] This invention combines machine vision with digital twin technology to provide an innovative solution that improves the operating accuracy and intelligence level of the robot arm in complex environments. With the continuous development of industrial automation technology, this invention will provide important theoretical support and practical guidance for the construction of future intelligent manufacturing and intelligent factories.

Claims

1. A robot arm motion recognition and control method based on machine vision and digital twin technology, characterized in that: The steps include: S1. Use industrial cameras to obtain motion images or videos of the robotic arm, process the image or video data through deep learning algorithms, and extract key motion features; S2. Use image annotation tools to accurately annotate the joint points of the segmented image and generate a standardized annotation file; S3, dividing the annotated files and their corresponding image data into a training set and a validation set according to the proportion, and training the target detection model; S4. Use the trained target detection model to identify the action of the robot arm, connect the marked joints with skeletons, accurately calculate the angles between the skeletons during the movement of the robot arm, and dynamically visualize them in the image; S5. Based on the geometric structure and motion model of the robotic arm, a digital twin model consistent with the physical robotic arm is established in the virtual environment to control the robotic arm.

2. The method for robot arm motion recognition and control based on machine vision and digital twin technology according to claim 1, characterized in that: The S1 includes: S1.

1. Arrange the position of the industrial camera according to the movement area of ​​the robot arm, adjust the resolution and frame rate of the camera according to the size and movement trajectory of the robot arm, use the industrial camera to shoot the movement process of the robot arm, and generate a continuous image frame sequence or video data; S1.2, decoding the original video data and converting it into a temporally continuous high-quality image frame sequence; S1.3, preprocessing each extracted image frame to meet the input requirements of the target detection model; S1.4, using yolov8 target detection algorithm, real-time analysis of the captured images is performed to identify the key parts of the robotic arm and its current motion status.

3. The method for robot arm motion recognition and control based on machine vision and digital twin technology according to claim 1, characterized in that: The S4 includes: S4.

1. Use the trained object detection model to detect the positions of the joints in the motion of the robot arm, build the skeleton of the robot arm according to the detection results of the joints, and dynamically connect the skeleton. In each frame, according to the detected joint coordinates, connect the adjacent joints through line segments to form a skeleton structure. S4.

2. Dynamically visualize the skeleton and joint angles of the robot arm's motion process in the image, annotate the calculated angle values ​​on the lines between the joint points in real time, update the joint point and skeleton information in each frame, and ensure that the entire robot arm motion process can be dynamically tracked and displayed.

4. The method for robot arm motion recognition and control based on machine vision and digital twin technology according to claim 3 is characterized in that: The robot arm joint angle calculation includes: Step 1: Extract the center coordinates (x i ,y i ), where i represents different joint numbers; Step 2: Calculate the two-dimensional angle: Assume that in the two-dimensional image plane, the coordinates of the two adjacent joints of the robot are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ between joint i and joint i+1 i The inverse tangent function can be used to calculate: θ i =tan -1 2(and i+1 -and i ,x i+1 -x i ) Among them, tan -1 2 is the inverse tangent function, which is used to process angles ranging from -π to π on a two-dimensional plane; Step 3: Angle calculation in three-dimensional space: Using 3D position information, use the vector method to calculate the angle of the joint. Assume that the coordinates of joints i and i+1 in 3D space are (x i ,y i ,z i ) and (x i+1 ,y i+1 ,z i+1 ), the angle calculation between joints can be based on the vector dot product formula: v i =(x i+1 -x i ,y i+1 -y i ,z i+1 -z i ) Calculate the angle θ using the dot product formula i : Among them, v i and v i+1 It is the vector between adjacent joints, and the result of the dot product calculation is used to get the angle through the arc cosine function.

5. The method for robot arm motion recognition and control based on machine vision and digital twin technology according to claim 1, characterized in that: The S5 includes: S5.

1. Establish the basic structure of the digital twin of the robot through its geometric parameters and kinematic model, and obtain the specific geometric parameters of the robot, including the connecting rod length, joint type, joint limit angle and working range; S5.

2. Use forward kinematics to describe the position from the base to the end effector, define the relative position of each link and joint based on DH parameters, establish the motion model of the robot arm, and calculate the joint angles required for a given end effector position through inverse kinematics; S5.

3. Create a digital twin model consistent with the physical robotic arm in the virtual environment, including the geometric structure and motion model. Select Unity industrial simulation software, use SolidWorks to create a 3D geometric model of the robotic arm, and import it into the Unity simulation platform; S5.

4. Map the real-time movements of the physical robotic arm to the digital twin model to achieve synchronization between virtual and reality. The joint angles obtained in real time by the machine vision system are transmitted to the robotic arm joint control system in the virtual environment to update the posture of the virtual model.

Images