Mechanical arm action recognition and control method based on machine vision and digital twin technology

Abstract

The present application relates to the fields of robotics, machine vision, deep learning, computer graphics and digital twinning, in particular a mechanical arm action recognition and control method based on machine vision and digital twinning technology. The present application can realize high-precision recognition of mechanical arm action by real-time capturing of motion state and environmental information of the mechanical arm through machine vision technology, combining with image processing algorithm, and real-time adjusting control strategy according to the recognition result; based on the digital twinning model, the state change of the mechanical arm can be monitored in real time, and synchronous driving with the virtual model is realized, so that the action of the mechanical arm is consistent with the expected action; through intelligent recognition and automatic adjustment, the human operation error is reduced, the work efficiency and production precision are improved, and the complexity of system maintenance is reduced; the combination of digital twinning and machine vision can realize real-time optimization of the mechanical arm operation process, and avoid potential faults through simulation and prediction, so as to ensure the stable operation of the production line.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of robotics, machine vision, deep learning, computer graphics and digital twinning, and in particular to a mechanical arm action recognition and control method based on machine vision and digital twinning technology. BACKGROUND

[0002] With the development of intelligent manufacturing and industrial automation, the importance of mechanical arms in various application scenarios is continuously increasing. The precise operation capability of a mechanical arm directly affects production efficiency and product quality, and the motion trajectory and posture recognition of a mechanical arm is the basis for its precise control. Traditional mechanical arm control methods mostly rely on predetermined path planning and fixed action instructions, lacking adaptability to dynamic environments and room for precision improvement.

[0003] To solve the above problems, many studies have begun to focus on mechanical arm action recognition and control technology based on computer vision. Through high-precision visual recognition technology, the action changes of a mechanical arm can be captured in real time, and then each operation of the mechanical arm can be precisely controlled. However, traditional mechanical arm visual recognition methods generally have problems such as insufficient accuracy, poor real-time performance and high computational complexity, making it difficult to cope with complex industrial environments and changing task requirements.

[0004] Existing mechanical arm control methods usually rely on manual programming and preset models, lacking flexibility and intelligence. With the rapid development of deep learning and artificial intelligence technologies, visual recognition technology based on deep learning has been gradually introduced into the motion control of mechanical arms, achieving certain progress. In particular, posture recognition based on the YOLO (You Only Look Once) model has achieved good application results in human action recognition, but still faces some challenges in mechanical arm action recognition and control. In the field of virtual simulation, digital twinning technology is gradually emerging, which realizes real-time monitoring and optimal control of the physical world by establishing an accurate mapping relationship between physical entities and virtual models. Combining digital twinning with real-time action recognition technology of a mechanical arm can achieve precise driving and scheduling of the mechanical arm in a virtual environment, improving the autonomy and flexibility of the mechanical arm.

[0005] Therefore, the present application proposes a mechanical arm action precise recognition and driving method based on machine vision and digital twinning, aiming to calibrate joint key points in images of a mechanical arm, train and track with a YOLOv8 Pose model, drive a mechanical arm model in Unity using extracted key point coordinates, and thus achieve accurate control and real-time simulation of mechanical arm actions, providing a more efficient and intelligent solution for intelligent manufacturing and robotics. SUMMARY

[0006] The application relates to a mechanical arm action recognition and control method based on machine vision and digital twin technology, aiming to realize efficient control and intelligent management of the mechanical arm in a complex working environment through deep integration of real-time image processing and digital twin technology.

[0007] To achieve the above-mentioned purpose, the application provides the following technical scheme.

[0008] The mechanical arm action recognition and control method based on machine vision and digital twin technology comprises the following steps.

[0009] S1, acquiring motion images or videos of the mechanical arm by using an industrial camera, processing the image or video data by a deep learning algorithm, and extracting key motion features;

[0010] S2, accurately labeling the joint nodes of the segmented images by using an image labeling tool, and generating standardized labeling files;

[0011] S3, dividing the labeled files and corresponding image data into a training set and a verification set according to a proportion, and training a target detection model;

[0012] S4, recognizing the action of the mechanical arm by using the trained target detection model, connecting the labeled joint nodes, accurately calculating the included angles between the skeletons in the motion process of the mechanical arm, and dynamically visualizing them in the images;

[0013] S5, establishing a digital twin model consistent with the physical mechanical arm in a virtual environment according to the geometric structure and motion model of the mechanical arm, and controlling the mechanical arm.

[0014] Further, the S1 comprises:

[0015] S1.1, arranging the position of the industrial camera according to the motion area of the mechanical arm, adjusting the resolution and frame rate of the camera according to the size and motion trajectory of the mechanical arm, using the industrial camera to shoot the motion process of the mechanical arm, and generating continuous image frame sequences or video data;

[0016] S1.2, decoding the original video data to convert it into a time-continuous high-quality image frame sequence;

[0017] S1.3, preprocessing each extracted image frame to meet the input requirements of the target detection model;

[0018] S1.4, using the yolov8 target detection algorithm to analyze the captured images in real time, and identifying the key parts of the mechanical arm and the current action state thereof.

[0019] Further, the S4 comprises:

[0020] S4.1, using the trained target detection model to detect the joint position in the motion of the robot arm, constructing the skeleton of the robot arm according to the detection result of the joint, and dynamically connecting the skeleton, in each frame, according to the detected joint coordinates, connecting adjacent joints through a line segment to form a skeleton structure;

[0021] S4.2, dynamically visualizing the skeleton and joint angle of the robot arm motion process in the image, real-time labeling the calculated angle value on the connecting line between the joints, updating the joint and skeleton information in each frame, and ensuring that the whole robot arm motion process can be dynamically tracked and displayed.

[0022] Further, the robot arm joint angle calculation includes:

[0023] Step one, through the target detection model, extracting the center coordinates (x i ,y i ) of each joint, where i represents different joint numbers;

[0024] Step two, two-dimensional angle calculation: assuming that the coordinates of the adjacent two joints of the robot arm in the two-dimensional image plane are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ i between joint i and joint i+1 can be calculated using the arctangent function:

[0025] θ i =tan -1 2(y i+1 -y i ,x i+1 -x i )

[0026] Where tan -1 2 is the arctangent function, used to handle angles in the range of -π to π in the two-dimensional plane;

[0027] Step three, angle calculation in three-dimensional space: using 3D position information, using the method of vector to calculate the included angle of the joint, assuming that the coordinates of joint i and i+1 in 3D space are (x i ,y i ,z i ) and (x i+1 ,y i+1 ,z i+1 ), the angle calculation between the joints can be based on the vector dot product formula:

[0028] v i =(x i+1 -x i ,yi+1 - y i , z i+1 - z i )

[0029] Calculate the angle θ by dot product formula i :

[0030]

[0031] where v i and v i+1 are the vectors between adjacent joints, and the dot product is calculated by the inverse cosine function to obtain the angle.

[0032] Further, the S5 comprises:

[0033] S5.1, establish the basic structure of its digital twin through the geometric parameters and kinematic model of the mechanical arm, obtain the specific geometric parameters of the mechanical arm, including link length, joint type, joint limit angle and working range;

[0034] S5.2, describe the pose from the base to the end effector using forward kinematics, define the relative position of each link and joint based on DH parameters, establish the motion model of the mechanical arm, and calculate the required joint angle when the end effector position is given through inverse kinematics;

[0035] S5.3, create a digital twin model consistent with the physical mechanical arm in the virtual environment, including geometric structure and motion model, choose Unity industrial simulation software, use SolidWorks to create a 3D geometric model of the mechanical arm, and import it into the unity simulation platform;

[0036] S5.4, map the real-time actions of the physical mechanical arm to the digital twin model, realize the synchronization of virtual and reality, and pass the angle data obtained by the machine vision system to the mechanical arm joint control system in the virtual environment, update the posture of the virtual model.

[0037] The beneficial effects of the present invention are as follows: the robot arm motion recognition and control method based on machine vision and digital twin technology provided by the present invention can effectively improve the robot arm's motion recognition accuracy, optimize the robot arm's motion control performance, and enhance the robot arm's adaptability in complex tasks. By using machine vision technology to capture the robot arm's motion state and environmental information in real time, combined with image processing algorithms, it can achieve high-precision recognition of the robot arm's motion and adjust the control strategy in real time based on the recognition results; based on the digital twin model, it can monitor the robot arm's state changes in real time and drive it synchronously with the virtual model to ensure that the robot arm's motion is consistent with the expected movement; through intelligent recognition and automatic adjustment, human operation errors are reduced, work efficiency and production accuracy are improved, and the complexity of system maintenance is reduced; the combination of digital twin and machine vision can achieve real-time optimization of the robot arm's operation process, and avoid potential failures through simulation and prediction, ensuring the stable operation of the production line. The present invention has broad application prospects and can be widely used in industrial automation, intelligent manufacturing, logistics handling, medical equipment and other fields. In these fields, efficient control and intelligent management of robot arms will significantly improve production efficiency, reduce operational risks, and improve the overall effect of human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The 3D camera coordinates P = (X, Y, Z) of the present invention are converted into 2D normalized plane coordinates (x, y, 1);

[0039] Figure 2 is the relative position of the camera and the robotic arm of the present invention;

[0040] Figure 3 It is the image data obtained after processing and segmentation;

[0041] Figure 4 It is a visualization image of the annotated skeleton connection and the angle of the robotic arm during movement;

[0042] Figure 5 It is a digital twin model of the robotic arm established by the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1: Figure 1-5 As shown, the present invention mainly creates an accurate robot arm motion recognition and control system by combining machine vision technology and digital twin technology. The robot arm's motion data is collected through machine vision, and the digital twin model is used to perform accurate motion comparison and recognition, thereby driving the robot arm to achieve precise motion control. The main steps include:

[0045] S1, acquire motion images or videos of the robot arm using industrial cameras, process the image or video data through deep learning algorithms, and extract key motion features, including:

[0046] S1.1, arrange the position of the industrial camera according to the motion area of the robot arm, adjust the resolution and frame rate of the camera according to the size and motion trajectory of the robot arm, to ensure the clarity of the image and the continuity of the data. Use industrial cameras to shoot the motion process of the robot arm, generate continuous image frame sequences or video data, the video should capture each motion stage of the robot arm, ensure the integrity of the data.

[0047] In this embodiment, high-resolution industrial cameras are used, with high-speed image capture and automatic adjustment functions, to capture image data of the robot arm and its working environment in real time. The position of the camera takes into account the motion range of the robot arm, ensuring that all key operation areas are covered. To adapt to different lighting conditions and background changes, the system has automatic exposure, white balance adjustment and noise suppression functions, making the captured images clear and high quality. Image acquisition is not just simple image capture, but also includes real-time monitoring of dynamic scenes. In a rapidly changing environment, the system can continuously acquire image frames, ensuring that every key moment is captured. These high-quality image data will provide a basis for subsequent action recognition.

[0048] S1.2, decode the original video data to convert it into a sequence of high-quality image frames in time; this process involves parsing the encoding format and container format of the video, extracting each frame of the video stream to ensure that subsequent image processing and analysis can be accurately operated at the frame level. By using professional video processing tools or libraries (OpenCV), the conversion of video to image frames can be efficiently completed.

[0049] S1.3, preprocess each extracted image frame to meet the input requirements of the target detection model. The captured image data needs to be processed in real time, including noise removal, image enhancement and illumination compensation, to improve the image quality.

[0050] Adjust the image frames to the input size required by the deep learning model, which usually involves precise scaling or cropping operations to ensure consistent spatial resolution and avoid model performance degradation due to size differences. Normalize the pixel values of the images by mapping them to a specific value range (0 to 1). This normalization operation helps to eliminate brightness and contrast differences between different images, reduce data variance, improve training process stability, and promote model convergence. In addition, normalization can prevent gradient vanishing or explosion problems, making the optimization process more stable and efficient. Normalization is a crucial step in image preprocessing, aiming to adjust the pixel values of images to a uniform numerical range or distribution to improve the training efficiency and performance of deep learning models.

[0051] Due to the different brightness and contrast of different images, normalization can minimize these differences, standardize the data distribution, and improve the stability and performance of model training. A unified data distribution helps optimization algorithms converge to the global optimal solution more quickly. A common method is to linearly scale the pixel values from the original range [0, 255] to [0, 1] using the formula: This linear normalization method is simple and efficient, suitable for initial data preprocessing in most cases, and helps to speed up the training process and improve the generalization ability of the model.

[0052] S1.4、In this embodiment, the yolov8 target detection algorithm is used to analyze the captured images in real time. This model has high recognition accuracy and real-time processing capability, and can quickly identify the key parts of the robot arm and its current action state. By analyzing the targets in the image, the system can accurately determine the motion direction, speed and posture of the robot arm.

[0053] S2、Use image annotation tools to accurately annotate the segmented images, and generate standardized annotation files; in this embodiment, the Labelme annotation tool is used to annotate the key joints of the robot arm.

[0054] S3、Divide the annotated files and their corresponding image data into training set and validation set according to the proportion, and train the target detection model; divide according to the proportion of 80% training set and 20% validation set. Ensure that the validation set can reflect the diversity of various robot arm postures, so as to evaluate the performance of the model in different scenarios. When dividing, ensure the randomness of the data to avoid including a large number of similar samples in the training set, which may cause the model to overfit. In addition, there are enough samples covering various types of robot arm postures and joint movements in the training set.

[0055] S4, action recognition of the mechanical arm is performed through the trained target detection model, the labeled joint nodes are connected, the angles between the joints in the movement of the mechanical arm are accurately calculated, and the dynamic visualization is presented in the image.

[0056] S4.1, the joint node positions in the movement of the mechanical arm are detected using the trained target detection model, the skeleton of the mechanical arm is constructed according to the detection results of the joint nodes, and the skeleton is dynamically connected, in each frame, the adjacent joint nodes are connected through a line segment according to the detected joint node coordinates, forming a skeleton structure;

[0057] S4.2, the skeleton and joint angle in the movement process of the mechanical arm are dynamically visualized and presented in the image, the calculated angle values are labeled on the connecting line between the joint nodes in real time, the joint node and skeleton information is updated in each frame, and the whole movement process of the mechanical arm can be dynamically tracked and displayed.

[0058] Further, the mechanical arm joint angle calculation includes:

[0059] Step one, through the target detection model, the center coordinates (x i ,y i ) of each joint are extracted, wherein i represents different joint numbers;

[0060] Step two, two-dimensional angle calculation: assuming that the coordinates of the adjacent two joints of the mechanical arm in the two-dimensional image plane are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ i between joint i and joint i+1 can be calculated using the arctangent function:

[0061] θ i =tan -1 2(y i+1 -y i ,x i+1 -x i )

[0062] Wherein, tan -1 2 is the arctangent function, used to process the angle in the range of-π to π in the two-dimensional plane;

[0063] Step three, angle calculation in three-dimensional space: using 3D position information, the angle between the joints is calculated using the vector method, assuming that the coordinates of joint i and i+1 in 3D space are (x i ,y i ,z i ) and (x i+1 ,y i+1 ,zi+1 ), the angle calculation between joints can be based on the vector dot product formula:

[0064] v i = (x i+1 -x i , y i+1 -y i , z i+1 -z i )

[0065] Calculate the included angle θ i by the dot product formula

[0066]

[0067] where v i and v i+1 are vectors between adjacent joints, and the dot product is calculated by the inverse cosine function to obtain the angle.

[0068] The multiple joints of the robot arm usually form a chain of joints, so the angle of each joint may have a certain geometric relationship with the adjacent joint. By calculating the angle of each joint, the pose information of the entire robot arm can be obtained.

[0069] Label the calculated angle parameters on each joint of the robot arm, and the system can display the angle change of each joint in real time. Using the joint position information detected by the yolov8 model and the calculated angle, the operator can be provided with visual joint angle feedback;

[0070] When the robot arm moves, the yolov8 model can continuously capture the joint position of each frame and calculate the angle in real time, thereby providing angle parameter support for accurate control and operation of the robot arm.

[0071] S5、According to the geometric structure and motion model of the robot arm, a digital twin model consistent with the physical robot arm is established in the virtual environment, and the robot arm is controlled. In the process of constructing the digital twin model, the system first creates a corresponding virtual model according to the motion parameters of the robot arm and the external environmental conditions. This model can receive state information from the robot arm in real time and combine it with actual operation. In this way, users can intuitively monitor the running state of the robot arm in the virtual environment, discover potential problems in time and make adjustments.

[0072] S5.1、Through the geometric parameters and kinematic model of the robot arm, the basic structure of its digital twin is established, and the specific geometric parameters of the robot arm are obtained, including link length, joint type, joint limit angle and working range;

[0073] S5.2, describe the pose from the base to the end effector using forward kinematics, define the relative position of each link and joint based on DH parameters, establish the motion model of the robot arm, and calculate the required joint angles for a given end effector position through inverse kinematics;

[0074] S5.3, create a digital twin model consistent with the physical robot arm in the virtual environment, including geometric structure and motion model, choose Unity industrial simulation software, create a 3D geometric model of the robot arm using SolidWorks, and import it into the unity simulation platform;

[0075] S5.4, map the real-time motion of the physical robot arm to the digital twin model, realize the synchronization of virtual and real, through the joint angle data obtained by the machine vision system in real time, transfer these angle data to the robot arm joint control system in the virtual environment, update the posture of the virtual model.

[0076] Once the action recognition and digital twin model are completed, the system enters the action-driven and control phase. In this phase, the system dynamically adjusts the motion instructions of the robot arm according to the state of the digital twin model. Through real-time data analysis of the robot arm, the system can identify and handle abnormal situations, ensuring that the robot arm can still operate stably in complex environments.

[0077] The control strategy of the present application adopts a closed-loop feedback mechanism, so that the robot arm can adjust itself according to the actual running situation. For example, when an obstacle is identified, the system can immediately change the motion path of the robot arm to avoid collision. At the same time, the system can also continuously optimize the operation efficiency of the robot arm through real-time feedback, realize more refined control, specific steps:

[0078] Step one: generate accurate motion instructions according to the state of the digital twin model and the action recognition results, to ensure that the motion of the robot arm meets the expectations;

[0079] Step two: the system can monitor the running state of the robot arm in real time, and dynamically adjust the motion strategy according to the environmental changes and task requirements;

[0080] Step three: when an obstacle or abnormal situation is identified, the system can quickly respond, adjust the path or change the operation strategy to avoid collision and ensure safety.

[0081] To improve the intelligent level of the system, the self-learning and optimization mechanism is also introduced. By collecting and analyzing historical data, the system can use machine learning algorithms to continuously optimize the action recognition and control strategy. This process not only includes fine adjustment of existing actions, but also adaptive learning in new environments, thereby continuously improving overall performance. For example, the system can identify which action combinations are more efficient and which environmental factors have a greater impact on operation accuracy based on past operation data, and automatically adjust the strategy in subsequent tasks. This self-learning ability enables the robotic arm to perform more flexibly and efficiently in complex tasks, adapting to different working conditions and task requirements. The specific steps are as follows:

[0082] Step one: The system collects historical data in each operation, including success rate, efficiency and error type of action execution, etc.

[0083] Step two: By analyzing historical data, use machine learning algorithms to continuously optimize action recognition models and control strategies to improve overall system performance.

[0084] Step three: Realize the feedback from the actual operation, form a closed loop of continuous learning, and improve the adaptability in new environment.

[0085] By combining machine vision and digital twin technology, this invention provides an innovative solution to improve the operation accuracy and intelligent level of the robotic arm in complex environments. With the continuous development of industrial automation technology, this invention will provide important theoretical support and practical guidance for future intelligent manufacturing and smart factory construction.

Claims

1. A robotic arm motion recognition and control method based on machine vision and digital twin technology, characterized in that: The steps include: S1. Use an industrial camera to capture motion images or videos of the robotic arm, process the image or video data using a deep learning algorithm, and extract key motion features. S2. Use image annotation tools to accurately annotate the joint points of the segmented image and generate a standardized annotation file; S3. Divide the annotated files and their corresponding image data into training sets and validation sets according to the proportions, and train the object detection model; S4. Use the trained object detection model to perform motion recognition on the robotic arm, connect the annotated joints to the skeleton, accurately calculate the angles between the skeletons during the robotic arm's motion, and dynamically visualize them in the image. S5. Based on the geometric structure and motion model of the robotic arm, a digital twin model consistent with the physical robotic arm is established in the virtual environment to control the robotic arm. The S4 includes: S4.

1. Use the trained object detection model to detect the positions of the joints in the robot arm's motion. Based on the joint detection results, construct the robot arm's skeleton and dynamically connect the skeleton. In each frame, connect adjacent joints with line segments based on the detected joint coordinates to form a skeleton structure. S4.

2. Dynamically visualize the skeleton and joint angles of the robotic arm during motion in the image. Calculated angle values ​​are annotated on the lines connecting the joints in real time. Joint and skeleton information is updated in each frame to ensure that the entire robotic arm motion process can be dynamically tracked and displayed. The robot arm joint angle calculation includes: Step 1: Extract the center coordinates (x i ,y i ), where i represents different joint numbers; Step 2: Calculate the two-dimensional angle: Assume that in the two-dimensional image plane, the coordinates of the two adjacent joints of the robot are (x i ,y i ) and (x i+1 ,y i+1 ), the rotation angle θ between joint i and joint i+1 i Use the inverse tangent function to calculate: θ i =tan -1 2(and i+1 -and i ,x i+1 -x i ) Among them, tan -1 2 is the inverse tangent function, which is used to process angles ranging from -π to π in a two-dimensional plane; Step 3: Angle calculation in three-dimensional space: Using 3D position information, use the vector method to calculate the angle of the joint. Assume that the coordinates of joints i and i+1 in 3D space are (c i ,y i ,z i ) and (x i+1 ,y i+1 ,z i+1 ), the angle calculation between joints is based on the vector dot product formula: v i =(x i+1 -x i ,y i+1 -y i ,z i+1 -z i ) Calculate the angle θ using the dot product formula i : Among them, v i and v i+1 It is the vector between adjacent joints, and the result of the dot product calculation is used to get the angle through the arc cosine function; The S5 includes: S5.

1. Build the basic structure of the digital twin of the robotic arm using its geometric parameters and kinematic model to obtain the specific geometric parameters of the robotic arm, including link length, joint type, joint limit angle, and working range. S5.

2. Use forward kinematics to describe the pose from the base to the end effector. Define the relative position of each link and joint based on DH parameters, establish the kinematic model of the manipulator, and calculate the joint angles required for a given end effector position using inverse kinematics. S5.

3. Create a digital twin model of the physical robotic arm in a virtual environment, including the geometric structure and motion model. Use Unity industrial simulation software to create a 3D geometric model of the robotic arm using SolidWorks and import it into the Unity simulation platform. S5.

4. Map the real-time movements of the physical robotic arm to the digital twin model to achieve synchronization between virtual and reality. The joint angles obtained in real time by the machine vision system are transmitted to the robotic arm joint control system in the virtual environment to update the posture of the virtual model.

2. The method for robot arm motion recognition and control based on machine vision and digital twin technology according to claim 1, characterized in that: Said S1 comprises: S1.

1. Arrange the position of the industrial camera according to the movement area of ​​the robotic arm, adjust the camera resolution and frame rate according to the size and movement trajectory of the robotic arm, and use the industrial camera to capture the movement process of the robotic arm to generate a continuous image frame sequence or video data; S1.

2. Decoding the original video data to convert it into a temporally continuous sequence of high-quality image frames; S1.

3. Preprocess each extracted image frame to meet the input requirements of the object detection model; S1.

4. Use the yolov8 target detection algorithm to perform real-time analysis on the captured images to identify the key parts of the robotic arm and its current motion status.

Images