An intelligent mechanical arm control system and control method based on machine vision

Abstract

The application provides a kind of intelligent mechanical arm control system and control method based on machine vision, it is related to mechanical arm control technical field, control system includes: mechanical arm, host computer, lower computer and signal receiver;The host computer includes image processing module, action recognition module and inverse kinematics calculation module;The image processing module identifies real-time hand coordinate by hand key point position, the action recognition module identifies hand action according to real-time hand coordinate, the inverse kinematics calculation module generates mechanical arm moving instruction according to hand action and sends to signal receiver;The signal receiver receives mechanical arm moving instruction and sends to lower computer;The lower computer controls mechanical arm to carry out corresponding action according to received mechanical arm moving instruction.The application realizes the stable interaction of operator and mechanical arm by using machine vision technology, guarantees the simplicity, accuracy, immediacy of mechanical arm operation.

Description

Technical Field

[0001] The present invention relates to the field of robotic arm control technology, and more specifically, to a machine vision-based intelligent robotic arm control system and control method. Background Art

[0002] In contemporary society, robotic arms can complete production processes such as welding, handling, packaging, inspection, gluing, polishing and grinding, and are widely used in industries such as automobiles, energy, food, new retail, 3C electronics, precision manufacturing, medical equipment, and education.

[0003] The current situation corresponding to the widespread application of robotic arms is that they are extremely difficult to use. Traditional robotic arm control methods require the operator to have professional knowledge of robotic arm control methods. Currently, the robotic arm simulation control methods on the market that aim to improve the difficulty of operation all use machine vision recognition of the operation target or the implementation of human hand simulation operation hardware to make the robotic arm move. The main disadvantage of the former is that the application scope is too narrow and too specific, and the main disadvantage of the latter is that it requires the purchase of additional operating hardware, which increases the cost and reduces the portability of the robotic arm deployment. In summary, it is necessary to invent an intelligent robotic arm control method that is easy to use, flexible in movement, and can be based on common devices such as mobile phones. Summary of the Invention

[0004] In response to the aforementioned limitations of reverse control and the flexibility and cost of simulated control, a machine vision-based intelligent robotic arm control system and control method are provided. This invention primarily utilizes machine vision recognition of human hands, enabling forward control of the robotic arm from the hand without the need for additional operating equipment.

[0005] The technical means adopted in the present invention are as follows:

[0006] An intelligent robotic arm control system based on machine vision, comprising: a robotic arm, a host computer, a slave computer and a signal receiver;

[0007] The host computer includes an image processing module, a motion recognition module and an inverse kinematics calculation module; the image processing module identifies the key point positions of the human hand to obtain real-time hand coordinates, the motion recognition module identifies the hand motion according to the real-time hand coordinates, and the inverse kinematics calculation module generates a robot arm movement instruction according to the hand motion and sends it to a signal receiver;

[0008] The signal receiver receives the robot arm movement instruction and sends it to the lower computer;

[0009] The lower computer controls the robotic arm to perform corresponding actions according to the received robotic arm movement instructions.

[0010] Furthermore, the image processing module recognizes the human hand image acquired by the camera, and recognizes the positions of key points of the human hand to obtain real-time hand coordinates.

[0011] The present invention also provides a method for controlling an intelligent robotic arm based on machine vision, which is implemented based on any of the above-mentioned intelligent robotic arm control systems based on machine vision, and includes the following steps:

[0012] The host computer uses a camera to obtain hand movement images;

[0013] Constructing a machine learning model, and training the machine learning model to obtain a trained machine learning model;

[0014] The trained machine learning model is used to calculate the hand motion image to obtain a three-dimensional hand space model;

[0015] Converting the hand space three-dimensional model into key point coordinates of all hand joints;

[0016] Perform hand motion recognition calculations based on the key point coordinates to obtain specific motion joint coordinates of the robotic arm;

[0017] Perform inverse kinematics calculation on the specific motion joint coordinates to obtain motion data of all servos of the robotic arm;

[0018] Transmitting the manipulator arm servo control instructions to the lower computer;

[0019] The lower computer controls the manipulator to complete the specified action according to the manipulator servo control instruction.

[0020] Furthermore, performing hand motion recognition calculation according to the key point coordinates specifically includes:

[0021] Process the hand motion images in terms of resolution, flipping and aspect ratio;

[0022] Execute a TensorFlowLite model on the host computer GPU to obtain the processed hand motion image and output the hand landmark model tensor;

[0023] Calculate the detection confidence based on the output hand landmark model tensor;

[0024] Determine whether the current hand exists based on the confidence level;

[0025] Feedback is provided based on the number of hands currently detected and the set maximum number to prevent redundant detections;

[0026] Calculate the binary hand dominance score based on the output hand landmark model tensor;

[0027] The output hand landmark model tensor is decoded into a list of 21 key point coordinates of all hand joints, and the coordinates are normalized according to the input image size.

[0028] Furthermore, it also includes:

[0029] If the hand exists, the normalized hand coordinates are processed according to the image size before processing and converted into drawing primitives;

[0030] If the hand exists, calculate the hand's bounding box based on the hand coordinates and convert it into a drawing primitive;

[0031] The obtained drawing primitives are drawn into an annotation overlay and placed on top of the original image before output.

[0032] Furthermore, the building of the machine learning model and training of the machine learning model include the following steps:

[0033] For hands without high-contrast recognition points, first train the palm and fist detectors;

[0034] Use square bounding box anchors in the palm / fist detector to model the palm / fist, ignoring the aspect ratio and reducing the total number of anchors;

[0035] Annotate various real hand or rendered hand images and use them as training set input to train the model.

[0036] Furthermore, the specific motion joint coordinates are subjected to inverse kinematics calculation, comprising the following steps:

[0037] Based on the arm lengths and arm angles of the robot, list the equations for solving X, Y, and Z in a forward direction;

[0038] Iterate over all possible angles of servo No. 1 and find all possible solutions;

[0039] After solving the possible angles of the No. 1 servo, the No. 1 servo will solve the angles of the remaining servos in turn.

[0040] The output accuracy meets the required solution.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] Compared with existing robotic arm patents on the market that use machine vision, the robotic arm described in this patent does not move the robotic arm through machine vision recognition of the operation target, but rather uses machine vision to recognize and restore human hand movements. It is much more flexible and broad in scope of application, and has human adaptability that the former does not have. The present invention uses machine vision technology to achieve stable interaction between the operator and the robotic arm, ensuring the simplicity, accuracy, and immediacy of robotic arm operation. Through the accumulation of pictures and autonomous learning within a certain period, the system can independently, accurately, and quickly judge the operation posture. It integrates mobile phones and other devices to ensure accurate, precise and stable operation while realizing remote network operation. It can operate instantly at a distance, and the operation data can be accessed and exported at any time, which can provide satisfactory results for both supply and demand parties, reduce training time, and thus reduce the impact of training personnel on the production rhythm. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 This is the system control block diagram of the present invention.

[0045] Figure 2 The annotation overlay of the present invention is placed on the top layer of the original image and then output. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] Unless otherwise specifically stated, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The techniques, methods and equipment known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0050] In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention: the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0051] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below their position devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0052] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0053] The present invention provides an intelligent manipulator control system and control method based on machine vision, wherein the control system comprises: a manipulator, a host computer, a slave computer and a signal receiver;

[0054] The host computer includes an image processing module, a motion recognition module and an inverse kinematics calculation module; the image processing module identifies the position of key points of the human hand to obtain real-time hand coordinates, the motion recognition module identifies hand movements based on the real-time hand coordinates, and the inverse kinematics calculation module generates a robotic arm movement instruction based on the hand movement and sends it to a signal receiver; the image processing module identifies the human hand image obtained by the camera, and identifies the position of key points of the human hand to obtain real-time hand coordinates.

[0055] The signal receiver receives the robot arm movement instruction and sends it to the lower computer;

[0056] The lower computer controls the robotic arm to perform corresponding actions according to the received robotic arm movement instructions.

[0057] Furthermore, the present invention is particularly applicable to situations where the host computer is specifically a smart terminal with an application page open. The host computer is specifically a smart terminal with an application page open, the image processing module of the host computer recognizes images captured by a camera on the smart terminal, the motion recognition module generates hand motion information based on the real-time coordinate information of the image processing module, the inverse kinematics calculation module automatically generates motion instructions for the manipulator servo at each moment based on the motion information, and transmits the motion instructions for the manipulator servo at each moment to the slave computer. The slave computer controls the manipulator to move to the specified position coordinates at each moment based on the position coordinate information of the end of the manipulator at each moment, thereby realizing real-time control of the manipulator by the hand.

[0058] like Figure 1 As shown, the present invention also provides a method for controlling an intelligent robotic arm based on machine vision, which is implemented based on the above-mentioned intelligent robotic arm control system based on machine vision, and includes the following steps:

[0059] The host computer uses a camera to obtain hand movement images;

[0060] Constructing a machine learning model, and training the machine learning model to obtain a trained machine learning model;

[0061] Constructing a machine learning model and training the machine learning model include the following steps:

[0062] For hands without high-contrast recognition points, first train the palm and fist detectors;

[0063] Use square bounding box anchors in the palm / fist detector to model the palm / fist, ignoring the aspect ratio and reducing the total number of anchors;

[0064] Annotate various real hand or rendered hand images and use them as training set input to train the model.

[0065] The trained machine learning model is used to calculate the hand motion image to obtain a three-dimensional hand space model;

[0066] Converting the hand space three-dimensional model into key point coordinates of all hand joints;

[0067] Perform hand motion recognition calculations based on the key point coordinates to obtain specific motion joint coordinates of the robotic arm;

[0068] Performing hand motion recognition operations according to the key point coordinates, specifically including:

[0069] Process the hand motion images in terms of resolution, flipping and aspect ratio;

[0070] Execute a TensorFlowLite model on the host computer GPU to obtain the processed hand motion image and output the hand landmark model tensor;

[0071] Calculate the detection confidence based on the output hand landmark model tensor;

[0072] Determine whether the current hand exists based on the confidence level;

[0073] Feedback is provided based on the number of hands currently detected and the set maximum number to prevent redundant detections;

[0074] Calculate the binary hand dominance score based on the output hand landmark model tensor;

[0075] The output hand landmark model tensor is decoded into a list of 21 key point coordinates of all hand joints, and the coordinates are normalized according to the input image size.

[0076] If the hand exists, the normalized hand coordinates are processed according to the image size before processing and converted into drawing primitives;

[0077] If the hand exists, calculate the hand's bounding box based on the hand coordinates and convert it into a drawing primitive;

[0078] The obtained drawing primitives are drawn into an annotation overlay and placed on top of the original image before output. The output image is as follows Figure 2 shown.

[0079] The specific motion joint coordinates are subjected to inverse kinematics calculation to obtain the motion data of all servos of the robotic arm. Taking a four-degree-of-freedom robotic arm as an example, the process includes the following steps:

[0080] Based on the arm lengths and arm angles of the robot, list the equations for solving X, Y, and Z in a forward direction;

[0081] Iterate over all possible angles of servo No. 1 and find all possible solutions;

[0082] After solving the possible angles of the No. 1 servo, the No. 1 servo will solve the angles of the remaining servos in turn.

[0083] The output accuracy meets the required solution.

[0084] Transmitting the manipulator arm servo control instructions to the lower computer;

[0085] The lower computer controls the manipulator to complete the specified action according to the manipulator servo control instruction.

[0086] The system control block diagram of this patent is as follows Figure 1As shown, after the human hand makes a movement and the camera captures the image information, the host computer performs image processing to obtain the coordinates of the key points of the hand. The host computer then continues to perform inverse kinematics calculations to solve the specific motion instructions of all the servos of the robotic arm. The signal is then transmitted to the signal receiver via wireless technology, and the signal receiver then transmits it to the connected lower computer. The lower computer then operates the robotic arm and ultimately faithfully reproduces the movement commanded by the human hand.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling an intelligent robotic arm based on machine vision, characterized in that: The steps include: The host computer uses a camera to obtain hand movement images; Constructing a machine learning model, and training the machine learning model to obtain a trained machine learning model; For hands without high-contrast recognition points, first train the palm and fist detectors; Use square bounding box anchors in the palm / fist detector to model the palm / fist, ignoring the aspect ratio and reducing the total number of anchors; Annotate various real or rendered hand images and use them as training set inputs for model training; The trained machine learning model is used to calculate the hand motion image to obtain a three-dimensional hand space model; Converting the hand space three-dimensional model into key point coordinates of all hand joints; Perform hand motion recognition calculations based on the key point coordinates to obtain specific motion joint coordinates of the robotic arm; Perform inverse kinematics calculation on the specific motion joint coordinates to obtain the motion data of all servos of the robotic arm and convert them into control instructions that can be directly executed; Inverse kinematics calculation includes the following steps: Based on the arm lengths and arm angles of the robot, list the equations for solving X, Y, and Z in a forward direction; Iterate over all possible angles of servo No. 1 and find all possible solutions; After solving the possible angles of the No. 1 servo, the No. 1 servo will solve the angles of the remaining servos in turn. Output a solution with accuracy that meets the usage requirements; Transmitting the manipulator arm servo control instructions to the lower computer; The lower computer controls the manipulator to complete the specified action according to the manipulator servo control instruction; Performing hand motion recognition operations according to the key point coordinates, specifically including: Process the hand motion images in terms of resolution, flipping and aspect ratio; Execute a TensorFlow Lite model on the host computer GPU to obtain the processed hand motion image and output the hand landmark model tensor; Calculate the detection confidence based on the output hand landmark model tensor; Determine whether the current hand exists based on the confidence level; Feedback is provided based on the number of hands currently detected and the set maximum number to prevent redundant detections; Calculate the binary hand dominance score based on the output hand landmark model tensor; The output hand landmark model tensor is decoded into a list of 21 key point coordinates of all hand joints, and the coordinates are normalized according to the input image size; If the hand exists, the normalized hand coordinates are processed according to the image size before processing and converted into drawing primitives; If the hand exists, calculate the hand's bounding box based on the hand coordinates and convert it into a drawing primitive; The obtained drawing primitives are drawn into an annotation overlay and placed on top of the original image before output.

2. A machine vision-based intelligent robotic arm control system, used to implement the machine vision-based intelligent robotic arm control method according to claim 1, characterized in that: include: Robotic arm, host computer, slave computer and signal receiver; The host computer includes an image processing module, a motion recognition module and an inverse kinematics calculation module; the image processing module identifies the key point positions of the human hand to obtain real-time hand coordinates, the motion recognition module identifies the hand motion according to the real-time hand coordinates, and the inverse kinematics calculation module generates a robot arm movement instruction according to the hand motion and sends it to a signal receiver; The signal receiver receives the robot arm movement instruction and sends it to the lower computer; The lower computer controls the robotic arm to perform corresponding actions according to the received robotic arm movement instructions.

3. The intelligent robotic arm control system based on machine vision according to claim 2, characterized in that: The image processing module identifies the human hand image captured by the camera, and identifies the key point positions of the human hand to obtain real-time hand coordinates.

Images