A robotic arm control method, device and storage medium based on image recognition

Abstract

The invention discloses a robot arm control method based on image recognition, which relates to the field of robot control technology. Firstly, the states of a palm and an arm are captured respectively, and a position signal is identified and sent in the form of a signal instruction, which is transmitted to a controller of the robot arm via a serial wireless communication module. The controller parses the instruction and compares the identified target position signal with the current target position instruction stored in the controller to obtain a deviation position signal. The deviation signal generates a pulse of a driving motor, and then controls the motor of the palm and the arm to move, so that the position of the robot arm follows the palm and the arm, so that human-machine interaction is convenient and fast, which conforms to the concept of human-machine control, and solves the problem of complex instructions in complex robot control. In the image recognition process, only one frame of image data is needed to infer the positions of the palm and the arm. The palm and the arm are identified separately, and instructions are transmitted respectively, so as to increase the amount of data transmitted at one time and improve the dynamic response of the control system.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot arm control method, device and storage medium based on image recognition. Background Art

[0002] The control accuracy of the robot's position depends to a large extent on whether the position signal it feeds back is accurate. The position sensors currently used in robot control can be divided into contact and proximity position sensors. Common ones include potentiometers, photoelectric position sensors, magnetic induction sensors, and capacitive position sensors. They have the disadvantages of being easily affected by environmental changes in measurement accuracy, having a small measuring range, and being inconvenient to install, which greatly limits the robot's working scenarios.

[0003] On the other hand, the more common robot control solutions nowadays are mostly in the form of sending commands via keyboard and mouse. This method has met the requirements of machine control and is used skillfully by people. In more complex robot controls, the commands are more complicated and difficult for people to understand. To a certain extent, this control method is not very consistent with people's operating habits. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above-mentioned background technology, the purpose of the present invention is to provide a robot arm control method, device and storage medium based on image recognition, which uses image recognition to replace the traditional keyboard and mouse to send instructions, so that the robot arm follows human gestures to move and operate, which is more in line with people's daily usage habits and is more convenient to operate, solving the problem of complex instructions in complex robot control; replacing the traditional position sensor with contactless position recognition, the feedback position parameters will not be affected by environmental changes, and at the same time solves the problem of the inconvenience of installing the position sensor.

[0005] The object of the present invention can be achieved by the following technical solution: a method for controlling a robotic arm based on image recognition, the method comprising the following steps:

[0006] The palm and arm gestures made by a person in front of the camera are captured by the camera, and the gesture content is stored as image data by the camera. The robotic arm realizes following movement according to the image data, the robotic arm follows the movement of the human arm, and the robotic palm follows the movement of the human palm. The image data processing of the palm and arm is realized by the detector-tracker machine learning pipeline. The detector machine learning pipeline first locates the posture area of ​​the palm and arm in one frame of image data, and returns the palm and arm detection frame. The tracker machine learning pipeline infers the high-fidelity three-dimensional key point position coordinates of the hand and arm from a single frame according to the posture area of ​​the palm and arm. The key point coordinates are filtered out by a filter to remove the coordinate jitter caused by the machine learning algorithm. The bending angle information of each joint of the palm and arm is calculated by the space vector from the filtered three-dimensional position coordinates, and converted into the position instructions of the palm and arm including the specific joint serial number and bending degree;

[0007] The palm and arm position instructions are sent to the palm and arm controllers which are also equipped with serial wireless communication modules at a certain baud rate through the serial wireless communication module to become the palm serial module and the arm serial module;

[0008] After receiving the palm and arm position instructions, the controller parses the palm and arm position instructions into position signals, compares the recognized position signals with the palm and arm positions stored in the controller, obtains the difference position signals, and converts the difference position signals into control pulses of the palm and arm motors, thereby changing the position of the robotic arm;

[0009] Repeat the above steps until the difference signal is zero, so as to achieve error-free tracking of the robot arm position and the human palm and arm posture.

[0010] Preferably, the palm and arm positions are captured using the same camera, and the palm and arm positions are identified separately through a machine learning pipeline and sent to corresponding controllers respectively.

[0011] Preferably, the communication mode of the wireless communication module is serial communication, using an HC-12 wireless module.

[0012] Preferably, the baud rates of the palm serial port module and the arm serial port module are 9600 and 115200 respectively.

[0013] Preferably, the palm and arm positions stored in the controller are theoretical position signals under the premise that the motor does not lose step.

[0014] Preferably, the coordinate filtering method of the key point is an improved limiting filtering algorithm, which dynamically adjusts the coordinate range by detecting whether it is within the filtering threshold range, thereby ensuring that the result after each filtering is the optimal coordinate point.

[0015] Preferably, the bending angle of each joint is obtained by calculating the space vector through the coordinates of the key points. Assume there are three points in the space: A = (xa, ya, za), B = (xb, yb, zb), C = (xc, yc, zc), The bending angle of the joint can be obtained by calculating the angle between the two vectors. The angle calculation is as follows:

[0016]

[0017] Preferably, the format of the converted position instruction is <1!090!1000>, where the first digit "1" represents the joint number, the number of which is determined by the structure of the robotic arm, "090" represents the target bending angle of the corresponding joint number, in degrees, and "1000" represents the time required for the joint to rotate to the target bending angle, in milliseconds. The numbers are separated by "!" and enclosed in "<" and ">".

[0018] Preferably, a device comprises:

[0019] one or more processors;

[0020] A memory for storing one or more programs;

[0021] When one or more of the programs are executed by one or more of the processors, the one or more processors implement the robotic arm control method based on image recognition as described above.

[0022] Preferably, a storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the above-mentioned image recognition-based robotic arm control method when executed by a computer processor.

[0023] Beneficial effects of the present invention:

[0024] 1. The present invention replaces the traditional keyboard and mouse command sending process, making human-computer interaction more convenient and quick, more in line with people's concept of machine control, and at the same time solving the problem of complex instructions in complex robot control.

[0025] 2. In the process of image recognition, the present invention only needs one frame to infer the position of the palm and arm. The palm and arm are identified separately and instructions are transmitted separately, thereby increasing the amount of data transmitted in one time and improving the dynamic response of the control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a schematic diagram of the control structure of the present invention;

[0028] Figure 2 It is a schematic diagram of palm key point recognition of the present invention;

[0029] Figure 3 It is a schematic diagram of the bending angle of the arm joint of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, a robotic arm control method based on image recognition comprises the following steps:

[0032] 1. The robotic arm structure includes a six-degree-of-freedom robotic arm and a bionic robotic palm installed on the top of the robotic arm. The camera captures the palm and arm gestures made by a person in front of the camera, and the gesture content is stored as image data through the camera. The robotic arm realizes following movement based on the captured palm and arm image data. The robotic arm follows the movement of the human arm, and the robotic palm follows the movement of the human palm. The image data processing of the palm and arm is implemented through the detector-tracker machine learning pipeline. The detector pipeline first locates the posture area of ​​the palm and arm in the frame and returns the palm and arm detection frame. The tracker pipeline further infers the high-fidelity three-dimensional key point position coordinates of the hand and arm from a single frame based on this posture area. The key point coordinates are filtered out by a filter to remove the coordinate jitter caused by the machine learning algorithm. The palm key points are shown in the image data. Figure 2 , where the bottom point 0 represents the wrist node, and the remaining four points form a finger model. Connect the points in sequence to get the key point coordinates, and the arm bending angle is shown in Figure 1. Figure 3, points 1, 2, 3, and 4 form the shape of the upper arm, points 5, 6, 7, and 8 form the shape of the lower arm, 1 and 3 represent the shoulder joint, 2 and 4 represent the elbow joint, θ1 is displayed as the bending angle of the elbow joint, and θ2 is displayed as the bending angle of the shoulder joint. The bending angle information of each joint of the palm and arm is calculated by the spatial vector from the filtered three-dimensional key point coordinates, and converted into the position instructions of the palm and arm including the specific joint number and bending degree.

[0033] 2. After obtaining the position command, the palm and arm position command is sent to the palm and arm controller which is also equipped with the serial wireless communication module at a certain baud rate through the serial wireless communication module.

[0034] 3. After receiving the command, the controller will parse the command into a position signal, and compare the recognized position command with the current position command stored in the controller to obtain the difference position command. The current palm and arm position stored in the controller is the ideal position of the palm and arm where the motor does not lose step, and the size of the difference signal determines the width and number of the control pulses of the palm and arm motors. The control pulses are amplified by the motor driver to generate a drive signal to make the motor rotate, and the motor drives the bionic robot arm to change its position. Once there is a difference signal, the robot arm will repeat the above process until the difference signal is 0, so that the robot arm position can follow the human palm and arm posture without any difference.

[0035] It needs to be further explained that, in the specific implementation process, the position capture of the palm and the arm is performed using the same camera, while the identification of the key points of the palm and the arm through the machine learning pipeline is performed separately and sent to the corresponding controllers respectively.

[0036] The communication mode of the wireless communication module is serial communication, using the HC-12 wireless module.

[0037] The baud rates of the palm serial port module and the arm serial port module are 9600 and 115200 respectively.

[0038] It needs to be further explained that, in the specific implementation process, the baud rates of the palm serial port module and the arm serial port module are different in order to prevent confusion in command transmission.

[0039] The palm and arm positions stored in the controller are theoretical position signals under the premise that the motor does not lose step.

[0040] It should be further explained that, in the specific implementation process, the coordinate filtering method of the key point is an improved limiting filtering algorithm, which dynamically adjusts the coordinates by detecting whether they are within the filtering threshold range to ensure that the result after each filtering is the optimal coordinate point.

[0041] It should be further explained that, in the specific implementation process, the bending angles of the joints are obtained by calculating the space vectors through the coordinates of the key points. Assume that there are three points in the space: A = (xa, ya, za), V = (xb, yb, zb), C = (xc, yc, zc), The bending angle of the joint can be obtained by calculating the angle between the two vectors. The angle calculation is as follows:

[0042]

[0043] It needs to be further explained that, in the specific implementation process, the conversion into the position instruction format is <1!090!1000>, where the first number "1" represents the joint number, the number of numbers is determined by the structure of the robotic arm, "090" represents the target bending angle of the corresponding joint number, in degrees, and "1000" represents the time required for the joint to rotate to the target bending angle, in milliseconds. The numbers are separated by "!" and enclosed in "<" and ">".

[0044] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0045] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0046] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0047] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A robotic arm control method based on image recognition, characterized in that: The method comprises the following steps: The palm and arm gestures made by a person in front of the camera are captured by the camera, and the gesture content is stored as image data by the camera. The robotic arm realizes following movement according to the image data, the robotic arm follows the movement of the human arm, and the robotic palm follows the movement of the human palm. The image data processing of the palm and arm is realized by the detector-tracker machine learning pipeline. The detector machine learning pipeline first locates the posture area of ​​the palm and arm in one frame of image data, and returns the palm and arm detection frame. The tracker machine learning pipeline infers the high-fidelity three-dimensional key point position coordinates of the hand and arm from a single frame according to the posture area of ​​the palm and arm. The key point coordinates are filtered out by a filter to remove the coordinate jitter caused by the machine learning algorithm. The bending angle information of each joint of the palm and arm is calculated by the space vector from the filtered three-dimensional position coordinates, and converted into the position instructions of the palm and arm including the specific joint serial number and bending degree; The palm and arm position instructions are sent to the palm and arm controllers which are also equipped with serial wireless communication modules at a certain baud rate by generating palm serial port modules and arm serial port modules through the serial wireless communication module; After receiving the palm and arm position instructions, the controller parses the palm and arm position instructions into position signals, compares the recognized position signals with the palm and arm positions stored in the controller, obtains the difference position signals, and converts the difference position signals into control pulses of the palm and arm motors, thereby changing the position of the robotic arm; Repeat the above steps until the difference signal is zero, so as to achieve error-free tracking of the robot arm position and the human palm and arm posture.

2. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The palm and arm position capture is performed using the same camera, and the palm and arm positions are identified separately through a machine learning pipeline and sent to corresponding controllers respectively.

3. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The communication mode of the wireless communication module is serial communication, using the HC-12 wireless module.

4. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The baud rates of the palm serial port module and the arm serial port module are 9600 and 115200 respectively.

5. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The palm and arm positions stored in the controller are theoretical position signals under the premise that the motor does not lose step.

6. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The coordinate filtering method of the key point is an improved limiting filtering algorithm, which dynamically adjusts the coordinate range by detecting whether it is within the filtering threshold range to ensure that the result after each filtering is the optimal coordinate point.

7. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The bending angle information of each joint is obtained by calculating the space vector through the coordinates of the key points. Assume that there are three points in the space: A = (xa, ya, za), B = (xb, yb, zb), and C = (xc, yc, zc). The bending angle information of the joint can be obtained by calculating the angle between the two vectors. The angle calculation is as follows:

8. The method for controlling a robotic arm based on image recognition according to claim 1, characterized in that: The format of the position command is <1!090!1000>, where the first number "1" represents the joint number. The number of numbers is determined by the structure of the robot arm. "090" represents the target bending angle of the corresponding joint number in degrees. "1000" represents the time required for the joint to rotate to the target bending angle in milliseconds. The numbers are separated by "!" and enclosed in "<" ">".

9. A device, characterized in that: include: one or more processors; A memory for storing one or more programs; When one or more of the programs are executed by one or more of the processors, the one or more processors implement a robotic arm control method based on image recognition as described in any one of claims 1-8.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute a robotic arm control method based on image recognition as described in any one of claims 1 to 8 when executed by a computer processor.

Images