A method, device and electronic device for controlling visual servo of a robotic arm

Abstract

The present application relates to a method, device, and electronic device for visual servo control of a robotic arm. The method comprises: acquiring a target image of a target to be identified and determining the center point of the target image; identifying the target outline in the target image based on a preset color recognition model; determining the relative position of the target outline and the center point; determining the spatial relative position of the robotic arm end and the target to be identified according to a preset spatial coordinate system; predicting the real-time position of the robotic arm end based on preset control rules, the relative position, and the spatial relative position; and outputting control instructions based on the real-time position of the robotic arm end and D-H inverse kinematics. The present application has the effect of improving stability during teleoperation.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a method, device and electronic equipment for visual servo control of a robotic arm. Background Art

[0002] During teleoperation, the robot accesses the communication system through a wireless network. There are sudden communication delays and uncertainties in the transmission of data and instructions during the communication process, which will have an adverse impact on the operation stability during teleoperation.

[0003] In addition, when a robotic arm is required to operate multiple switch buttons on the operating panel of an electrical cabinet, due to the small size of the switch buttons relative to the robotic arm and the instability and proficiency of manual operation, relying solely on manual visual operation to control the robotic arm to reach the specified position is inefficient and the accuracy of the positioning results cannot be guaranteed. Summary of the Invention

[0004] In order to improve the stability during teleoperation, the present application provides a robotic arm visual servo control method, device and electronic equipment.

[0005] In a first aspect, the present application provides a method for visual servo control of a robotic arm, which adopts the following technical solution:

[0006] A visual servo control method for a robotic arm, comprising:

[0007] Acquire a target image of the target to be identified, and determine the center point of the target image;

[0008] Recognizing a target contour in the target image based on a preset color recognition model;

[0009] Determining the relative position of the target contour and the center point;

[0010] Determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system;

[0011] Predicting the real-time position of the end of the robotic arm according to a preset control rule, the relative position, and the spatial relative position;

[0012] Output control instructions based on the real-time position of the end of the robot arm and DH inverse kinematics.

[0013] By adopting the above technical solution, when operating an electrical cabinet device via a robotic arm, the end of the robotic arm is first controlled to move to a preset position. At this time, a target image of the target to be identified is obtained, and the center point of the target image is determined. After obtaining the target image, the target outline in the target image is identified using a preset color recognition model. After determining the target outline, the relative position of the target outline and the center point of the target image is determined. Then, based on a preset spatial coordinate system, the spatial relative position of the end of the robotic arm and the target to be identified is determined. Then, based on the relative position, the movement direction of the end of the robotic arm is determined. Then, based on the preset control rules, the end of the robotic arm is controlled to move a preset distance each time. Then, based on the distance and direction of the movement, the actual position of the end of the robotic arm after the movement is predicted in real time. Based on the real-time position of the end of the robotic arm and DH inverse kinematics, control instructions are output to control the movements of other joints of the robotic arm to drive the end of the robotic arm to a specified position. Using the above solution, after determining the movement direction of the robotic arm, the actual position of the end of the robotic arm after the movement is predicted in combination with the preset control rules. Then, the movement mode of other joints of the robotic arm is analyzed to control the movement of the robotic arm until the end reaches the target position to be identified. Using the above method, precise control of each joint of the robotic arm is achieved, improving the stability during teleoperation.

[0014] Optionally, acquiring a target image of the area to be identified and determining a center point of the target image specifically includes:

[0015] Obtain an image captured and uploaded by a camera installed at the end of the robotic arm at a preset position, which is the target image;

[0016] An intersection point of the diagonal lines of the target image is determined, where the intersection point is the center point of the target image.

[0017] Optionally, the method for identifying the target contour in the target image based on a preset color recognition model specifically includes:

[0018] Get the setting instructions and set the threshold ranges of hue, saturation and brightness respectively;

[0019] The background area of the target image is filtered by a preset HSV model to determine the target outline. The background area is all areas in the target image except the target to be identified.

[0020] Optionally, determining the relative position of the target contour and the center point includes:

[0021] A rectangular coordinate system is established with the center point of the target image as the origin, the width direction perpendicular to the target image as the X axis, and the length direction perpendicular to the target image as the Y axis;

[0022] Determine the coordinates of the center point of the target outline and the coordinates of the center point of the target image;

[0023] The relative position of the target outline and the center point is determined according to the coordinates of the center point of the target outline and the coordinates of the center point of the target image.

[0024] Optionally, determining the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system includes:

[0025] Obtaining a selected instruction, and determining a preset spatial coordinate system according to the selected instruction;

[0026] Determine the three-dimensional coordinates of the end of the robotic arm and the target to be identified in the preset spatial coordinate system respectively;

[0027] The spatial relative position of the end of the manipulator and the target to be identified is determined according to the three-dimensional coordinates of the end of the manipulator and the target to be identified.

[0028] Optionally, the preset control rule is:

[0029] The spatial movement amplitude of each servo adjustment is 1 cm.

[0030] In a second aspect, the present application provides a robotic arm visual servo control device, which adopts the following technical solution:

[0031] A visual servo control device for a robotic arm, comprising:

[0032] An acquisition module is used to acquire a target image of a target to be identified and determine a center point of the target image;

[0033] A recognition module, configured to recognize a target contour in the target image based on a preset color recognition model;

[0034] A first position determination module, configured to determine the relative position of the target outline and the center point;

[0035] A second position determination module is used to determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system;

[0036] a third position determination module, configured to predict the real-time position of the end of the robotic arm according to a preset control rule, the relative position, and the spatial relative position;

[0037] The control module is used to output control instructions based on the real-time position of the end of the robot arm and DH inverse kinematics.

[0038] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0039] An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a user profile-based account security monitoring method.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0041] A computer-readable storage medium stores a computer program that can be loaded by a processor and executes a user profile-based account security monitoring method.

[0042] In summary, this application has the following beneficial technical effects:

[0043] By identifying the relative position of the target outline and the center point of the target image in the target image, the movement direction of the robotic arm is determined. Then, the actual position of the end of the robotic arm after movement is predicted in combination with the preset control rules. The movement mode of other joints of the robotic wall is analyzed to control the movement of the robotic arm until the end reaches the target position to be identified. The above method is used to achieve precise control of each joint of the robotic arm and improve the stability during remote operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the visual servo control method for a robotic arm provided in this application.

[0045] Figure 2 This is a structural block diagram of the robotic arm visual servo control device provided in this application.

[0046] Figure 3 It is a structural diagram of an electronic device.

[0047] Explanation of the accompanying drawings: 200, robotic arm visual servo control device; 201, acquisition module; 202, recognition module; 203, first position determination module; 204, second position determination module; 205, third position determination module; 206, control module; 301, CPU; 302, ROM; 303, RAM; 304, I / O interface; 305, input part; 306, output part; 307, storage part; 308, communication part; 309, drive; 310, removable medium. DETAILED DESCRIPTION

[0048] The following is combined with Figure 1-3 This application is described in further detail.

[0049] A robotic arm is a spatial mechanism with multiple joints. To describe the spatial position and posture of the arm's end joints or actuators, a spatial coordinate system is established at the revolute joint at the end of each link. The relationship between these coordinate systems is used to describe the position of the end arm. The DH method is generally used to establish the coordinate system and derive the manipulator's equations of motion. The DH method is a matrix method for establishing relative position and posture. It uses homogeneous transformations to describe the spatial geometric relationship of each link relative to a fixed reference coordinate system. A 4×4 homogeneous transformation matrix is used to describe the spatial relationship between two adjacent links. This allows the equivalent homogeneous coordinate transformation matrix of the manipulator's end coordinate system relative to the base coordinate system to be derived, establishing the manipulator's equations of motion.

[0050] The embodiment of the present application discloses a method for visual servo control of a robotic arm. In order to facilitate the execution of the method, a camera is provided at the end of the robotic arm. The position of the camera is consistent with the position of the end of the robotic arm. Figure 1 , the visual servo control method of the robotic arm includes:

[0051] S101: Obtain the preset position of the end of the robotic arm.

[0052] Specifically, when operating multiple switch buttons on the operating panel of the electrical cabinet equipment through a robotic arm, the end of the robotic arm is first controlled to move to a preset position. The preset position is a position near the button to be operated. The preset position can be obtained by obtaining the position information input by the staff when operating the robotic arm.

[0053] S102: Acquire a target image of the target to be identified and determine the center point of the target image.

[0054] Specifically, after the end of the robotic arm is controlled to move to a preset position, the camera end of the camera set at the end of the robotic arm is perpendicular to the plane where the target to be identified is located. At this time, a shooting command is output, the camera is started, and the target image of the target to be identified is obtained. The target image captured by the camera is rectangular as a whole, so the two diagonals of the target image are determined separately, and the intersection of the diagonals is the center point of the target image.

[0055] S103: Recognize the target contour in the target image based on a preset color recognition model.

[0056] Specifically, since the camera has a certain wide angle and the target to be identified is relatively small, the captured target image includes not only the target to be identified but also other backgrounds. It is necessary to identify the target contour of the target to be identified in the image. In this embodiment, the target to be identified is a plurality of switch buttons on the operation panel. Usually, the color of the switch buttons is bright and can be significantly distinguished from the metal shell of the electrical cabinet. Therefore, a preset color recognition model is used to identify the target contour in the target image.

[0057] Furthermore, in this embodiment, the preset color recognition model is the HSV model. The HSV model is a color space, also known as the hexagonal pyramid model. The color parameters in the HSV model are: hue, saturation, and brightness. Hue, measured as an angle, ranges from 0° to 360°. Starting from red and counting counterclockwise, red is 0°, green is 120°, and blue is 240°. Their complementary colors are yellow at 60°, cyan at 180°, and magenta at 300°. Saturation ranges from 0 to 255, with larger values indicating more saturated colors. Brightness ranges from 0 (black) to 255 (white). Therefore, after reading the color information of the target image, different color regions can be distinguished by setting a reasonable threshold range.

[0058] When determining the target contour, first, obtain the setting instructions for the target to be identified. The setting instructions include setting the threshold ranges corresponding to hue, saturation, and brightness in the HSV model. For example, taking the green button of the distribution cabinet as an example, the color of the green switch on the operation panel is recognized and located. In the HSV model, the HSV threshold ranges of green are hue, 78°-99°, saturation (43-255), and V (46-255). After the threshold range is set, the image is processed through the HSV model to filter out the background area. The background area is all areas in the target image except the target to be identified. After filtering out the background area, only the green pixel part remains in the image, completing the extraction of the target contour.

[0059] S104: Determine the relative position of the target contour and the center point.

[0060] Specifically, after determining the center point of the target image, a rectangular coordinate system is established in the target image with the center point of the target image as the origin, the width direction perpendicular to the target image as the X-axis, and the length direction perpendicular to the target image as the Y-axis. After the rectangular coordinate system is established, the coordinates of the target outline and the coordinates of the center point of the target image are determined in the rectangular coordinate system. According to the coordinates of the two points, the relative positions of the target outline and the center point in the target image can be determined. Since the camera is perpendicular to the plane where the target to be identified is located at this time, and the position of the camera is consistent with the position of the end of the robotic arm, after determining the relative position of the center point of the target image and the target outline, it can be equivalent to the relative position of the end of the robotic arm and the target to be identified in the plane perpendicular to the end of the robotic arm. In this embodiment, the coordinates of the target outline are the coordinates of the center point of the target outline, and the target outline is rectangular or circular.

[0061] S105: Determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system.

[0062] Specifically, a selection instruction is obtained. The selection instruction can be manually input by the staff, and the spatial coordinate system is determined according to the selection instruction. In this embodiment, the preset spatial coordinate system is a spatial coordinate system established on the base joint of the robotic arm. The three-dimensional coordinates of the end of the robotic arm and the target to be identified in the spatial coordinate system are determined respectively, so that the spatial relative positions of the end of the robotic arm and the target to be identified can be determined. At the same time, the spatial distance between the end of the robotic arm and the target to be identified can be calculated based on the three-dimensional coordinates of the end of the robotic arm and the target to be identified in the spatial coordinate system.

[0063] S106: Predicting the real-time position of the end of the robotic arm according to the preset control rules, relative position and spatial relative position.

[0064] Specifically, after determining the relative position relationship, the moving direction of the end of the robotic arm can be determined. For example, when it is determined that the target contour is in the third quadrant of the above-mentioned rectangular coordinate system, the target to be identified is determined to be at the lower left of the end of the robotic arm. At this time, the end of the robotic arm needs to move to the left and downward. When it is determined that the target contour is in the first quadrant of the above-mentioned rectangular coordinate system, the target to be identified is determined to be at the upper right of the end of the robotic arm. At this time, the end of the robotic arm needs to move to the right and upward. After determining the moving direction of the end of the robotic arm, the end of the robotic arm is controlled to approach the target to be identified. Each servo adjustment reduces the spatial distance between the end of the robotic arm and the target to be identified by 1 cm. In the above-mentioned three-dimensional space coordinate system, when the end of the robotic arm moves, the coordinates of the end of the robotic arm in the three-dimensional space coordinates, that is, the real-time position of the end of the robotic arm, can be determined in real time. The coordinates of the end of the robotic arm in the three-dimensional space coordinates can be calculated by trigonometric functions. This calculation method is a technical means well known to those skilled in the art and will not be described in detail here.

[0065] S107: Output control instructions based on the real-time position of the end of the robot arm and DH inverse kinematics.

[0066] Specifically, based on the predicted real-time position of the end arm, the DH inverse kinematics principle can be used to calculate the motion of the other joints of the robotic arm, including the distance and direction of movement. Because the DH method is used to establish the coordinate system and derive the motion equations of the robotic arm, once the direction and distance of movement of the end arm are determined, the DH inverse kinematics principle can be used to calculate the motion of the other joints of the robotic arm and output corresponding control instructions to control the robotic arm's movements.

[0067] In this example, the Robotics Toolbox toolbox is used to simulate and solve the inverse kinematics of the robotic arm. The toolbox provides the fkine and ikine functions. The fkine function calculates the robotic arm's pose using the forward solution of the robotic arm's joint angles θ, while the ikine function inversely solves the joint angles θ based on the robotic arm's pose. The DH inverse kinematics principle is well known to those skilled in the art, and the inverse solution process is not detailed here.

[0068] Implementation Principle: During the servo adjustment process of the robotic arm, the robotic arm is first moved to a preset position and the camera is adjusted so that it is perpendicular to the plane of the target to be identified. The camera is then kept perpendicular to the plane of the target to be identified throughout the movement of the robotic arm. During this movement, the target image is captured and the target outline is identified. The direction of the robotic arm's end movement is determined based on the relative position of the target outline and the center point of the target image. During each servo adjustment, the spatial distance is shortened, gradually approaching the target. The spatial movement amplitude of each servo adjustment is set to 1 cm. The coordinates of the robotic arm's end position at the desired position are then added to the existing camera spatial coordinate values according to the movement direction. This is the actual position of the robotic arm's end. The existing camera spatial coordinate values are consistent with the coordinates of the robotic arm's end in the preset spatial coordinate system.

[0069] Through DH inverse kinematics, the joint deflection angle of the robotic arm can be calculated, the end of the robotic arm can be controlled to reach the corresponding position, and a visual servo adjustment can be completed. The above process is repeated until the target contour is adjusted to the center of the image and the spatial distance between the end of the robotic arm and the target to be identified is less than the specified threshold.

[0070] The present application discloses a visual servo control device for a robotic arm, referring to Figure 2 , the robot arm visual servo control device 200 includes:

[0071] The acquisition module 201 is used to acquire a target image of the target to be identified and determine the center point of the target image;

[0072] The recognition module 202 is used to recognize the target contour in the target image based on a preset color recognition model;

[0073] A first position determination module 203 is used to determine the relative position of the target outline and the center point;

[0074] A second position determination module 204 is used to determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system;

[0075] The third position determination module 205 is used to determine the real-time position of the end of the robot arm according to the preset control rules, relative position and spatial relative position;

[0076] The control module 206 is configured to output control instructions based on the real-time position of the end of the robot arm and DH inverse kinematics.

[0077] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0078] The embodiment of the present application discloses an electronic device. Figure 3 The electronic device includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 307 to the random access memory (RAM) 303. Various programs and data required for system operation are also stored in the RAM 303. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus. An input / output (I / O) interface 304 is also connected to the bus.

[0079] The following components are connected to the I / O interface 304: an input section 305 including a keyboard, a mouse, and the like; an output section 306 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 307 including a hard disk; and a communication section 308 including a network interface card such as a LAN card or a modem. The communication section 308 performs communication processing via a network such as the Internet. A drive 309 is also connected to the I / O interface 304 as needed. Removable media 310, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 309 as needed, so that computer programs read therefrom can be installed into the storage section 307 as needed.

[0080] In particular, according to the embodiment of the present application, the above reference flow chart Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 308 and / or installed from a removable medium 310. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the apparatus of the present application are performed.

[0081] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0082] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the aforementioned application concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions applied for in this application.

Claims

1. A visual servo control method for a robotic arm, characterized by: The method includes: Acquire a target image of the target to be identified, and determine the center point of the target image; Recognizing a target contour in the target image based on a preset color recognition model; Determining the relative position of the target contour and the center point; Determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system; Predicting the real-time position of the end of the robotic arm according to a preset control rule, the relative position, and the spatial relative position; Output control instructions based on the real-time position of the end of the robot arm and DH inverse kinematics; Determining the relative position of the target contour and the center point includes: A rectangular coordinate system is established with the center point of the target image as the origin, the width direction perpendicular to the target image as the X axis, and the length direction perpendicular to the target image as the Y axis; Determine the coordinates of the center point of the target outline and the coordinates of the center point of the target image; Determining the relative position of the target contour and the center point according to the coordinates of the center point of the target contour and the coordinates of the center point of the target image; wherein the coordinates of the target contour are the coordinates of the center point of the target contour, and the target contour is a rectangle or a circle; Determining the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system includes: Obtaining a selected instruction, and determining a preset spatial coordinate system according to the selected instruction; Determine the three-dimensional coordinates of the end of the robotic arm and the target to be identified in the preset spatial coordinate system respectively; Determine the spatial relative position of the end of the manipulator and the target to be identified based on the three-dimensional coordinates of the end of the manipulator and the target to be identified; The preset control rules are: The spatial movement amplitude of each servo adjustment is 1 cm.

2. The visual servo control method for a robotic arm according to claim 1, wherein: The step of obtaining a target image of the target to be identified and determining a center point of the target image specifically includes: Obtain an image captured and uploaded by a camera installed at the end of the robotic arm at a preset position, which is the target image; An intersection point of the diagonal lines of the target image is determined, where the intersection point is the center point of the target image.

3. The visual servo control method for a robotic arm according to claim 1, wherein: The method for identifying the target contour in the target image based on a preset color recognition model specifically includes: Get the setting instructions and set the threshold ranges of hue, saturation and brightness respectively; The background area of the target image is filtered by a preset HSV model to determine the target outline. The background area is all areas in the target image except the target to be identified.

4. A visual servo control device for a robotic arm, characterized in that: A method for executing a visual servo control method for a robotic arm according to any one of claims 1 to 3, comprising: An acquisition module (201) is used to acquire a target image of a target to be identified and determine a center point of the target image; A recognition module (202) is configured to recognize a target contour in the target image based on a preset color recognition model; A first position determination module (203) is used to determine the relative position of the target contour and the center point; A second position determination module (204) is used to determine the spatial relative position of the end of the robotic arm and the target to be identified according to a preset spatial coordinate system; A third position determination module (205) is used to predict the real-time position of the end of the robotic arm according to a preset control rule, the relative position and the spatial relative position; The control module (206) is used to output control instructions according to the real-time position of the end of the robot arm and DH inverse kinematics.

5. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 3.

Images