Clamping mechanical arm based on visual perception and control method thereof

Abstract

The invention relates to a clamping mechanical arm based on visual perception and a control method thereof. According to the method, an image of a working area is obtained through a visual perception system, and the position and posture information of a target object is determined through the steps of image preprocessing, feature recognition, target object feature analysis and the like; path planning is conducted through the position and posture information of the target object, the motion track of an end effector of the mechanical arm is calculated, and a corresponding mechanical arm joint angle sequence is obtained through an inverse kinematics algorithm; and an execution instruction is generated based on the angle sequence, and the mechanical arm is controlled to move to the target position and clamp the target object. Therefore, accurate control over the mechanical arm is achieved, the mechanical arm can automatically recognize and grab the target object in the working area, and the automation level and the operation efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot arms, and in particular relates to a visually-perceived clamping robot arm and a control method thereof. Background Art

[0002] Robotic arms are increasingly used in many fields such as modern industrial manufacturing and automated logistics. However, traditional robotic arms often face many challenges when performing tasks.

[0003] On the one hand, in the target grasping operation, most traditional robotic arms lack accurate visual perception capabilities, which results in the inability of the robotic arms to accurately locate the target in a complex and changeable working environment, greatly reducing the grasping success rate, and are prone to collisions with surrounding objects, reducing work efficiency and possibly damaging equipment and objects.

[0004] On the other hand, as production needs develop towards refinement and intelligence, higher requirements are placed on the operational flexibility and autonomy of robotic arms. For example, on the electronic product assembly line, the parts are small in size and of various shapes, and traditional robotic arms are difficult to efficiently complete precise grasping and assembly tasks; in logistics warehouses, goods are placed in disorder, and if the robotic arms cannot intelligently identify and adapt to the position and posture of different goods, the efficiency of cargo handling will be relatively low.

[0005] To overcome these problems, there is an urgent need for a visually-perceived gripping robot and its control method, which can quickly and accurately identify the target object and plan the reasonable motion path and posture adjustment strategy of the robot accordingly, thereby significantly improving the working efficiency and intelligence level of the robot in various complex task scenarios, and meeting the strict requirements of modern industrial production and logistics automation for efficient, precise and intelligent operation. Summary of the invention

[0006] Based on this, it is necessary to provide a visual perception-based clamping robot arm and its control method that can improve the working efficiency and intelligence level of the robot arm in various complex task scenarios in response to the above technical problems.

[0007] In a first aspect, the present application provides a control method for a clamping robot arm based on visual perception, comprising:

[0008] Acquire an initial image of the working area, and preprocess the initial image to obtain an image to be recognized;

[0009] Perform feature recognition on the image to be identified to obtain the features of the target object;

[0010] Analyze the characteristics of the target object to obtain the location and posture information of the target object;

[0011] The end effector of the robot arm is path-planned according to the position information and posture information of the target object to obtain the motion trajectory of the end effector of the robot arm; the end effector is used to clamp the target object;

[0012] According to the motion trajectory, the robot arm inverse kinematics algorithm is used to calculate the robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory;

[0013] Based on the robot arm joint angle sequence, an execution instruction is generated, and the execution instruction is used to control the robot arm to move to the target shape to clamp the target object.

[0014] In a second aspect, the present application also provides a control system for a clamping robot arm based on visual perception, comprising:

[0015] An image acquisition module is used to acquire an initial image of the working area, pre-process the initial image, and obtain an image to be recognized;

[0016] A visual recognition module is used to perform feature recognition on the image to be recognized and obtain the features of the target object;

[0017] The target object information acquisition module is used to analyze the target object characteristics and obtain the target object's position information and posture information;

[0018] The motion trajectory planning module is used to plan the path of the end effector of the robot arm according to the position information and posture information of the target object, and obtain the motion trajectory of the end effector of the robot arm; the end effector is used to clamp the target object;

[0019] A robot arm joint angle calculation module is used to calculate the robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory using a robot arm inverse kinematics algorithm according to the motion trajectory;

[0020] The execution instruction generation module is used to generate execution instructions based on the robot arm joint angle sequence, and the execution instructions are used to control the robot arm to move to the target shape to clamp the target object.

[0021] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a visual perception-based gripping robot arm and a control method thereof as described in the first aspect.

[0022] The above-mentioned visual perception-based gripping robot and its control method obtains the image of the working area through the visual perception system, and determines the position and posture information of the target object through steps such as image preprocessing, feature recognition, and target feature analysis; uses the position and posture information of the target object for path planning, calculates the motion trajectory of the robot end effector, and applies the inverse kinematics algorithm to obtain the corresponding robot joint angle sequence; generates execution instructions based on the angle sequence, controls the robot arm to move to the target position and clamp the target object. In this way, precise control of the robot arm is achieved, enabling it to automatically identify and grasp the target object in the working area, and improving the automation level and operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic flow chart of a control method for a clamping robot arm based on visual perception provided by the present invention;

[0025] Figure 2 A schematic diagram of the structure of a control system of a gripping robot arm based on visual perception provided by the present invention;

[0026] Figure 3 It is a stereoscopic diagram of the overall structure of the clamping mechanical arm of the present invention;

[0027] Figure 4 This is an experiment of the clamping robot arm of the present invention gripping an object (a) and a display diagram of its rotation angle and movement speed (b).

[0028] In the figure:

[0029] 1. Support base; 2. A-arm motion link; 3. B-arm motion link; 4. C-arm motion link; 5. D-arm motion link; 6. Robot gripper. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] Embodiment 1:

[0032] refer to Figure 1, which shows a flow chart of a control method of a gripping robot arm based on visual perception provided by the present application, the method comprising the following steps:

[0033] S101, obtaining an initial image of a working area, and preprocessing the initial image to obtain an image to be recognized.

[0034] Specifically, the working area can be photographed by a suitable visual sensor (such as a camera) to obtain the original image data of the area. The visual sensor should be reasonably installed in a place that can fully cover the working area and clearly capture the possible location of the target object. For example, on an industrial production line, the camera may be installed on a specific bracket above or on the side of the robot arm to ensure that its field of view covers the entire work surface where the target object to be operated is located.

[0035] The images captured by the sensor can contain a lot of information such as various objects in the working area, background, and possible light changes.

[0036] Preprocessing can include denoising, grayscale processing and contrast enhancement processing.

[0037] After the above series of preprocessing operations, the image obtained is the image to be identified, which is more suitable for subsequent target feature recognition and other processing than the initial image.

[0038] S102, performing feature recognition on the image to be recognized to obtain target object features.

[0039] Specifically, for the image to be identified, a variety of feature extraction methods can be used to identify the features of the target object. Common methods include edge detection, such as the Sobel operator and the Canny operator. The Sobel operator detects the edge information of the image by calculating the grayscale change rate of each pixel in the image in the horizontal and vertical directions. These edges are often an important part of the target object's outline. The Canny operator further refines and filters the edges based on the Sobel operator, and can obtain more accurate and clear edge features.

[0040] In addition, you can also use region-based feature extraction methods, such as color feature-based extraction. If the target has obvious color features and is significantly different from the background color, you can extract the color-related features of the target by dividing and counting different color regions in the image, such as calculating the main color components and color distribution of the target region.

[0041] There are also extraction methods based on texture features. For some targets with specific texture surfaces, their texture features can be extracted by analyzing the grayscale change pattern, periodicity and other texture characteristics of the pixels in the image, such as calculating the roughness, directionality and other parameters of the texture.

[0042] Based on the appropriate feature extraction method, the image to be identified can be comprehensively scanned and analyzed. Taking edge detection as an example, when using the Sobel operator, the corresponding gradient value will be calculated at each pixel point in the image according to the calculation rules of the operator, and these gradient values ​​will be used to determine which pixels belong to the edge part, thereby outlining the general outline edge features of the target object.

[0043] If it is based on color feature extraction, color analysis will be performed on each pixel in the image, and pixels with similar colors will be classified together to form different color areas. Then, by comparing the degree of matching between these color areas and the color features of known targets, the position of the target in the image and its color-related features will be determined.

[0044] For texture feature extraction, a window of appropriate size (such as a rectangular window) is selected on the image, moved to different positions, and the texture characteristics of the pixels in the window are analyzed. By comparing and matching with the known target texture feature template, the texture characteristics of the target object and its position in the image are identified.

[0045] Through the above-mentioned various feature extraction and recognition processes, the characteristics of the target object are finally obtained. These characteristics may include information such as the contour shape, color, texture, etc. of the target object.

[0046] S103: Analyze the characteristics of the target object to obtain the position information and posture information of the target object.

[0047] Specifically, based on the previously obtained target features, the position information of the target is determined. If the contour features of the target are obtained by edge detection, the position of the target in the image can be roughly determined by calculating the center position of the contour. For example, for a simple circular target, after its contour is accurately detected, the coordinates of the center of the circle surrounded by the contour can be calculated as the position coordinates of the target in the image.

[0048] In the case where the target is identified by color features or texture features, the position of the target can also be determined based on the center position of the color area or texture area. For example, for a rectangular target with a specific color, after determining its color area, the geometric center coordinates of the color area can be calculated as the position coordinates of the target.

[0049] In addition, the conversion relationship between the image coordinate system and the actual work area coordinate system also needs to be considered. Because the position information in the image obtained by the visual sensor is based on the image's own coordinate system, and the actual operation of the robot is performed in the coordinate system of the actual work area, it is necessary to use a pre-established coordinate conversion model (such as a conversion matrix determined by calibration, etc.) to convert the position coordinates in the image into the position coordinates in the actual work area, so as to accurately guide the movement of the robot.

[0050] For analyzing the posture information of the target object, some targets with specific shapes, such as cuboids and cylinders, can determine their posture by analyzing their contour features and comparing them with known standard shapes. For example, for a cuboid target, if the contours of its various faces have been identified, the posture of the cuboid can be determined by observing the relative positions and angles of the various faces in the image, such as whether it is flat, vertical, or tilted.

[0051] For objects with texture features, their posture can also be inferred based on the directionality of the texture. For example, for a cylinder with regular texture on its surface, by observing the direction of the texture in the image, it can be determined whether the cylinder is upright or lying on its side.

[0052] Similarly, when determining the posture information, it is also necessary to convert the posture description obtained based on the image coordinate system into a posture description in the actual working area coordinate system.

[0053] S104, performing path planning for the end effector of the robot arm according to the position information and posture information of the target object to obtain a motion trajectory of the end effector of the robot arm; the end effector is used to clamp the target object.

[0054] Specifically, the main goal of path planning is to enable the end effector of the robot arm to move safely, efficiently and accurately from its current position to the position of the target object, and to contact the target object in a suitable posture for clamping operations.

[0055] First, the target position that the end effector needs to reach is determined based on the position information of the target object. The position coordinates of the target object in the actual working area coordinate system are used as the final position point that the end effector needs to reach.

[0056] Then, the posture information of the target object is combined to determine the posture that the end effector should have when it reaches the target position. For example, if the target object is a rectangular parallelepiped and is lying flat, then when the end effector reaches the top of the target object and is ready to clamp it, its posture should also be adjusted to be parallel to the surface of the target object and accurately aligned with the clamping part of the target object.

[0057] Finally, according to the above target position and target posture, the motion trajectory of the end effector of the robot arm is generated through the corresponding path planning algorithm. This motion trajectory is a series of coordinate points and corresponding posture descriptions in the actual working area coordinate system, which specifies in detail the movement route and posture change process of the end effector from the current position to the target position.

[0058] S105. According to the motion trajectory, a robot arm inverse kinematics algorithm is used to calculate a robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory.

[0059] Specifically, the inverse kinematics algorithm of the robot arm is a method used to reversely infer the angles that each joint of the robot arm should have based on the position and posture of the end effector of the robot arm (that is, the position and posture corresponding to each trajectory point in the motion trajectory here).

[0060] For a robotic arm with multiple joints, the position and posture of its end effector are determined by the angles of each joint. The inverse kinematics algorithm is to establish a mapping relationship from the position and posture of the end effector to the angles of each joint. This mapping relationship is usually implemented through complex mathematical models, involving knowledge fields such as geometry and trigonometric functions. For example, for a common six-degree-of-freedom robotic arm, the rotation of each joint will affect the position and posture of the end effector. The inverse kinematics algorithm will analyze this influence relationship and use the known position and posture information of the end effector to calculate the angle that each joint should have.

[0061] First, for each trajectory point in the motion trajectory, the position and posture information of the corresponding end effector are used as known input conditions. These position and posture information are accurate descriptions in the actual working area coordinate system.

[0062] Then, these input conditions are substituted into the appropriate inverse kinematics algorithm model of the robot arm. Different types of robot arms may require different inverse kinematics algorithm models, such as inverse kinematics algorithm models based on analytical methods, inverse kinematics algorithm models based on numerical methods, etc. Taking the inverse kinematics algorithm model based on analytical methods as an example, it will calculate the angle value corresponding to each joint through a series of geometric derivations and trigonometric function operations based on the specific structure of the robot arm (such as the number, type, connection method of the joints, etc.) and known geometric relationships.

[0063] Finally, after calculation, the robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory is obtained. This angle sequence specifies in detail the angles that each joint of the robot arm should have at each trajectory point, providing the necessary parameters for the subsequent generation of execution instructions to control the movement of the robot arm.

[0064] S106. Generate an execution instruction based on the robot arm joint angle sequence, where the execution instruction is used to control the robot arm to move to a target shape to clamp the target object.

[0065] Specifically, first, according to the obtained robot arm joint angle sequence, each joint angle value is converted into a format suitable for receiving by the robot arm control system. Different robot arm control systems may have different instruction format requirements, such as some may require digital encoding, and some may require a specific string format. Therefore, it is necessary to perform corresponding format conversion according to the specific robot arm control system.

[0066] Then, these converted instructions are combined in a certain order to form a complete execution instruction set. This execution instruction set clearly specifies how each joint of the robot arm should move, from the initial state to the state where it can finally reach the target shape and clamp the target object. For example, for a six-degree-of-freedom robot arm, the execution instruction set may specify in detail how many degrees the first joint should rotate first, then how many degrees the second joint should rotate, and so on, until all joints move to the appropriate angles so that the robot arm reaches the target shape and can successfully clamp the target object.

[0067] The generated execution instruction is sent to the core controller of the robot arm. After receiving the execution instruction, the core controller of the robot arm will drive the motors and other actuators of each joint to move in sequence according to the requirements of the instruction.

[0068] During the movement of the robot arm, the control system can continuously monitor the actual movement angle of each joint and other parameters, and compare them with the angle specified in the execution instruction. If the actual movement angle is found to deviate from the specified angle, the control system can make timely adjustments, and through the feedback mechanism, ensure that the robot arm can accurately move to the target shape and clamp the target object according to the requirements of the execution instruction.

[0069] The above-mentioned control method of a gripping robot based on visual perception obtains an image of the working area through a visual perception system, and determines the position and posture information of the target object through steps such as image preprocessing, feature recognition, and target feature analysis. Then, this information is used for path planning, the motion trajectory of the end effector of the robot arm is calculated, and the inverse kinematics algorithm is applied to obtain the corresponding robot arm joint angle sequence. Finally, execution instructions are generated based on these angle sequences to control the robot arm to move accurately to the target position and clamp the target object. This technology realizes precise control of the robot arm, enabling it to automatically identify and grasp the target objects in the working area, improving the level of automation and operating efficiency.

[0070] In an optional embodiment, performing feature recognition on the image to be recognized to obtain the target object features includes the following steps:

[0071] The image to be identified is input into the trained convolutional neural network, and the category information of the target object is output C = {c 1 , c 2 , …, c n} and the bounding box coordinate information of the target object in the image to be identified Among them, c i represents the i-th possible target category, i = 1, 2, ..., n, is the coordinate of the upper left corner of the bounding box of the jth target, is the coordinate of the lower right corner of the bounding box of the jth target object, and m is the number of detected targets;

[0072] According to the preset target category priority list P = {p 1 , p 2 ,…,p N}, filter out the target category C with the highest priority from the identified category information C target and its corresponding bounding box coordinates Among them, p k is the target object category and has a corresponding priority value, k = 1, 2, ..., N;

[0073] The bounding box coordinates as target feature.

[0074] Specifically, a convolutional neural network (CNN) is a deep learning model specifically designed to process grid-structured data (such as images). In this step, the preprocessed image to be recognized is input into the trained one.

[0075] CNN automatically extracts various features from images through its convolutional layers. The convolution kernel in the convolutional layer slides over the image and performs convolution operations on local areas of the image to detect different feature patterns, such as edges, textures, etc. As the number of network layers increases, these features are gradually combined and abstracted to form a higher-level feature representation of the target object.

[0076] Next, the features are processed through the pooling layer to reduce the dimension, reduce the amount of calculation and retain the key feature information. Finally, the extracted features are mapped to the category information and bounding box coordinate information of the target object through the fully connected layer.

[0077] Target object category information C = {c 1 , c 2 , …, c n} represents the n possible object categories that the neural network can identify. For example, in an industrial parts sorting scenario, c1 May represent screws, c 2 May represent a nut, etc.

[0078] Bounding box coordinate information Used to determine the position and size of each target object in the image. is the coordinate of the upper left corner of the bounding box of the jth object, is the lower right corner coordinate of the bounding box of the jth object, and m is the number of detected objects. For example, if a screw is detected, the upper left corner coordinate of its bounding box is (10, 20) and the lower right corner coordinate is (50, 80), then this bounding box defines the approximate range of the screw in the image.

[0079] Target category priority list P = {p 1 , p 2 ,…,p N} is pre-set according to specific application scenarios and task requirements. For example, in an automated assembly line, if the current task is to install a specific large part first, then the category of this large part will be given a higher value in the priority list.

[0080] The significance of setting priorities is that when there are multiple objects in the image, the objects can be processed in order of importance. This helps the robot arm prioritize key objects, improve work efficiency and accuracy, and avoid confusion or erroneous operations caused by processing multiple objects at the same time.

[0081] Filter from the identified category information C. For each identified target category c i , find its corresponding priority value in the priority list P. For example, if a screw (c 1 Category) and nuts (c 2 Category), and the nut has a higher priority value in the priority list, then the nut c 2 Determined as the target category c with the highest priority target .

[0082] At the same time, obtain the bounding box coordinates corresponding to the highest priority target category These coordinates will be used for subsequent analysis of the target object's position and posture and the operation of the robotic arm.

[0083] The bounding box coordinates can accurately describe the position and size of the target in the image, which is crucial for the subsequent calculation of the position and posture information of the target. For example, the center coordinates of the bounding box can be used to preliminarily estimate the position of the target, while the shape and size information of the bounding box can assist in determining the posture of the target, such as whether the target is tilted.

[0084] In robotic arm control, these bounding box coordinate features will serve as an important reference for planning the motion path of the robotic arm end effector, so that it can accurately reach the location of the target object and perform the gripping operation in a suitable posture. They are the key intermediate information connecting visual perception and robotic arm motion control, ensuring that the entire system can complete the task accurately and efficiently.

[0085] In an optional embodiment, obtaining the position information and posture information of the target object based on the target object features includes the following steps:

[0086] S1: Based on the image bounding box coordinates of the target with the highest priority, obtain the coordinate information of the target with the highest priority in the camera coordinate system;

[0087] S2: Convert the coordinate information in the camera coordinate system into the coordinate information in the world coordinate system as position information;

[0088] S3: Calculate the posture parameters of the target object with the highest priority in the world coordinate system as posture information.

[0089] Specifically, when the highest priority target and its corresponding image bounding box coordinates are determined through the previous steps After that, it needs to be converted into the coordinates in the camera coordinate system. The image bounding box coordinates are a two-dimensional coordinate system based on the image plane, while the camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin and the camera optical Z axis as the axis direction.

[0090] Usually, in camera imaging models (such as pinhole camera models), there are certain geometric relationships that can map points on the image plane to the camera coordinate system. According to the camera's intrinsic parameter matrix (including parameters such as focal length) and the projection relationship between the image plane and the camera coordinate system, the four vertex coordinates of the image bounding box can be converted from the image plane to the camera coordinate system through a series of calculations. For example, for a point (x, y) on the image plane, its three-dimensional coordinates (x, y) in the camera coordinate system can be calculated through the intrinsic parameter matrix K and the corresponding projection formula. c , Y c , Z c ).

[0091] For the bounding box of the target object with the highest priority, by performing such transformation on its four vertices respectively, the overall coordinate information of the target object in the camera coordinate system can be obtained. This coordinate information can more accurately reflect the spatial position of the target object under the camera's perspective.

[0092] The camera coordinate system is a local coordinate system established based on the camera itself, while the operation of the robotic arm is performed in the actual world coordinate system. In order for the robotic arm to accurately reach the position of the target object, the coordinates in the camera coordinate system need to be converted into coordinates in the world coordinate system.

[0093] This conversion is usually achieved through an extrinsic matrix, which contains the position and attitude information of the camera in the world coordinate system (such as rotation and translation). By performing matrix multiplication operation on the extrinsic matrix and the coordinates in the camera coordinate system, the coordinates in the camera coordinate system can be converted to the coordinates in the world coordinate system. For example, if the coordinates of a point in the camera coordinate system are (X c , Y c , Z c ), the external parameter matrix is ​​R|T, (where R is the rotation matrix and T is the translation vector), then we can calculate [(X w , Y w , Z w )] T =R[(X c , Y c , Z c )] T +T, you can get the coordinates of the point in the world coordinate system (X w , Y w , Z w ), this coordinate is the position information of the target object in the world coordinate system, and the robot arm can plan the motion path according to this position information.

[0094] The pose parameters of the target object in the world coordinate system can be calculated in many ways. A common method is to determine it based on the geometric shape of the target object and its features in the image. For example, if the target object is a cuboid of known shape, and its bounding box and some feature points (such as the vertices of the cuboid) have been identified in the image.

[0095] The posture of the target object can be determined by calculating the coordinate relationship of these feature points in the world coordinate system. For example, by calculating the direction vectors of the three adjacent edges of a cuboid in the world coordinate system, the rotation matrix of the target object can be obtained. This rotation matrix describes the posture of the target object in the world coordinate system.

[0096] In addition, if there are specific marks or characteristic patterns on the target, the target's posture parameters in the world coordinate system can be calculated by identifying and analyzing these marks or patterns in the image, combined with the relationship between the known marks and the target's posture. These posture parameters will serve as the basis for the robot's end effector to adjust its posture to ensure accurate contact and clamping operations with the target.

[0097] In an optional embodiment, obtaining coordinate information of the target object with the highest priority in the camera coordinate system based on the image bounding box coordinates of the target object with the highest priority includes the following steps:

[0098] S11: Obtain the camera's intrinsic parameter matrix K and extrinsic parameter matrix R through the camera calibration method ext , T ext ;

[0099] in, f x and f y are the focal lengths of the camera in the x and y directions, c x and c y are the coordinates of the image center; R ext Represents the rotation relationship of the camera relative to the world coordinate system, T ext Characterize the translation relationship of the camera relative to the world coordinate system;

[0100] S12: Based on the bounding box coordinates of the highest priority object in the image By formula and Calculate the coordinates of the center of the highest priority target in the image coordinate system (u c , v c );

[0101] S13: According to the camera model formula Calculate the coordinates of the highest priority target in the camera coordinate system (X c , Y c , Z c );

[0102] Convert the coordinate information in the camera coordinate system to the coordinate information in the world coordinate system, including: c , Y c , Z c ), using the coordinate transformation formula Get the position coordinates (X w , Y w , Z w );

[0103] Calculate the posture parameters of the highest priority target in the world coordinate system as posture information, including: based on the Euler angle conversion formula, using the rotation matrix Calculate the posture parameters of the highest priority target in the world coordinate system

[0104] in,

[0105] Specifically, camera calibration is the process of determining the internal geometric and optical properties of the camera (intrinsic parameters) and the position and orientation of the camera in the world coordinate system (extrinsic parameters). This step provides the basic parameters for the subsequent accurate calculation of the coordinates of the target in different coordinate systems.

[0106] Commonly used calibration methods include Zhang Zhengyou calibration method. During the calibration process, a calibration plate of known shape and size (such as a checkerboard calibration plate) is required to capture images of the calibration plate from different angles. By detecting and analyzing feature points (such as the corner points of the checkerboard) in the calibration plate image and combining geometric relationships, the camera's intrinsic parameter matrix K and extrinsic parameter matrix R can be calculated. ext , T ext .

[0107] f in the internal parameter matrix K x and f y Respectively represent the focal length of the camera in the x and y directions, which determine the size and proportion of the object in the image. x and c y is the image center coordinate, which is used to determine the center position of the image plane. The external parameter matrix R ext Through its element r ij Characterizes the rotation relationship of the camera relative to the world coordinate system and describes the rotation posture of the camera in three-dimensional space; T ext Then through t x ,t y ,t z Characterizes the translation relationship of the camera relative to the world coordinate system and determines the position of the camera in the world coordinate system. These parameters accurately establish the connection between the camera coordinate system and the world coordinate system.

[0108] The bounding box coordinates of the highest priority object in the image are known By formula and To calculate the coordinates of the center of the target object in the image coordinate system (u c , v c ).

[0109] The significance of calculating the center coordinates of the target object is that it can be used as a representative point to more concisely describe the position of the target object in the image. Compared with the four vertex coordinates of the bounding box, the center coordinates are more convenient and effective in the subsequent calculation of the coordinates of the target object in the camera coordinate system. This center coordinate will be used as an intermediate variable for further coordinate transformation calculations to determine the position of the target object in three-dimensional space.

[0110] According to the camera model formula To calculate the coordinates of the highest priority target in the camera coordinate system (X c , Y c , Z c ).

[0111] Here K -1 is the inverse matrix of the internal parameter matrix K. By first transforming the center coordinates (u c , v c ) multiplied by K -1 , then multiplied by the depth information Z c (In some cases, depth information can be obtained by other means, such as binocular vision, depth camera, etc.), and the three-dimensional coordinates of the target object in the camera coordinate system can be obtained. This coordinate describes the position of the target object in the coordinate system centered on the camera, and is a key step in converting visual information from a two-dimensional image plane to a three-dimensional space.

[0112] According to the coordinate transformation formula The coordinates of the target object in the camera coordinate system (X c , Y c , Z c ) is converted to the position coordinates in the world coordinate system (X w , Y w , Z w ).

[0113] in is the external parameter matrix R ext The inverse matrix of this formula is used to convert the coordinates in the camera coordinate system into the coordinates in the world coordinate system through the rotation and translation relationship between the camera and the world coordinate system described by the extrinsic matrix. In this way, the position of the target in the world coordinate system can be accurately determined, and the robotic arm and other equipment can perform corresponding operations based on the position information in this world coordinate system.

[0114] Based on the Euler angle conversion formula, using the rotation matrix To calculate the posture parameters of the highest priority target in the world coordinate system

[0115] Here R wc is the inverse matrix of the rotation matrix of the camera coordinate system relative to the world coordinate system, through a specific Euler angle conversion formula, such as (where r′ ij YesR wcThe elements in the rotation matrix can be used to extract the target's attitude information in the world coordinate system, expressed in the form of Euler angles. These attitude parameters accurately describe the target's attitude in the world coordinate system, and the robot can adjust its own attitude based on this attitude information in order to accurately interact with the target.

[0116] In an optional embodiment, based on a path planning algorithm, according to the position information and posture information of the target object, the motion trajectory of the end effector of the robot arm is obtained, including the following steps:

[0117] Get the current joint angle information of the robot arm {α 1 , α 2 , …, α l}; where l is the number of joints of the robot arm, α q is the qth joint, q = 1, 2, ..., l;

[0118] Using the robot arm kinematics model, according to the current joint angle information {α 1 , α 2 , …, α l}, calculate the current position coordinates of the end effector of the robot arm in the world coordinate system and current posture parameters

[0119] Using the A* algorithm and geometric model collision detection algorithm, the current joint angle information {α 1 , α 2 , …, α l}、Current location coordinates and current posture parameters The determined robot arm configuration is used as the starting node, which will enable the end effector to reach the target position (X w , Y w , Z w ) and the posture parameters are The robot arm configuration is taken as the target node to obtain the motion trajectory of the robot arm;

[0120] The motion trajectory is a series of intermediate position points of the end effector of the robot arm. And the corresponding posture parameters s is the number of trajectory points, h = 1, 2, …, s.

[0121] Specifically, to obtain the current joint angle information of the robot arm {α 1 , α 2 , …, α l}, usually achieved by installing angle sensors at the joints of the robot. These sensors can measure the current angle position of each joint in real time. For example, for a common six-degree-of-freedom robot, there will be six angle sensors measuring the angles of the six joints respectively.

[0122] These current joint angle information are key parameters for describing the current configuration of the robot. They reflect the state of each joint of the robot at the current moment and are an important basis for the subsequent calculation of the position and posture of the end effector of the robot and path planning. Because the position and posture of the end effector of the robot in space are jointly determined by the angles of each joint, accurately obtaining the current joint angle information is the starting point of the entire control process.

[0123] Using the robot arm kinematics model, according to the current joint angle information {α 1 , α 2 , …, α l} to calculate the current position coordinates of the end effector in the world coordinate system and current posture parameters

[0124] The kinematic model of the robot arm is a mathematical model that establishes the mapping relationship between the joint space of the robot arm and the task space of the end effector (here is the world coordinate system). It realizes this mapping through a series of complex geometric derivations and calculations based on factors such as the geometric structure of the robot arm and the joint connection method. For example, for a common serial robot arm, the kinematic model considers how the rotation or translation movement of each joint affects the position and posture of the end effector. By substituting the current joint angle into the corresponding formula in the kinematic model, after a series of trigonometric function operations, matrix multiplication and other calculation processes, the accurate position and posture information of the end effector in the world coordinate system can be obtained. This step converts the joint angle information inside the robot arm into a state description of the end effector in the actual workspace (world coordinate system).

[0125] The A* algorithm is a commonly used path search algorithm. In the path planning of the robot arm, it is based on the current joint angle information {α 1 , α 2 , …, α l}、Current location coordinates and current posture parameters The determined robot arm configuration is used as the starting node to enable the end effector to reach the target position (X w , Y w , Z w ) and the posture parameters are The robot configuration of is taken as the target node and the optimal path is searched in the robot configuration space.

[0126] The A* algorithm will comprehensively consider the cost of the path (such as path length, smoothness of motion, etc.) and the estimated cost of reaching the target (heuristic function) during the search process. It continuously expands nodes, evaluates the cost of each node to the target node, and selects the node with the smallest cost to continue expanding until the target node is found or it is determined that there is no feasible path. In this way, a relatively optimal motion trajectory from the current robot configuration to the target configuration can be found, allowing the robot to reach the target position and adjust to the target posture with high efficiency and accuracy.

[0127] The geometric model collision detection algorithm is used to avoid collisions between the robot and obstacles in the working area during path planning. It performs collision detection on each possible robot configuration generated during the A* algorithm search process based on the robot's geometric model (including information such as the shape and size of each joint and link) and the known obstacle geometry model in the working area.

[0128] If a configuration is detected that will cause the robot to collide with an obstacle, then this configuration and its subsequent extension path will be excluded from the feasible path. In this way, the final motion trajectory is ensured to be safe and feasible, avoiding the robot arm colliding with the surrounding environment during movement, causing damage or mission failure.

[0129] The final motion trajectory is a series of intermediate position points of the end effector of the robot arm. And the corresponding posture parameters Where s is the number of trajectory points, h = 1, 2, ..., s.

[0130] These intermediate position points and posture parameters describe in detail the position and posture of the end effector at each stage of the movement from the starting position to the target position. Based on this information, the robot control system can control the joint movement of the robot in sequence, so that the end effector can move accurately along the planned trajectory and achieve effective operation of the target object.

[0131] In an optional embodiment, according to the motion trajectory, the robot arm inverse kinematics algorithm is used to calculate the robot arm joint angle sequence corresponding to each trajectory point, including the following steps:

[0132] For each trajectory point h, set the joint angles of the robot arm to the estimated values

[0133] According to the forward kinematics equation of the robot arm, using the preset guess value Calculate the position and attitude of the robot end effector

[0134] Calculate the error of the end effector position and attitude

[0135] Calculate the Jacobian matrix J of the forward kinematics equation at the current joint angle; the element J of the Jacobian matrix J ab represents the partial derivative of the ath position or attitude component of the end effector with respect to the bth joint angle;

[0136] By iterative formula Update the estimated value of the joint angle; where g is the number of iterations;

[0137] When the norm of the error between the end effector position and attitude is When it is less than the preset threshold ∈, the convergence condition is met and the current joint angle Confirm the robot arm joint angle sequence of trajectory point h.

[0138] Specifically, for each trajectory point h, first set the joint angle of the robot arm to the estimated value This estimate can usually be selected based on some prior knowledge or experience. For example, if the robot arm has similar trajectory points in previous movements, the corresponding joint angles can be used as the initial estimate.

[0139] These initial estimates provide a starting point for subsequent iterative calculations. Although they may not be accurate, they will gradually approach the true joint angle values ​​during the iteration process, allowing the calculation to proceed in the right direction.

[0140] Using the forward kinematics equation of the robot arm, the preset guess value (i.e. the current estimated value of the joint angle) is Substitute it and calculate the position and posture of the end effector of the robot arm

[0141] The forward kinematics equation of the robot arm describes the mapping relationship from the joint space to the task space (here, the position and posture of the end effector). It is based on the geometric structure of the robot arm and the principle of joint kinematics. Through a series of matrix transformations and trigonometric functions, it calculates the position and posture of the end effector in the world coordinate system according to the given joint angle. For example, for a simple planar two-joint robot arm, the forward kinematics equation considers how the rotation angle of the joint affects the coordinate position and orientation angle of the end effector in the plane. Through this step, the theoretical position and posture of the end effector based on the current estimated joint angle can be obtained.

[0142] Calculate the error of the end effector position and attitude in is the actual desired end effector position and posture (obtained by path planning) at that point in the motion trajectory, and It is the position and posture of the end effector calculated by the current joint angle estimate.

[0143] This error vector accurately reflects the difference between the current calculated end-effector state and the actual desired state. The components of the error are expressed in position (X, Y, Z directions) and attitude (θ x ,θ y ,θ z By calculating the error, the accuracy of the current joint angle estimate can be evaluated, and a basis for subsequent adjustment of the joint angle is provided. That is, the purpose of iteration is to gradually reduce this error until the convergence condition is met.

[0144] The element J of the Jacobian matrix J ab It represents the partial derivative of the ath position or attitude component of the end effector with respect to the bth joint angle. Calculating the Jacobian matrix requires the derivative of the forward kinematics equation, which involves relevant knowledge of the robot's kinematics and complex mathematical derivation.

[0145] The Jacobian matrix is ​​of great significance in the motion control of robotic arms. It describes the linear mapping relationship between the joint space velocity and the task space velocity (end effector velocity). In inverse kinematics calculations, it is used to convert the position and posture errors of the end effector into the adjustment of the joint angle. -1 For example, if the end effector has a position error in a certain direction, the Jacobian matrix can help determine which joint angles need to be adjusted and by how much, thereby guiding the joint angles to be updated in the direction of reducing the error.

[0146] Using the Iterative Formula To update the estimated value of the joint angle, where g is the number of iterations. The basic principle of this formula is to correct the current estimated value of the joint angle based on the current error e and the inverse matrix of the Jacobian matrix.

[0147] In each iteration, the error is multiplied by the inverse of the Jacobian matrix to obtain the adjustment of the joint angle, and then this adjustment is added to the current joint angle estimate to obtain a new estimate. This iterative process is repeated continuously, so that the joint angle estimate gradually approaches the actual joint angle value that can make the end effector reach the actual desired position and posture. As the number of iterations increases, the error will gradually decrease, and the joint angle estimate will get closer and closer to the optimal solution.

[0148] When the norm of the error between the end effector position and attitude is When it is less than the preset threshold ∈, the convergence condition is met. The error norm here is a comprehensive measure of the error vector e, which takes into account the error size in all directions of position and posture.

[0149] When the convergence condition is met, it means that the currently calculated joint angle estimate is accurate enough to enable the end effector to reach a position and posture that is very close to the actual expectation. The robot arm joint angle sequence for the trajectory point h is confirmed. This joint angle sequence is the angle value that the robot arm joint corresponding to the trajectory point should reach. By repeating the above iterative calculation process for each trajectory point, the robot arm joint angle sequence corresponding to the entire motion trajectory can be obtained, thereby providing accurate joint angle instructions for controlling the robot arm to move along the planned trajectory.

[0150] The present application implements a control method for a clamping robot arm based on visual perception. Through visual perception, a camera is used to obtain an image of the working area. After preprocessing, feature recognition (with the help of a convolutional neural network), and target feature analysis (including obtaining the image boundary box coordinates and converting them into position and posture information in the camera and world coordinate systems), the robot arm path planning (using the A* algorithm combined with collision detection) and inverse kinematics calculation (through iterative solution) are performed based on this to determine the joint angle sequence, and finally generate execution instructions to control the movement of the robot arm. The technical effects achieved include being able to accurately identify the target object, obtain its precise position and posture information, so that the robot arm can efficiently, safely and accurately move to the target form and successfully clamp the target object, which improves the autonomy, accuracy and adaptability of the robot arm in the process of executing the task, and is suitable for a variety of industrial automation operation scenarios, such as parts sorting, assembly, etc.

[0151] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0152] Based on the same inventive concept, the embodiment of the present application also provides a control system for a clamping robot arm based on visual perception for implementing the above-mentioned method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of one or more embodiments of the control system for a clamping robot arm based on visual perception provided below can be referred to the limitations of a control method for a clamping robot arm based on visual perception above, and will not be repeated here.

[0153] In an exemplary embodiment, Figure 2 As shown, a control system 200 for a gripping robot arm based on visual perception is provided, and the system includes:

[0154] The image acquisition module 201 is used to acquire an initial image of the working area and pre-process the initial image to obtain an image to be recognized.

[0155] The visual recognition module 202 is used to perform feature recognition on the image to be recognized to obtain target object features.

[0156] The target object information acquisition module 203 is used to analyze the target object features and obtain the position information and posture information of the target object.

[0157] The motion trajectory planning module 204 is used to plan the path of the end effector of the robot arm according to the position information and posture information of the target object, and obtain the motion trajectory of the end effector of the robot arm; the end effector is used to clamp the target object.

[0158] The robot arm joint angle calculation module 205 is used to calculate the robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory using the robot arm inverse kinematics algorithm according to the motion trajectory.

[0159] The execution instruction generation module 206 is used to generate an execution instruction based on the robot arm joint angle sequence, and the execution instruction is used to control the robot arm to move to a target shape to clamp the target object.

[0160] Optionally, the visual recognition module 202 includes:

[0161] The feature recognition unit is used to input the image to be recognized into the trained convolutional neural network and output the category information C of the target object. 1 , c 2 , …, c n} and the bounding box coordinate information of the target object in the image to be identified Among them, c i represents the i-th possible target category, i= is the coordinate of the upper left corner of the bounding box of the jth target, is the coordinate of the lower right corner of the bounding box of the jth object, and m is the number of detected objects.

[0162] The feature screening unit is used to select the target object according to the preset target object category priority list P = {p 1 , p 2 ,…,p N}, filter out the target category c with the highest priority from the identified category information C target and its corresponding bounding box coordinates Among them, p k is the target object category and has a corresponding priority value, k = 1, 2, …, N.

[0163] The target feature determination unit is used to convert the bounding box coordinates as target feature.

[0164] Optionally, the target information acquisition module 203 includes:

[0165] The camera coordinate calculation unit 2031 is used to obtain the coordinate information of the target object with the highest priority in the camera coordinate system based on the image bounding box coordinates of the target object with the highest priority.

[0166] The world coordinate calculation unit 2032 is used to convert the coordinate information in the camera coordinate system into the coordinate information in the world coordinate system as the position information.

[0167] The posture parameter calculation unit 2033 is used to calculate the posture parameters of the target object with the highest priority in the world coordinate system as posture information.

[0168] Optionally, the camera coordinate calculation unit 2031 is specifically used to perform the following operations:

[0169] Through the camera calibration method, the camera's intrinsic parameter matrix K and extrinsic parameter matrix R are obtained ext , T ext ;

[0170] in, f x and f y are the focal lengths of the camera in the x and y directions, c x and c y are the coordinates of the image center; R ext Represents the rotation relationship of the camera relative to the world coordinate system, T ext Characterize the translation relationship of the camera relative to the world coordinate system;

[0171] Based on the bounding box coordinates of the highest priority object in the image By formula and Calculate the coordinates of the center of the highest priority target in the image coordinate system (u c , v c );

[0172] According to the camera model formula Calculate the coordinates of the highest priority target in the camera coordinate system (X c , Y c , Z c ).

[0173] The world coordinate calculation unit 2032 is specifically used to perform the following operations:

[0174] According to the coordinates (X c , Y c , Z c ), using the coordinate transformation formula Get the position coordinates (X w , Y w , Z w ).

[0175] The posture parameter calculation unit 2033 is specifically used to perform the following operations:

[0176] Based on the Euler angle conversion formula, using the rotation matrix Calculate the posture parameters of the highest priority target in the world coordinate system

[0177] in,

[0178] Optionally, the motion trajectory planning module 204 includes:

[0179] The current joint angle information acquisition unit is used to obtain the current joint angle information of the robot arm {α 1 , α 2 , …, α l}; where l is the number of joints of the robot arm, α q is the qth joint, q=1, 2,…, l.

[0180] The end effector information calculation unit is used to use the robot arm kinematics model to calculate the current joint angle information {α 1 , α 2 , …, α l}, calculate the current position coordinates of the end effector of the robot arm in the world coordinate system and current posture parameters

[0181] The path planning unit is used to use the A* algorithm and the geometric model collision detection algorithm to convert the current joint angle information {α 1 , α 2 , …, α l}、Current location coordinates and current posture parameters The determined robot arm configuration is used as the starting node, which will enable the end effector to reach the target position (X w , Y w , Z w ) and the posture parameters are The robot arm configuration is taken as the target node and the motion trajectory of the robot arm is obtained.

[0182] The motion trajectory is a series of intermediate position points of the end effector of the robot arm. And the corresponding posture parameters s is the number of trajectory points, h = 1, 2, …, s.

[0183] Optionally, the robot arm joint angle calculation module 205 includes:

[0184] The joint angle estimation unit is used to set the joint angle of the robot arm to the estimated value for each trajectory point h.

[0185] The end effector position and posture calculation unit is used to calculate the position and posture of the robot arm using the preset guess value according to the forward kinematic equation of the robot arm. Calculate the position and attitude of the robot end effector

[0186] Error calculation unit, used to calculate the error of the end effector position and posture

[0187] The Jacobian matrix J acquisition unit is used to calculate the Jacobian matrix J of the forward kinematics equation at the current joint angle; among them, the element J of the Jacobian matrix J ab Represents the partial derivative of the ath position or attitude component of the end effector with respect to the bth joint angle.

[0188] Iteration unit, used to iterate formulas Update the estimated joint angles; where g is the number of iterations.

[0189] The robot arm joint angle sequence determination unit, when the norm of the error of the end effector position and posture When it is less than the preset threshold ∈, the convergence condition is met and the current joint angle Confirm the robot arm joint angle sequence of trajectory point h.

[0190] The embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0191] Embodiment 2:

[0192] refer to Figure 3 and Figure 4 , a clamping robot arm, applicable to the above-mentioned control method of the clamping robot arm based on visual perception;

[0193] It also includes: a support base 1, an A-arm motion link 2, a B-arm motion link 3, a C-arm motion link 4, a D-arm motion link 5 and a manipulator claw 6;

[0194] The bottom end of the A-arm motion link 2 is fixedly connected to the supporting base 1, and one side of the supporting base 1 is connected with an A rotating drive component for driving the supporting base 1 to rotate around the axis of the supporting base 1 itself; the top end of the A-arm motion link 2 is rotationally connected to the B-arm motion link 3 through the A link joint, and one side of the B-arm motion link 3 is connected with a B rotating drive component for driving the B-arm motion link 3 to rotate around the axis of the A link joint; the end of the B-arm motion link 3 away from the A-arm motion link 2 is rotationally connected to the C-arm motion link 4 through the B link joint, and one side of the C-arm motion link 4 is connected with a B rotating drive component for driving the C-arm motion link 4 to rotate around the axis of the B link joint; the end of the C-arm motion link 4 away from the B-arm motion link 3 is rotationally connected to the D-arm motion link 5 through the C link joint, and one side of the D-arm motion link 5 is connected with a C rotating drive component for driving the D-arm motion link 5 to rotate around the axis of the C link joint; the manipulator claw 6 is fixedly installed on the D-arm motion link 5 and is located at the end of the D-arm motion link 5 away from the C-arm motion link 4.

[0195] The design principle of the gripper robot is to imitate the structure and movement of the human arm. It consists of multiple joints and connectors. Its main components include the following 4 parts:

[0196] (1) Support base 1: The support base 1 of the clamping robot arm is usually fixed on a workbench or the ground, and serves as the main structure for the stability and support force source of the clamping robot arm. In this design, a motor is installed on the wheels to form the base of the clamping robot arm, which is equivalent to a simple trolley to facilitate the movement of the clamping robot arm.

[0197] (2) Linking joints: The linking joints of the clamping robot are used to connect the rotating connection points of the various parts of the clamping robot. They are like the elbow joints of the human hand and can achieve multiple degrees of freedom, allowing the clamping robot to rotate and bend in different directions. Generally, the clamping robot has multiple joints, and each joint can be controlled independently.

[0198] (3) Kinematic link: The kinematic link refers to a rigid rod that connects the joints of the clamping robot arm. Its length and shape can be designed and adjusted according to the actual working range and flexibility requirements, and the angles of each servo can be adjusted as needed.

[0199] (4) Robotic gripper 6; with a finger structure similar to that of a human, the industrial gripping robot arm gripper has each finger joint end connected by a small servo, and the flexion and extension of the finger are controlled by a connecting rod, so that the fisting and stretching movements can be completed, thereby realizing the operation of grasping moving objects.

[0200] The control system of the gripper arm based on visual perception generates a wireless signal from the remote control to send the corresponding motion command to the main controller. After receiving the signal from the remote control, the main controller processes the signal and sends a motion signal to each slave controller. After receiving the motion command, the controller sends a pulse signal or motion command to the corresponding servo drive motor, and the drive motor drives the servo to move.

[0201] The clamping robot arm can be assembled through a mechanical steel structure; 3D software can also be used to draw a model diagram according to the size and hole position of each hardware module. The 3Done software is used, which has a simple interface, powerful functions, and relatively simple operation, making it very suitable for primary and secondary school students to learn and use. After drawing, use tools such as "chain fillet" and "single-click trimming" to modify and adjust the details. Use the software's move, rotate and other functions to adjust the position and angle of each module to complete the production of the three-dimensional model of the robot arm. Finally, use a cutting machine to cut the wood as the raw material and assemble it into the main structure of the robot arm.

[0202] The main controller uses the micro:bit launched by the British BBC as the main controller. It is an open source microcontroller for programming by teenagers. It has a compact appearance, two programmable buttons A and B on the front of the mainboard, a 5×5 programmable LED dot matrix display and connector, and a USB data cable connector, LED status light, reset button, battery pack cable connector, main control chip and sensor chip on the back. BBC provides an online programming platform Makecode. After programming, the program can be downloaded to the micro:bit mainboard through the USB data cable connector. The main controller is used in combination with the expansion board to expand the pins and facilitate connection with various hardware modules.

[0203] At the same time, the gripping robot can also be equipped with 5 MG996R servos and 1 micro switch module to realize the movement of the robot and grasp objects. The servo mainly includes a circuit board, a drive motor, a reducer and a position detection element; the micro switch is acted on the action reed by an external force through a transmission element, so that the fixed contact and the moving contact at the end are quickly connected or disconnected. Four small tires are installed on the base of the robot arm, and two N20 motors are used to drive the robot arm, so that the robot arm can move like a car. The flame sensor module is combined with an active buzzer and connected to the pin of the micro:bit. When a flame (760-1100nm light source) is detected, the signal is transmitted to the processor, and the processor makes the active buzzer sound an alarm according to the signal.

[0204] The programming tool used for program design in the main controller is the Makecode online programming platform officially recommended by micro:bit. It can be used to write programs in Python or JavaScript, or it can be programmed graphically, which is suitable for teenagers of different levels. Makecode mainly consists of three parts: simulation demonstration area, tool area, and programming area. The simulation demonstration area has a simulation function, which can simulate the effect of program running and display the corresponding results on the main board. The program design adopts the form of wireless transmission and reception. The transmitter and the receiver set any same "wireless communication group ID" or the same number. When the machine is turned on, the receiver can receive a series of instructions from the transmitter and run the corresponding program. The transmitter contains the relevant programs for controlling 2 N20 motors, 1 micro switch and 5 MG996R servos. The receiver contains the programs of all modules, which can be divided into 5 parts: fire alarm system (flame sensor and active buzzer), robotic arm part (4 MG996R servos), mechanical claw part (1 MG996R servo and 1 micro switch), robotic arm base part (2 N20 motors and small wheels) and display part (OLED display).

[0205] The angle adjustment of the gripping robot in each direction is achieved by controlling the corresponding four steering gears. The robot claw 6 adopts a two-finger structure, consisting of a steering gear and a micro switch module, which is used for a simple demonstration of gripping objects. The effect of gripping objects is as follows Figure 4 (a). The OLED display shows the angle and speed of the robot arm at this time ( Figure 4 (b)), where "degree: 45" means the current rotation angle of the servo is 45 degrees, and "speed: 0" means the current motor speed is 0, that is, the base of the robot arm is in a stationary state. In this design, the active buzzer is combined with the flame sensor module. When the flame sensor detects the presence of flames around, the active buzzer sounds an alarm.

[0206] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0207] The above-mentioned embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the embodiments of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A control method for a gripping robot arm based on visual perception, characterized in that: The method comprises: Acquire an initial image of the working area, and preprocess the initial image to obtain an image to be recognized; Performing feature recognition on the image to be recognized to obtain target object features; Analyze the characteristics of the target object to obtain the position information and posture information of the target object; Performing path planning for the end effector of the robotic arm according to the position information and posture information of the target object to obtain a motion trajectory of the end effector of the robotic arm; the end effector is used to clamp the target object; According to the motion trajectory, a robotic arm inverse kinematics algorithm is used to calculate a robotic arm joint angle sequence corresponding to each trajectory point in the motion trajectory; Based on the joint angle sequence of the robotic arm, an execution instruction is generated, and the execution instruction is used to control the robotic arm to move to a target shape to clamp the target object.

2. The method according to claim 1, characterized in that The step of performing feature recognition on the image to be recognized to obtain target object features includes: The image to be identified is input into the trained convolutional neural network, and the category information C of the target object is output. n } and the bounding box coordinate information of the target object in the image to be identified Among them, c i represents the i-th possible target category, i = 1, 2, ..., n, is the coordinate of the upper left corner of the bounding box of the jth target, is the coordinate of the lower right corner of the bounding box of the jth target object, and m is the number of detected targets; According to the preset target category priority list P = {p1, p2, ..., p N }, filter out the target category C with the highest priority from the identified category information C target and its corresponding bounding box coordinates Among them, p k is the target object category and has a corresponding priority value, k = 1, 2, ..., N; The bounding box coordinates as the target feature.

3. The method according to claim 2, characterized in that The obtaining of the position information and the posture information of the target object based on the target object features includes: S1: based on the image bounding box coordinates of the target object with the highest priority, obtaining the coordinate information of the target object with the highest priority in the camera coordinate system; S2: Converting the coordinate information in the camera coordinate system into coordinate information in the world coordinate system as the position information; S3: Calculate the posture parameter of the target object with the highest priority in the world coordinate system as the posture information.

4. The method according to claim 3, characterized in that The obtaining, based on the image bounding box coordinates of the target object with the highest priority, coordinate information of the target object with the highest priority in the camera coordinate system includes: S11: Obtain the camera's intrinsic parameter matrix K and extrinsic parameter matrix R through the camera calibration method ext , T ext ; in, f x and f y are the focal lengths of the camera in the x and y directions, c x and c y are the coordinates of the image center; R ext Represents the rotation relationship of the camera relative to the world coordinate system, T ext Characterize the translation relationship of the camera relative to the world coordinate system; S12: Based on the bounding box coordinates of the target object with the highest priority in the image By formula and Calculate the coordinates of the center of the target object with the highest priority in the image coordinate system (u c ,v c ); S13: According to the camera model formula Calculate the coordinates (X c ,Y c ,Z c ); The converting the coordinate information in the camera coordinate system into the coordinate information in the world coordinate system comprises: converting the coordinate information in the camera coordinate system into the coordinate information in the world coordinate system according to the coordinate (X c ,Y c ,Z c ), using the coordinate transformation formula Get the position coordinates (X w ,Y w ,Z w ); The calculation of the posture parameters of the target object with the highest priority in the world coordinate system as the posture information includes: based on the Euler angle conversion formula, using the rotation matrix Calculate the posture parameters of the target object with the highest priority in the world coordinate system in, 5. The method according to claim 3, characterized in that: The path planning algorithm is based on the position information and posture information of the target object to obtain the motion trajectory of the end effector of the robot arm, including: Get the current joint angle information of the robot arm {α1, α2, ..., α l }; where l is the number of joints of the robot arm, α q is the qth joint, q=1,2,…,l; Using the robot arm kinematics model, according to the current joint angle information {α1, α2, …, α l }, calculate the current position coordinates of the end effector of the robot arm in the world coordinate system and current posture parameters Using the A* algorithm and the geometric model collision detection algorithm, the current joint angle information {α1, α2, …, α l }、Current location coordinates and the current posture parameters The determined robot arm configuration is used as the starting node, which will enable the end effector to reach the target position (X w ,Y w ,Z w ) and the posture parameters are The robot arm configuration is used as the target node to obtain the motion trajectory of the robot arm; The motion trajectory is a series of intermediate position points of the end effector of the robot arm. And the corresponding posture parameters s is the number of trajectory points, h = 1, 2,…, s.

6. The method according to claim 5, characterized in that The method of calculating the robot arm joint angle sequence corresponding to each trajectory point using a robot arm inverse kinematics algorithm according to the motion trajectory includes: For each trajectory point h, set the joint angle of the robot arm to the estimated value According to the forward kinematics equation of the robot arm, using the preset guess value Calculate the position and posture of the end effector of the robot arm Calculate the error of the end effector position and attitude Calculate the Jacobian matrix J of the forward kinematics equation at the current joint angle; wherein the element J of the Jacobian matrix J ab represents the partial derivative of the ath position or posture component of the end effector with respect to the bth joint angle; By iterative formula Update the estimated value of the joint angle; where g is the number of iterations; When the norm of the error of the end effector position and attitude is When it is less than the preset threshold ∈, the convergence condition is met and the current joint angle The robot arm joint angle sequence of the trajectory point h is confirmed.

7. A control system for a gripping robot arm based on visual perception, characterized in that: The system comprises: An image acquisition module is used to acquire an initial image of the working area and preprocess the initial image to obtain an image to be recognized; A visual recognition module is used to perform feature recognition on the image to be recognized to obtain target object features; A target object information acquisition module, used to analyze the target object characteristics and obtain the position information and posture information of the target object; A motion trajectory planning module, used for planning the path of the end effector of the robot arm according to the position information and posture information of the target object, so as to obtain the motion trajectory of the end effector of the robot arm; the end effector is used for clamping the target object; A robot arm joint angle calculation module is used to calculate the robot arm joint angle sequence corresponding to each trajectory point in the motion trajectory using a robot arm inverse kinematics algorithm according to the motion trajectory; An execution instruction generation module is used to generate an execution instruction based on the joint angle sequence of the robotic arm, and the execution instruction is used to control the robotic arm to move to a target shape to clamp the target object.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A clamping robot arm, characterized in that: Applicable to the control method of the gripping robot arm based on visual perception as described in any one of claims 1 to 6; It also includes: a support base, an A-arm motion link, a B-arm motion link, a C-arm motion link, a D-arm motion link and a mechanical gripper; The bottom end of the A-arm motion link is fixedly connected to the support base, and one side of the support base is connected with an A rotation driving component for driving the support base to rotate around the axis of the support base itself; the top end of the A-arm motion link is rotationally connected to the B-arm motion link through the A link joint, and one side of the B-arm motion link is connected with a B rotation driving component for driving the B-arm motion link to rotate around the axis of the A link joint; the end of the B-arm motion link away from the A-arm motion link is rotationally connected to the C-arm motion link through the B link joint, and one side of the C-arm motion link is connected with the B rotation driving component for driving the C-arm motion link to rotate around the axis of the B link joint; the end of the C-arm motion link away from the B-arm motion link is rotationally connected to the D-arm motion link through the C link joint, and one side of the D-arm motion link is connected with a C rotation driving component for driving the D-arm motion link to rotate around the axis of the C link joint; the robotic claw is fixedly installed on the D-arm motion link and is located at the end of the D-arm motion link away from the C-arm motion link.

Images