Manipulator control system based on visual identification and touch sensing

Abstract

The invention provides a manipulator control system based on visual identification and touch sensing, and relates to the technical field of manipulator control. Comprising the steps that a manipulator control module obtains joint parameters, constructs a multi-finger connecting rod model and obtains a control parameter instruction; and the visual identification module analyzes the underwater target image by adopting an SSD300 algorithm to obtain the position and shape features of the object. The path optimization module controls grabbing according to the control parameter instruction, adjusts the instruction according to the pressure distribution data to obtain a feedback control signal, and optimizes the path according to the object characteristics. And the control module operates the manipulator, and the underwater target object is grabbed according to the feedback control signal and the optimized path. According to the method, through the generated control parameter instruction, the control precision of the manipulator can be improved, the manipulator can reach the target position more accurately and conduct grabbing operation, meanwhile, the SSD300 algorithm is used for introducing underwater target detection, and underwater target detection and positioning can be achieved with high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of manipulator control, and in particular to a manipulator control system based on visual recognition and tactile sensing. Background Art

[0002] As one of the key components of an underwater robot, the performance of the manipulator has a decisive impact on the execution of underwater operation tasks. However, achieving precise and safe grasping in an underwater environment remains a challenge. Traditional underwater rigid manipulators usually adopt a rigid link structure or a gripper mechanism. Although they have the advantages of simple structure, convenient operation, high precision, large clamping force, and good reliability, due to the lack of flexibility, they have poor adaptability to the grasped object, are prone to damage it, and the operation scenarios are mostly limited to specific occasions, making it difficult to meet the grasping requirements of target objects with different shapes and sizes. In addition, the degrees of freedom of the rigid manipulator are limited and the flexibility is insufficient, making it difficult to achieve fine control of the grasping position and force, which is particularly prominent in underwater operations that require gentle operation to avoid damage.

[0003] With the increasing growth and diversification of underwater operation requirements, the limitations of traditional rigid manipulators have become increasingly prominent. There is an urgent need for a new type of underwater grasping tool, and the flexible underactuated manipulator has thus emerged. It has the characteristics of good dexterity and strong flexibility, and can achieve precise control of the grasping position and grasping force. This kind of manipulator can adapt to the grasping of objects of any shape, and at the same time avoid damaging the grasped object, especially showing great potential in scenarios that require fine operation such as underwater archaeology and biological sampling.

[0004] In the field of underwater operations, the research purpose of the flexible underactuated manipulator is to develop a grasping tool that can adapt to different underwater environments and task requirements. This kind of manipulator can achieve precise control of the grasping force and position while maintaining high flexibility to meet the high requirements for precision and safety in underwater operations. In short, the research on the flexible underactuated manipulator can not only promote the development of underwater robot technology, but also expand the application fields of underwater operations, improve the operation efficiency and safety, and is of great significance for promoting marine scientific research and the development and utilization of marine resources.

[0005] In summary, there is an urgent need for a manipulator control system based on visual recognition and tactile sensing to solve the above problems. Summary of the Invention

[0006] The present invention provides a manipulator control system based on visual recognition and tactile sensing to solve the defects of poor control accuracy of the manipulator and difficulty in identifying and positioning underwater target objects in the prior art.

[0007] On the one hand, the present invention provides a manipulator control system based on visual recognition and tactile sensing, including:

[0008] The manipulator control module obtains the joint parameters of the manipulator, constructs a multi-finger linkage model based on D-H coordinate transformation, inputs the joint parameters, and recursively obtains the mathematical relationship between the joints and the fingertip positions of each finger through homogeneous coordinate transformation relations, and analyzes the mathematical relationship to obtain control parameter instructions.

[0009] The vision recognition module is used to obtain the image data of the underwater target object, and uses the SSD300 algorithm to analyze the image data to obtain the object position and shape features.

[0010] The path optimization module is used to control the manipulator to grasp the underwater target object according to the control parameter instructions, obtain the operation path and pressure distribution data, adjust the control parameter instructions according to the pressure distribution data to obtain a feedback control signal, and optimize the operation path according to the object position and shape features to obtain an optimized path.

[0011] The control module is used to operate the manipulator to grasp the underwater target object according to the feedback control signal and the optimized path.

[0012] For a manipulator control system based on vision recognition and tactile sensing provided by the present invention, the steps of constructing a multi-finger linkage model include:

[0013] Establish a coordinate system for each joint, determine the origin according to the intersection point of the common normal line between the z-axis and different joints, define the x-axis according to the common normal line direction between different joints, and determine the y-axis according to the right-hand rule.

[0014] Calculate the D-H transformation matrix of each joint according to the D-H parameters.

[0015] For each finger of the manipulator, starting from the base joint, successively multiply the D-H transformation matrix of each joint to obtain the pose matrix of the end effector relative to the coordinate system.

[0016] Repeat the iteration, calculate the pose matrix of each end effector, and integrate them to form a multi-finger linkage model.

[0017] For a manipulator control system based on vision recognition and tactile sensing provided by the present invention, the steps of obtaining the mathematical relationship include:

[0018] Recursively obtain the homogeneous transformation matrix for the pose matrix of each finger according to the homogeneous coordinate transformation relationship.

[0019] Extract the first three elements of the fourth column from the homogeneous transformation matrix as the position information of the target coordinate system relative to the coordinate system.

[0020] Take the three-by-three sub-matrix formed by the first three columns in the homogeneous transformation matrix as the rotation matrix, and calculate the attitude parameters through the elements of the rotation matrix.

[0021] Repeat the above steps to obtain the position information and attitude parameters of each finger of the manipulator, and integrate them to obtain the mathematical relationship between the joints and the fingertip positions of each finger.

[0022] According to a manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of obtaining the control parameter instructions include:

[0023] Compare the position information and attitude parameters with the preset control target, and calculate the position deviation and attitude deviation.

[0024] According to the position deviation and attitude deviation, use the numerical iteration method to solve the parameters that need to be adjusted for each joint, and organize them to obtain the adjusted joint parameters.

[0025] Adjust the adjusted joint parameters in terms of joint angle, speed, and acceleration until they meet the preset range.

[0026] According to each joint parameter and combined with the Lagrange equation, calculate the torque that needs to be applied to each joint after adjustment.

[0027] Convert the adjusted joint parameters and torque into the corresponding control instruction format according to the preset interface requirements and communication protocol to obtain the control parameter instructions.

[0028] According to a manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of calculating the torque include:

[0029] Calculate the kinetic energy of the rotating joint according to the moment of inertia and angular velocity to obtain the kinetic energy of the rotating joint, calculate the kinetic energy of the moving joint according to the mass and linear velocity to obtain the kinetic energy of the moving joint, and add the kinetic energy of each rotating joint and moving joint of the manipulator to obtain the total joint kinetic energy.

[0030] According to the different relative positions of each part of the manipulator, calculate the corresponding gravitational potential energy for each joint and the connected parts.

[0031] Substitute the total joint kinetic energy and gravitational potential energy into the definition of the Lagrange equation to obtain the Lagrangian function.

[0032] Take the partial derivative of the Lagrangian function with respect to the generalized velocity to obtain the joint generalized velocity expression, and use the chain rule to take the total derivative of the joint generalized velocity expression with respect to time to obtain the joint time expression.

[0033] Substitute the joint generalized velocity expression and joint time expression into the Lagrange equation, and through algebraic operations, solve for the torque applied to each joint.

[0034] A manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of analyzing and obtaining the object position and shape features include:

[0035] Use histogram equalization to improve the contrast of the image in the image data, and use Gaussian filtering to remove noise to obtain image processing data.

[0036] Adjust the image processing data to the size that meets the requirements of the SSD300 algorithm, and perform normalization processing on the pixel values to obtain migration data. Utilize the feature extraction advantage of the convolutional neural network to construct a style transfer neural network, input the image data, and output the style transfer image data.

[0037] Use a deep learning framework to construct an SSD300 model, and adopt a relative ratio strategy to associate and generate preset boxes, so that the feature units on different feature layers are associated with different preset boxes. Input the style transfer image data, and calculate multiple bounding boxes through forward propagation.

[0038] Use the non-maximum suppression algorithm to retain the bounding box with the highest confidence according to the confidence of the bounding box, and suppress other bounding boxes with a high degree of overlap with it to obtain the optimal bounding box. Obtain the object position and shape features according to the coordinate information of the optimal bounding box.

[0039] A manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of obtaining the image processing data include:

[0040] Obtain the corresponding gray value by weighted averaging the three channels of the color image in the image data, count the frequency of each gray value in the image data, and obtain the gray histogram.

[0041] Calculate the cumulative distribution function according to the gray histogram, and map the corresponding gray value to a new gray value through the cumulative distribution function to obtain the histogram equalization image.

[0042] For each pixel in the histogram equalization image, calculate the weighted average value of the pixels in the neighborhood according to the preset Gaussian filter parameters, and use it as the new value of the current pixel, so as to obtain the image processing data.

[0043] A manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of obtaining the feedback control signal include:

[0044] Perform normalization processing on the pressure distribution data, map the pressure value to a preset range, and perform data smoothing processing to remove outliers.

[0045] Extract features that can reflect the stability of grasping and the tightness of contact from the processed pressure distribution data as key features.

[0046] According to the key features, determine the adjustment strategy of the control parameter instruction. According to the adjustment strategy, calculate the adjustment amount of the stored control parameter instruction, and combine the adjustment amount with the control parameter instruction to generate a feedback control signal.

[0047] According to a manipulator control system based on visual recognition and tactile sensing provided by the present invention, the steps of optimizing to obtain an optimized path include:

[0048] Obtain the environmental data around the underwater target object, and screen out the obstacle factor data that will affect the operation path from it.

[0049] According to the shape features, further calculate the shape parameters that can describe the underwater target object, and combine the structure of the manipulator to determine the grasping part and the grasping posture.

[0050] Adjust the operation path according to the obstacle factor data, and combine the grasping part and the grasping posture to obtain an optimized path.

[0051] According to a manipulator control system based on visual recognition and tactile sensing provided by the present invention, the control module includes:

[0052] A receiving unit for receiving the feedback control signal and the optimized path from the path optimization module.

[0053] A processing unit for grasping the underwater target object according to the feedback control signal and the optimized path.

[0054] A manipulator control system based on visual recognition and tactile sensing provided by the present invention also obtains the joint parameters of the manipulator, constructs a multi-finger link model based on D-H coordinate transformation, inputs the joint parameters, and recursively obtains the mathematical relationship between each joint of each finger and the fingertip position through homogeneous coordinate transformation relations, and analyzes the mathematical relationship to obtain control parameter instructions. It solves the problem of how to accurately control the movement of each joint of the manipulator to achieve accurate grasping of the target object, can accurately control the movement of the manipulator, and improve the accuracy and flexibility of grasping. By acquiring the image data of the underwater target object and using the SSD300 algorithm to analyze the image data, the position and shape characteristics of the object are obtained, solving the problem of how to accurately identify and locate the target object in the underwater environment, improving the accuracy and efficiency of target object recognition, and being able to quickly locate the target in a complex underwater environment. It also controls the manipulator to grasp the underwater target object according to the control parameter instructions, obtains the operation path and pressure distribution data, adjusts the control parameter instructions according to the pressure distribution data to obtain a feedback control signal, and optimizes the operation path according to the object position and shape characteristics to obtain an optimized path. It solves the problem of how to optimize the grasping path of the manipulator to improve the grasping efficiency and success rate. Through path optimization and feedback control, the grasping efficiency and success rate of the manipulator are improved, and unnecessary energy consumption is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 FIG. is a schematic structural diagram of a manipulator control system based on visual recognition and tactile sensing provided by an embodiment of the present invention;

[0057] Figure 2 FIG. is a schematic flow diagram of a manipulator control system based on visual recognition and tactile sensing provided by an embodiment of the present invention;

[0058] Figure 3 FIG. is a schematic flow diagram of a manipulator control system based on visual recognition and tactile sensing provided by an embodiment of the present invention;

[0059] Figure 4 FIG. is a schematic flow diagram of a manipulator control system based on visual recognition and tactile sensing provided by an embodiment of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] The following combines Figures 1-4 to describe a manipulator control system based on visual recognition and tactile sensing of the present invention.

[0062] As Figure 1 shown, a manipulator control system based on visual recognition and tactile sensing provided by an embodiment of the present invention includes:

[0063] As Figure 2 shown, a manipulator control module obtains the joint parameters of the manipulator, constructs a multi-finger link model based on D-H coordinate transformation, inputs the joint parameters, and recursively obtains the mathematical relationship between each joint and the fingertip position of each finger through homogeneous coordinate transformation relationships, and analyzes the mathematical relationship to obtain control parameter instructions.

[0064] The steps of constructing the multi-finger link model include:

[0065] Establish a coordinate system for each joint, determine the origin according to the intersection point of the common normal line between the z-axis and different joints, define the x-axis according to the common normal line direction between different joints, and determine the y-axis according to the right-hand rule.

[0066] Calculate the D-H transformation matrix of each joint according to the D-H parameters, and the formula is expressed as:

[0067]

[0068] In the formula, θ i , α i , a i , d i are D-H parameters.

[0069] For each finger of the manipulator, starting from the base joint, multiply the D-H transformation matrix of each joint in turn to obtain the pose matrix of the end effector relative to the coordinate system.

[0070] Repeat the iteration, calculate the pose matrix of each end effector, and integrate them to form a multi-finger link model.

[0071] The steps of obtaining the mathematical relationship include:

[0072] Recursively obtain the homogeneous transformation matrix according to the homogeneous coordinate transformation relationship for the pose matrix of each finger.

[0073] Extract the first three elements of the fourth column from the homogeneous transformation matrix as the position information of the target coordinate system relative to the coordinate system.

[0074] Take the three-by-three submatrix formed by the first three columns in the homogeneous transformation matrix as the rotation matrix, and calculate the attitude parameters through the elements of the rotation matrix.

[0075] Repeat the above steps to obtain the position information and attitude parameters of each finger of the manipulator, and integrate them to obtain the mathematical relationship between each joint and the fingertip position of each finger.

[0076] The steps to obtain the control parameter instructions include:

[0077] Compare the position information and attitude parameters with the preset control target, and calculate the position deviation and attitude deviation.

[0078] According to the position deviation and attitude deviation, use the numerical iteration method to solve the parameters that need to be adjusted for each joint, and organize them to obtain the adjusted joint parameters.

[0079] Adjust the adjusted joint parameters from three aspects: joint angle, speed, and acceleration until they meet the preset range.

[0080] According to each joint parameter and combined with the Lagrange equation, calculate the torque that needs to be applied to each joint after adjustment.

[0081] The steps to calculate the torque include:

[0082] Calculate the kinetic energy of the rotating joint according to the moment of inertia and angular velocity to obtain the kinetic energy of the rotating joint, calculate the kinetic energy of the moving joint according to the mass and linear velocity to obtain the kinetic energy of the moving joint, and add the kinetic energy of each rotating joint and moving joint of the manipulator to obtain the total joint kinetic energy.

[0083] According to the different relative positions of each part of the manipulator, calculate the corresponding gravitational potential energy for each joint and the connected parts.

[0084] Substitute the total joint kinetic energy and gravitational potential energy into the definition of the Lagrange equation to obtain the Lagrangian function.

[0085] Take the partial derivative of the Lagrangian function with respect to the generalized velocity to obtain the joint generalized velocity expression, and use the chain rule to take the total derivative of the joint generalized velocity expression with respect to time to obtain the joint time expression.

[0086] Substitute the joint generalized velocity expression and the joint time expression into the Lagrange equation, and through algebraic operations, solve for the torque applied to each joint.

[0087] The adjusted joint parameters and torques are converted into corresponding control parameter instructions in the format of control instructions according to the preset interface requirements and communication protocols.

[0088] As Figure 3 shown, the visual recognition module is used to obtain the image data of the underwater target object and analyze the image data using the SSD300 algorithm to obtain the object position and shape features.

[0089] The steps of analyzing to obtain the object position and shape features include:

[0090] Use histogram equalization to improve the contrast of the image in the image data, and use Gaussian filtering to remove noise to obtain image processing data.

[0091] The steps of obtaining the image processing data include:

[0092] The corresponding gray values are obtained by weighted averaging the three channels of the color image in the image data, the frequency of each gray value in the image data is counted to obtain a gray histogram.

[0093] Calculate the cumulative distribution function according to the gray histogram, and map the corresponding gray values to new gray values through the cumulative distribution function to obtain a histogram equalized image.

[0094] For each pixel in the histogram equalized image, according to the preset Gaussian filter parameters, calculate the weighted average value of the pixels in the neighborhood as the new value of the current pixel, thereby obtaining the image processing data.

[0095] Adjust the image processing data to the size that meets the requirements of the SSD300 algorithm, and perform normalization processing on the pixel values to obtain migration data. Utilize the feature extraction advantage of the convolutional neural network to construct a style transfer neural network, input the image data, and output the style transfer image data.

[0096] Use a deep learning framework to construct an SSD300 model, and adopt a relative ratio strategy to associate and generate preset boxes, so that the feature units on different feature layers are associated with different preset boxes. Input the style transfer image data, and calculate multiple bounding boxes through forward propagation. The bounding boxes include the class probability of the target object and the position information of the object in the image.

[0097] Use the non-maximum suppression algorithm to retain the bounding box with the highest confidence according to the confidence of the bounding boxes, and suppress other bounding boxes with a high degree of overlap with it to obtain the optimal bounding box. Obtain the object position and shape features according to the coordinate information of the optimal bounding box.

[0098] As Figure 4As shown in the figure, the path optimization module is used to control the manipulator to grasp the underwater target object according to the control parameter instructions, obtain the operation path and pressure distribution data, adjust the control parameter instructions according to the pressure distribution data to obtain the feedback control signal, and optimize the operation path according to the object position and shape characteristics to obtain the optimized path.

[0099] The method for obtaining the pressure distribution data: A tactile sensor is used. The tactile sensor uses a resistive film pressure sensor. A model is selected that feedbacks the magnitude of the acting force based on the piezoresistive effect. The resistive flexible pressure sensor converts pressure into resistance changes and is placed at the proximal phalanx, middle phalanx, and distal phalanx. According to the definition of resistance:

[0100]

[0101] In the formula, ρ is the resistivity, L is the length, S is the cross-sectional area, and R is the resistance.

[0102] The sensing mechanism of the resistive pressure sensor is simple, the preparation process is simple, and the energy consumption is small, which has received extensive attention and research. The allowable pressure range of the selected resistive pressure sensor is 20 g to 6 kg. The magnitude of the acting force is feedback through the negative correlation between the acting force and the pressure. It has a millimeter-level response speed, and a waterproof sensor is selected, which is suitable for the working environment of the manipulator underwater.

[0103] The sensing mechanism of the resistive flexible pressure sensor: When no external pressure is applied, the network structure is in a natural relaxed state, and there are few contact points between the internal network skeleton fibers. At this time, the resistance value of the resistive flexible pressure sensor is the largest. When an external pressure is applied, within a lower pressure range, the internal micropores are compressed, and the skeleton fibers come into contact with each other, forming more conductive paths, and the resistivity drops significantly, resulting in a rapid decrease in the resistance value. Therefore, the resistive flexible pressure sensor has high sensitivity and high linearity within the low pressure range. When the pressure reaches a certain magnitude, the micropores inside the material basically disappear. At this time, the sponge can be approximately equivalent to a solid elastomer. The resistive flexible pressure sensor is compressed by the pressure and its thickness decreases. At this time, the resistance value of the material basically remains unchanged. The main mechanism of the resistance change is to decrease with the decrease of its thickness. Therefore, the sensitivity of the resistive flexible pressure sensor is low within the high pressure range. The change in sensitivity coincides with the stress-strain curve of the composite conductive sponge above. The strain changes greatly under unit pressure within the low pressure range and changes little within the high pressure range.

[0104] The steps for obtaining the feedback control signal include:

[0105] Perform normalization processing on the pressure distribution data, map the pressure values to a preset range, and perform data smoothing processing to remove outliers.

[0106] Extract features that can reflect the stability of grasping and the tightness of contact from the processed pressure distribution data as key features. The key features include the average pressure value, the maximum pressure value and its position, and the uniformity index of the pressure distribution.

[0107] According to the key features, determine the adjustment strategy of the control parameter instruction. According to the adjustment strategy, calculate the adjustment amount of the stored control parameter instruction, and combine the adjustment amount with the control parameter instruction to generate a feedback control signal. When it is found that the pressure distribution is uneven, adjust the posture or position of the fingers to improve the contact situation. If the pressure value is too small, increase the grasping force. The adjustment amount can be the change amount of the finger joint angle that needs to be adjusted according to the uneven pressure distribution.

[0108] The steps for optimizing to obtain the optimized path include:

[0109] Obtain the environmental data around the underwater target object, and screen out the obstacle factor data that will affect the operation path from it. The obstacle factor data includes whether there are obstacles, water flow speed and direction, etc.

[0110] According to the shape features, further calculate the shape parameters that can describe the underwater target object, and combine the structure of the manipulator to determine the grasping part and the grasping posture. If the object has specific orientation requirements (such as the pin direction of an electronic component), adjust the path so that the manipulator has a suitable posture when reaching the grasping position. The end point of the path can be adjusted through rotation and translation operations to meet the requirements of the grasping posture.

[0111] Adjust the operation path according to the obstacle factor data, and combine the grasping part and the grasping posture to obtain the optimized path.

[0112] The control module is used to operate the manipulator to grasp the underwater target object according to the feedback control signal and the optimized path.

[0113] The control module includes:

[0114] A receiving unit for receiving the feedback control signal and the optimized path from the path optimization module. The receiving unit is equipped with specific communication interfaces and signal processing circuits, and can establish a stable and reliable connection with the path optimization module. When the path optimization module completes the optimization of the operation path and generates a feedback control signal based on the analysis of various aspects such as the object position, shape features, and environmental factors, the receiving unit will capture these signals in real time and accurately. The feedback control signal contains the adjustment instructions for the motion parameters of each joint of the manipulator, such as the fine adjustment of the joint angle, the control parameters of speed and acceleration, etc. These signals are obtained through complex calculations and analyses, aiming to enable the manipulator to complete the grasping task of the underwater target object more efficiently and precisely.

[0115] The processing unit is used to grasp the underwater target object according to the feedback control signal and the optimized path. To ensure the accuracy and integrity of the received feedback control signal and the optimized path, the receiving unit also has the functions of signal processing and verification. It will filter the received signal to remove possible noise interference, and at the same time decode and analyze the signal, converting it into a format that the system can understand and process. In addition, the receiving unit will verify the data through a verification algorithm. Once it is found that there are errors or abnormalities in the data, it will send a request to the path optimization module in a timely manner, asking to resend the relevant information to ensure the normal operation of the system.

[0116] Example 1: Manipulator control module: Assume that the manipulator has 3 fingers, and each finger has 3 joints. Obtain the parameters of each joint. For example, the angle of joint 1 is 30°, the joint length is 0.1 m, the twist angle is zero degree, the offset is zero, etc. (other joint parameters are set similarly). Based on the D-H coordinate transformation, a multi-finger link model is constructed. After inputting the parameters, the mathematical relationship between each joint and the fingertip position is obtained by recursive homogeneous coordinate transformation. Through analysis and calculation, control parameter instructions are obtained. For example, the target angle of joint 1 is adjusted to 35°.

[0117] Vision recognition module: Use an underwater camera to obtain image data of the underwater target object (assumed to be a cylinder). The SSD300 algorithm is used to analyze the image, and the position coordinates of the object in the image coordinate system are obtained as (100, 150) (pixel values). According to the camera calibration parameters, the approximate position in the world coordinate system is (0.5 m, 0.3 m, 0.2 m). At the same time, the shape of the object is recognized as a cylinder, with a radius (r = 0.05 m) and a height (h = 0.15 m).

[0118] Path optimization module: Control the manipulator to grasp the target object according to the control parameter instructions of the manipulator control module, and obtain the initial operation path and pressure distribution data. Assume that during the grasping process, the pressure distribution of the contact between finger 1 and the object is [10 N, 12 N, 8 N] (pressure values at different positions). Adjust the control parameter instructions according to the pressure distribution data to obtain a feedback control signal. For example, increase the torque of joint 2 of finger 1. At the same time, optimize the operation path according to the object position and shape characteristics. In the original path, the manipulator would pass through a virtual obstacle, and after optimization, the path avoids this obstacle.

[0119] Control module: Operate the manipulator to grasp the underwater target object according to the feedback control signal and the optimized path. Finally, the manipulator successfully grasps the target object, verifying the effectiveness of the system.

[0120] A manipulator control system based on visual recognition and tactile sensing provided in this embodiment can accurately describe the movement of the manipulator and generate accurate control parameter instructions through a modeling and control method based on D-H coordinate transformation, thereby improving the control accuracy of the manipulator and enabling it to reach the target position more accurately and perform grasping operations. By introducing the SSD300 algorithm into underwater target detection, the detection and positioning of underwater targets can be achieved with high precision. The operation path is optimized according to the object position and shape features, and the control parameter instructions are adjusted in combination with the pressure distribution data, making the operation path of the manipulator more reasonable, avoiding the collision risk, and improving the grasping efficiency and success rate. And it can adjust the path in real time according to the actual situation to adapt to different underwater environments and target objects. By analyzing the pressure distribution data to adjust the control parameter instructions, the accurate control of the grasping force of the manipulator can be realized, which not only ensures that the object can be stably grasped, but also avoids damaging the object, improving the quality and reliability of the grasping.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A manipulator control system based on visual recognition and tactile sensing, characterized in that, Including: A manipulator control module, which acquires the joint parameters of the manipulator, constructs a multi-finger link model based on D-H coordinate transformation, inputs the joint parameters, and recursively obtains the mathematical relationship between the joints and the fingertip positions of each finger through homogeneous coordinate transformation relationships, and analyzes the mathematical relationship to obtain control parameter instructions; A vision recognition module, which is used to acquire the image data of the underwater target object and analyze the image data using the SSD300 algorithm to obtain the object position and shape features; A path optimization module, which is used to control the manipulator to grasp the underwater target object according to the control parameter instructions, obtain the operation path and pressure distribution data, adjust the control parameter instructions according to the pressure distribution data to obtain a feedback control signal, and optimize the operation path according to the object position and the shape features to obtain an optimized path; A control module, which is used to operate the manipulator to grasp the underwater target object according to the feedback control signal and the optimized path.

2. The manipulator control system based on visual recognition and tactile sensing according to claim 1, characterized in that, The steps of constructing the multi-finger link model include: Establish a coordinate system for each joint, determine the origin according to the intersection point of the common normal line between the z-axis and different joints, define the x-axis according to the common normal line direction between different joints, and determine the y-axis according to the right-hand rule; Calculate the D-H transformation matrix of each joint according to the D-H parameters; For each finger of the manipulator, starting from the base joint, multiply the D-H transformation matrix of each joint in turn to obtain the pose matrix of the end effector relative to the coordinate system; Repeat the iteration, calculate the pose matrix of each end effector, and integrate them to form the multi-finger link model.

3. The manipulator control system based on visual recognition and tactile sensing according to claim 1, characterized in that, The steps of obtaining the mathematical relationship include: Recursively obtain the homogeneous transformation matrix for the pose matrix of each finger according to the homogeneous coordinate transformation relationship; Extract the first three elements of the fourth column from the homogeneous transformation matrix as the position information of the target coordinate system relative to the coordinate system; Take the three-by-three submatrix composed of the first three columns in the homogeneous transformation matrix as the rotation matrix, and calculate the attitude parameters through the elements of the rotation matrix; Repeat the above steps to obtain the position information and attitude parameters of each finger of the manipulator, and integrate them to obtain the mathematical relationship between the joints and the fingertip positions of each finger.

4. The manipulator control system based on visual recognition and tactile sensing according to claim 3, characterized in that, The steps of obtaining the control parameter instructions include: Compare the position information and attitude parameters with the preset control target, and calculate the position deviation and attitude deviation; According to the position deviation and attitude deviation, use the numerical iteration method to solve the parameters that need to be adjusted for each joint, and organize them to obtain the adjusted joint parameters; Adjust the adjusted joint parameters from three aspects of joint angle, speed and acceleration until they meet the preset range; According to the joint parameters and combined with the Lagrangian equation, calculate the torque that needs to be applied to each joint after adjustment; 5. The manipulator control system based on visual recognition and tactile sensing according to claim 4, characterized in that, Convert the adjusted adjusted joint parameters and the torque into the corresponding control instruction format according to the preset interface requirements and communication protocol to obtain the control parameter instructions. The steps of calculating the torque include: Calculate the kinetic energy of the rotating joint according to the moment of inertia and angular velocity to obtain the kinetic energy of the rotating joint, calculate the kinetic energy of the prismatic joint according to the mass and linear velocity to obtain the kinetic energy of the prismatic joint, and add the kinetic energy of each rotating joint and the kinetic energy of the prismatic joint of the manipulator to obtain the total joint kinetic energy; According to the different relative positions of the parts of the manipulator, calculate the corresponding gravitational potential energy for each joint and the connected parts; Substitute the total joint kinetic energy and the gravitational potential energy into the definition of the Lagrangian equation to obtain the Lagrangian function; Take the partial derivative of the Lagrangian function with respect to the generalized velocity to obtain the joint generalized velocity expression, and use the chain rule to take the total derivative of the joint generalized velocity expression with respect to time to obtain the joint time expression; Substitute the joint generalized velocity expression and the joint time expression into the Lagrangian equation, and through algebraic operations, solve for the torques applied to each joint.

6. The control system of a manipulator based on visual recognition and tactile sensing according to claim 1, characterized in that, The steps of analyzing to obtain the position and shape features of the object include: Use histogram equalization to improve the contrast of the image in the image data, and use Gaussian filtering to remove noise to obtain image processing data; Adjust the image processing data to the size that meets the requirements of the SSD300 algorithm, and perform normalization processing on the pixel values to obtain migration data. Utilize the feature extraction advantage of the convolutional neural network to construct a style transfer neural network, input the image data, and output the style transfer image data; Use a deep learning framework to construct an SSD300 model, and adopt a relative proportion strategy to associate and generate preset boxes, so that the feature units on different feature layers are associated with different preset boxes. Input the style transfer image data, and calculate multiple bounding boxes through forward propagation; Use the non-maximum suppression algorithm to retain the bounding box with the highest confidence according to the confidence of the bounding boxes, and suppress other bounding boxes with a high overlap degree with it to obtain the optimal bounding box. Obtain the position and shape features of the object according to the coordinate information of the optimal bounding box.

7. A manipulator control system based on visual recognition and tactile sensing according to claim 6, characterized in that, The steps of obtaining the image processing data include: Obtain the corresponding gray value by weighted averaging the three channels of the color image in the image data, and count the frequency of each gray value in the image data to obtain a gray histogram; Calculate the cumulative distribution function according to the gray histogram, and map the corresponding gray value to a new gray value through the cumulative distribution function to obtain a histogram equalized image; For each pixel in the histogram equalized image, calculate the weighted average of the pixels in the neighborhood according to the preset Gaussian filter parameters as the new value of the current pixel, thereby obtaining the image processing data.

8. A manipulator control system based on visual recognition and tactile sensing according to claim 1, characterized in that, The steps of obtaining the feedback control signal include: Perform normalization processing on the pressure distribution data, map the pressure value to a preset range, and perform data smoothing processing to remove outliers; Extract the features that can reflect the stability of grasping and the tightness of contact from the processed pressure distribution data as key features; According to the key features, determine the adjustment strategy of the control parameter instruction. According to the adjustment strategy, calculate the adjustment amount of the stored control parameter instruction, and combine the adjustment amount with the control parameter instruction to generate the feedback control signal.

9. A manipulator control system based on visual recognition and tactile sensing according to claim 1, characterized in that, The steps of optimizing to obtain the optimized path include: Obtain the environmental data around the underwater target object, and screen out the obstacle factor data that will affect the operation path from it; According to the shape features, further calculate the shape parameters that can describe the underwater target object, and combine the structure of the manipulator to determine the grasping part and the grasping posture Adjust the operation path according to the obstacle factor data, and combine the grasping part and the grasping posture to obtain the optimized path.

10. A manipulator control system based on visual recognition and tactile sensing according to claim 1, characterized in that, The control module includes: A receiving unit for receiving the feedback control signal and the optimized path from the path optimization module; A processing unit for grasping the underwater target object according to the feedback control signal and the optimized path.

Images