A tactile sensor, system and control method for a robotic arm

Abstract

This invention proposes a robotic arm tactile sensor, system, and robotic arm control method, belonging to the field of machine tactile and sensor technology. The robotic arm tactile sensor comprises, from bottom to top, a bottom rigid resin plate, a high-resistance layer, a low-resistance layer, and a neoprene rubber layer. When the neoprene rubber layer is subjected to downward pressure, a portion of the low-resistance fabric sheet in the low-resistance layer contacts the high-resistance layer, forming an equivalent circuit of a parallel resistor. By applying current excitation to the electrode pairs composed of adjacent electrodes on the rigid resin plate of the robotic arm tactile sensor and measuring the voltage values ​​of each electrode, the conductivity matrix of the object contacted by the robotic arm tactile sensor is obtained based on the principle of resistivity tomography. The robotic arm tactile sensor of this invention expands the tactile sensing range of the robotic arm. Its application in a robotic arm control system enables the robotic arm to perceive its surrounding environment more comprehensively and perform tasks more accurately and efficiently.

Description

Technical Field

[0001] This invention belongs to the field of machine tactile and sensor technology, specifically relating to a robotic arm tactile sensor, system, and robotic arm control method. Background Technology

[0002] Equipping robotic arms with tactile sensors enables them to perceive and measure information such as pressure, shape, and texture of objects. This allows for more precise control of force application, adapting the robotic arm to different grasping tasks and environments.

[0003] With the continuous advancement of machine perception technology, researchers are increasingly focusing on applying tactile perception to human-computer interaction technology. However, traditional tactile sensors are mostly dot matrix tactile sensors. Due to their complex manufacturing process, numerous electrodes, and wiring limitations, they have limitations in perceiving large areas and complex shapes. This makes it difficult for robotic arms to acquire global and effective tactile information like humans, hindering their ability to effectively complete tasks such as grasping. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a tactile sensor, system, and control method for a robotic arm. The tactile sensor of this invention employs electrical impedance tomography (EIT), which not only overcomes the limitations of traditional dot matrix structures, enabling tactile perception to cover a wider area, but also handles various sensing patterns required by the robotic arm, further enhancing its sensing capabilities and allowing it to better perform complex grasping tasks.

[0005] This invention provides a robotic arm tactile sensor, comprising, from bottom to top, a bottom resin rigid plate, a high-resistance layer, a low-resistance layer, and a neoprene rubber layer. The bottom resin rigid plate is non-conductive. Multiple electrodes are uniformly placed at the boundaries of the bottom resin rigid plate. A conductive graphite spray is applied to the surface of the bottom resin rigid plate to form a conductive graphite spray plane. Silver paste is applied to the contact area between the electrodes and the conductive graphite spray plane, and the conductive graphite spray plane and the electrodes together form the high-resistance layer. The low-resistance layer is an array of uniformly spaced low-resistance fabric sheets. The neoprene rubber layer is composed of neoprene rubber foam, and the fabric sheets of the low-resistance layer are adhered to the bottom of the neoprene rubber foam.

[0006] In a specific embodiment of the present invention, the robotic arm tactile sensor operates as follows: when the neoprene layer is not under pressure, if a current excitation is applied to two adjacent electrodes, the current flows through the high-resistivity layer and does not pass through the low-resistivity fabric sheet; when the neoprene layer is subjected to downward pressure, part of the low-resistivity fabric sheet contacts the high-resistivity layer, forming an equivalent circuit of a parallel resistor, thereby increasing the conductivity of the contact area between the high-resistivity layer and the low-resistivity layer; by sequentially applying current excitation to the electrode pairs composed of adjacent electrodes in the robotic arm tactile sensor and measuring the voltage value of each electrode, the conductivity matrix of the object contacted by the robotic arm tactile sensor is obtained based on the principle of resistivity tomography.

[0007] This invention also proposes a control circuit based on the aforementioned robotic arm tactile sensor, comprising: the robotic arm tactile sensor, a power supply module, a multiplexer, and a data acquisition and microcontroller module; the power supply module is connected to the multiplexer, the multiplexer is connected to the data acquisition and microcontroller module, and the data acquisition and microcontroller module is connected to the robotic arm tactile sensor; the power supply module is used to output a constant current; the multiplexer is used to connect the electrode pairs formed by selected adjacent electrodes on the robotic arm tactile sensor to inject a constant current; the data acquisition and microcontroller module includes a data acquisition submodule and a microcontroller unit; the data acquisition submodule is connected to each electrode of the robotic arm tactile sensor respectively, and is used to acquire the voltage signals output by each electrode of the robotic arm tactile sensor; the microcontroller unit is used to control the multiplexer to inject a constant current into the electrode pairs formed by adjacent electrodes on the robotic arm tactile sensor according to the adjacent excitation mode.

[0008] In one specific embodiment of the present invention, the control circuit further includes a voltage signal processing module, which uses MATLAB to convert the voltage signal output by the data acquisition submodule into a conductivity matrix.

[0009] In one specific embodiment of the present invention, the voltage signal processing module is further configured to map the conductivity matrix into a corresponding tactile perception image.

[0010] This invention also proposes a robotic arm control system based on the above-described control circuit, comprising: a human-computer interaction module, a vision module, a tactile module, and a robotic arm control module;

[0011] The human-machine interaction module is used to receive natural language commands issued by the operator through instant messaging software, then extract key information from the natural language commands and send it to the robotic arm control module. The key information includes the type, weight class and containment area of ​​the object to be grasped.

[0012] The robotic arm control module is used to send the category information of the grasped object to the vision module, and generate the spatial position coordinates of the grasped object based on the coordinate information of the center position of the object of the same category returned by the vision module, the conductivity matrix returned by the tactile module, and the weight level information; combined with the spatial position coordinates of the receiving area, the robotic arm is controlled to complete the grasping task.

[0013] The vision module is used to acquire environmental images of the robotic arm performing tasks. By performing target detection on the images, based on the received category information of the grasped object, it outputs the coordinate information of the center position of the object with the same category as the grasped object and sends it to the robotic arm control module.

[0014] The tactile module employs the control circuit, wherein the robotic arm tactile sensor is placed on the surface of the robotic arm. By placing objects of different weights and the same type of grasping object on the surface of the robotic arm tactile sensor, the control circuit obtains the conductivity matrix of the corresponding surface of the robotic arm tactile sensor and sends it to the robotic arm control module.

[0015] In one specific embodiment of the present invention, the instant messaging software is WeChat.

[0016] In one specific embodiment of the present invention, the extraction of key information from the natural language instruction adopts the UIE general information extraction model.

[0017] In one specific embodiment of the present invention, the target detection of the image is performed using the YoloX target detection model.

[0018] This invention also proposes a robotic arm control method based on the above-described robotic arm control system, comprising:

[0019] 1) The operator sends a natural language grasping task instruction to the robotic arm via instant messaging software to the human-machine interaction module. The human-machine interaction module uses the UIE model to extract the key information of the task instruction and sends it to the robotic arm control module. The key information includes the type, weight class and containment area of ​​the grasped object.

[0020] 2) The robotic arm control module sends the category information of the grasped object to the vision module. The vision module collects environmental images of the robotic arm working through the connected camera, and then uses the YoloX object detection model to mark the objects belonging to the category in the image and returns the center position coordinates to the robotic arm control module.

[0021] 3) By placing objects of different weights and the same type of grasping objects on the surface of the robotic arm tactile sensor in the tactile module, the conductivity of the objects on the surface of the robotic arm tactile sensor is measured based on the control circuit of the robotic arm tactile sensor, and the corresponding conductivity matrix is ​​returned to the robotic arm control module.

[0022] 4) The robotic arm control module obtains the final coordinates of the grasped object based on the coordinate information returned by the vision module, the conductivity matrix information returned by the tactile module, and the weight level information.

[0023] 5) The robotic arm control module controls the robotic arm to move to complete the grasping based on the position coordinates of the grasped object and the position coordinates of the receiving area.

[0024] The features and beneficial effects of this invention are as follows:

[0025] This invention proposes a large-area tactile sensor for robotic arms that overcomes the limitations of traditional dot matrix structures. Based on the concept of electrical impedance tomography, it adopts a sparse electrode layout to significantly expand the tactile sensing range of robotic arms. It can also meet the diverse needs of robotic arms in various sensing scenarios, thereby significantly enhancing the sensing capabilities of robotic arms.

[0026] The present invention also proposes a robotic arm control system that utilizes the aforementioned robotic arm tactile sensor. This system enables the robotic arm to perceive the surrounding environment more comprehensively and obtain information such as the shape and quality of objects by introducing tactile information, thereby making it more accurate and efficient in performing tasks.

[0027] The robotic arm tactile sensor and control system of this invention can play an important role in fields such as industrial automation. For example, on a production line, the robotic arm can detect and address quality issues of objects through tactile sensing, improving production efficiency and product quality. In the medical field, the robotic arm can utilize tactile information to perform precise and safe surgical procedures. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of a robotic arm tactile sensor according to an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of a control circuit based on a robotic arm tactile sensor, according to a specific embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of a robotic arm control system according to a specific embodiment of the present invention. Detailed Implementation

[0031] This invention proposes a tactile sensor, system, and control method for a robotic arm. To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0032] This invention provides a tactile sensor for a robotic arm, the structure of which is as follows: Figure 1 As shown, the robotic arm tactile sensor has a four-layer structure, consisting of a bottom-layer rigid resin plate, a high-resistivity layer (i.e., a conductive layer), a low-resistivity layer, and a neoprene rubber layer, from bottom to top. The robotic arm tactile sensor of this embodiment has no shape requirements, but excessive size can weaken the imaging effect. In one specific embodiment, three square sensors of different sizes (side lengths of 20cm, 30cm, and 40cm) were used for testing. The testing experiment showed that the imaging effect gradually deteriorated with increasing size. The bottom-layer rigid resin plate is non-conductive. Its surface is frosted to provide good surface conditions for spraying. Multiple uniformly sized circular holes are evenly arranged along the boundaries of the rigid resin plate to place electrodes. In one specific embodiment, four 2mm diameter circular holes are evenly arranged along each boundary of the resin plate to place electrodes, for a total of 16 holes. A conductive graphite spray (GRAPHIT 33 from Germany, in one specific embodiment of the invention) is sprayed onto the surface of a rigid resin plate to form a conductive graphite spray plane. After installing electrodes at pre-reserved positions, silver paste is applied to the contact area between the electrodes and the conductive graphite spray plane to enhance the conductivity at the electrode contact point. The conductive graphite spray plane and the electrodes together form a complete conductive plane, which is the high-resistance layer. In one specific embodiment of the invention, the low-resistance layer is composed of an array of square low-resistance (high-conductivity) fabric sheets with a thickness of 0.06 mm and a length of 7 mm, with the fabric sheets spaced 1 mm apart. The top layer (i.e., the neoprene layer) is neoprene foam, and the highly conductive low-resistance fabric sheets of the low-resistance layer are evenly pasted to the bottom of the top neoprene foam. Neoprene is a non-conductive fabric, and its function is to make the force on the sensor more uniform, playing a force buffering role. Through the above layered assembly, the robotic arm tactile sensor realizes a working mechanism based on the piezoresistive principle.

[0033] Furthermore, the working principle of the robotic arm tactile sensor described in this embodiment is as follows:

[0034] This embodiment describes a robotic arm tactile sensor based on resistive tomography (ERT) technology. ERT is an inference imaging technique that infers the distribution of conductivity (1 / resistance) throughout the sensing medium by injecting current and measuring voltage on a small number of selected electrodes. It comprises two sub-problems: a forward problem and an inverse problem. The forward problem determines the voltage difference between the boundary electrodes of the sensor based on a given medium conductivity and boundary conditions, while the inverse problem reconstructs the conductivity distribution based on the measured voltage.

[0035] Specifically, the working principle of the robotic arm tactile sensor in this embodiment is as follows: Under normal conditions (when the neoprene layer of the sensor is not under pressure), current excitation is applied to two adjacent electrodes. At this time, the current only flows in the high-resistivity layer and does not pass through the low-resistivity fabric sheet. There is contact resistance between the graphite spray of the high-resistivity layer and the fabric sheet of the low-resistivity layer in this sensor structure.

[0036] When a downward pressure is applied to the neoprene layer of the sensor, a portion of the low-resistivity fabric layer comes into contact with the high-resistivity layer, forming an equivalent circuit of a parallel resistor. In this case, current will partially flow through the low-resistivity fabric layer. According to the parallel resistor formula R... 总 =(R high *R low ) / (R high +R low Parallel connection reduces the resistance of the contact area. And R high / R low The larger R is high Much larger than R low , R 总 The smaller and closer to R, the better. low In the embodiments of the present invention, R high / R low Since it is approximately equal to 5, it can be determined that the resistance will change sufficiently when the high-resistance layer comes into contact with the low-resistance layer. Wherein, R... high R represents the resistance value of the high-resistivity layer. low This represents the resistance value of the low-resistivity layer. In summary, it can be concluded that applying pressure increases the conductivity of the contact area between the high-resistivity and low-resistivity layers.

[0037] Since the conductivity distribution on the sensor surface cannot be directly measured, this embodiment, based on the ERT principle, applies adjacent current excitation to the boundary electrodes and measures the voltage values ​​of the electrodes. Then, through inverse problem calculation, the conductivity distribution is derived. In a specific embodiment of the invention, the sensor has 16 electrodes, numbered E1, E2, E3, E4…E16. The selection order of adjacent current excitation electrode pairs is E1E2, E2E3, E3E4, E4E5…E16E1, resulting in 16 adjacent current excitation electrode pairs.

[0038] Furthermore, in order to use the ERT method on the robotic arm's tactile sensor, this embodiment requires adding suitable electronic components outside the robotic arm's tactile sensor to achieve functions such as rapid selection of the current excitation mode and acquisition of boundary electrode voltages. Therefore, this embodiment designs a control circuit based on the robotic arm's tactile sensor, the structure of which is as follows: Figure 2 As shown, in addition to the aforementioned robotic arm tactile sensor, the system also includes: a power module, a multiplexer, and a data acquisition and microcontroller module. The power module is connected to the multiplexer, which in turn is connected to the data acquisition and microcontroller module, which is connected to the robotic arm tactile sensor. In one specific embodiment, the power module uses a 12V adapter with an adjustable resistor to output a constant current. The multiplexer uses two MAX306 chips with 16-channel selection and 1-channel operation as analog switches. The multiplexer connects the selected adjacent electrodes on the robotic arm tactile sensor to inject a constant current. The data acquisition and microcontroller module comprises a data acquisition submodule and a microcontroller unit. In one specific embodiment of the present invention, the data acquisition submodule consists of two 16-bit synchronous sampling AD7606 chips with a sampling rate of up to 200K. The data acquisition submodule is connected to each electrode of the sensor to acquire the voltage signals output by each electrode of the sensor. The microcontroller is a 32-bit STM32F107VCT6 microcontroller based on the ARM Cortex-M3 core. The microcontroller is used to control the multiplexer to inject constant current into the 16 adjacent electrode pairs on the robotic arm tactile sensor according to the adjacent excitation mode.

[0039] Furthermore, in this embodiment, the control circuit also includes a voltage signal processing module. This module can generate a conductivity matrix based on the voltage signal output by the data acquisition and microcontroller modules, and realize the imaging of the signals acquired by the robotic arm's tactile sensor to obtain the corresponding tactile perception image. In a specific embodiment of the present invention, a host computer image reconstruction system was designed using MATLAB software. This system mainly realizes functions such as excitation current switching, voltage data acquisition control, and image reconstruction, thereby obtaining the conductivity distribution image of the high-resistivity layer of the sensor. To achieve this goal, this embodiment uses the EIDORS toolkit based on MATLAB, which is specifically designed for resistivity tomography and diffusion optics reconstruction. It has extensive functions, including support for finite element model construction, excitation-measurement mode selection, inverse conductivity calculation, and image reconstruction. The specific implementation is as follows: First, based on the sensor structure in this embodiment, a matching simulation model was constructed using the EIDORS toolkit, and the finite element method was used to mesh the simulation model. Furthermore, by setting the excitation current mode (adjacent current excitation) and voltage measurement mode (measuring the voltage values ​​of all electrodes), and selecting appropriate current source parameters (24mA, using a multimeter to measure the current value between the positive and negative terminals of the power module), the current is guided to the sensor surface by applying current between adjacent electrodes. In the program, the required data acquisition device library functions are first loaded, and the ports of each data acquisition device are initialized. We set the sampling frequency (20kHz) and the number of consecutive samples per channel (5) for the data acquisition device. Next, a while loop is written to continuously acquire voltage data. Following the adjacent current excitation mode, the voltage values ​​of 16 electrodes are acquired after current injection, for a total of 256 voltage values ​​acquired in each loop. Then, an inverse problem algorithm is used to map the 256 measured voltage values ​​into a 24x24 conductivity matrix. Finally, using the pcolor and mesh functions in the MATLAB library, this matrix is ​​mapped into the corresponding tactile sensing image. After this loop, a complete data acquisition and image reconstruction process is completed. By using the aforementioned electronic components and designing the host computer image reconstruction system, the ERT method can be fully implemented in the robotic arm tactile sensor of this embodiment. This allows the sensor to acquire tactile information such as the shape and mass of objects, and convert the conductivity matrix into a visualized image, providing a more comprehensive sensing capability for robotic arm operation.

[0040] Furthermore, this embodiment of the invention also proposes a robotic arm control system based on the above-described control circuit, the principle of which is as follows: Figure 3As shown, the system includes: a human-computer interaction module, a vision module, a tactile module, and a robotic arm control module. The human-computer interaction module is used to send natural language commands issued by the operator to the robotic arm control module via instant messaging software. In one specific embodiment of the invention, the instant messaging software is WeChat. The human-computer interaction module allows the operator to send commands to the robotic arm through the WeChat platform to achieve human-computer interaction. Through the WeChat interface, the operator can send commands to the robotic arm in the form of natural language text, describing the operations desired by the robotic arm.

[0041] The robotic arm control module is used to extract the object category, weight level, and containment area information from the natural language command, send the object category information to the vision module, and send the weight level information to the tactile module; then, based on the coordinate information of the center position of the object of the same category returned by the vision module and the conductivity matrix of the object of that category returned by the tactile module, the spatial position coordinates of the object are generated; combined with the spatial position coordinates of the containment area in the command, the robotic arm is controlled to complete the grasping task.

[0042] To enable robots to better understand natural language commands issued by operators, a specific embodiment of this invention employs a UIE (Universal Information Extraction) model to perform entity extraction from natural language commands. The UIE model extracts the grasping category, weight level, and containment area information from a given command, and transforms this information into keyword form, thereby extracting the necessary key information from the operator's command. After extraction and transformation, this information can be better understood and processed by the robot, enabling it to perform the corresponding task. Formally, the UIE takes a given structural pattern guide(s) and a text sequence (x) as input to generate a linearized structured extraction language (y):

[0043] y = UIE(s⊕x)

[0044] Define a vector y = {y o ,y w ,y p}, where y o Indicates the category (Object) of the object to be grabbed, y w Indicates the weight level of the object being crawled, y p This indicates the position (i.e., the containment area) where the grabbed object should be placed.

[0045] In one specific embodiment of the invention, given the input text sequence (x) as "Please grab the heaviest large bottle and put it on the right plate", and defining the structure pattern guide (s) as ['Object', 'Weight', 'Position'), information extraction is performed using the UIE model based on the input, resulting in the structured extraction language (y) as {'large bottle', 'heaviest', 'right plate'}. This indicates that the task the robotic arm should perform is to grab the heaviest large bottle and place it on the right plate.

[0046] In this example, the UIE model, based on a predefined structural pattern guide(s) and the input text sequence(x), extracts information to determine the object's category, weight class, and containment area. In the results, 'largebottle' indicates the object is a large bottle, 'heaviest' indicates the heaviest object should be selected, and 'rightplate' indicates the object should be placed on the plate on the right.

[0047] The vision module is used to acquire environmental images of the robotic arm performing its tasks. By performing target detection on each image, and based on the received category information of the grasped object, it outputs the coordinate information of the center position of the object of the same category to the robotic arm control module.

[0048] In one specific embodiment of this invention, the YoloX object detection network framework is used for visual feature extraction. However, YoloX currently does not support bottle category recognition. To address this issue, this embodiment also creates a corresponding dataset to supplement the categories of large bottles and small bottles. To enhance the model's generalization ability, this embodiment uses a camera to collect visual data under two different lighting conditions, capturing images containing both large and small bottles, collecting a total of 500 images. The dataset is then expanded to 2000 images through rotation and mirroring operations. The training set contains 1600 images, and the validation set contains 400 images. To label object categories, the images are manually labeled using the Labelimg tool. The input to the YoloX model can be video or images with a resolution of 320×320. The model can detect all objects of the same category as the grasped object and return their center coordinates.

[0049] The tactile module employs a control circuit based on the robotic arm tactile sensor as described above. By placing objects of different weights and the same type of grasping object on the surface of the robotic arm tactile sensor, the control circuit obtains the conductivity matrix of the corresponding surface of the robotic arm tactile sensor and sends it to the robotic arm control module.

[0050] Furthermore, this embodiment of the invention also proposes a robotic arm control method based on the above system, comprising the following steps:

[0051] 1) The human-machine interaction module will send natural language grasping task instructions to the robotic arm via instant messaging software. The human-machine interaction module will use the UIE model to extract key information of the task instructions and send it to the robotic arm control module. The key information includes the type, weight class and containment area of ​​the grasped object.

[0052] 2) The robotic arm control module sends the extracted category information of the grasped object to the vision module. The vision module acquires environmental images of the robotic arm working through the camera, and then uses the YoloX object detection model to mark the objects belonging to the category in the images acquired by the camera, and returns the center position coordinates of the objects to the robotic arm control module.

[0053] 3) By placing objects of different weights and the same type of grasping object on the surface of the robotic arm tactile sensor in the tactile module, the conductivity of the objects on the surface of the robotic arm tactile sensor is measured based on the control circuit of the robotic arm tactile sensor, the corresponding conductivity matrix is ​​obtained and sent to the robotic arm control module.

[0054] 4) The robotic arm control module obtains the final coordinates of the grasped object based on the coordinate information returned by the vision module, the conductivity matrix information returned by the tactile module, and the weight level information.

[0055] 5) The robotic arm control module controls the robotic arm to move to complete the grasping based on the position coordinates of the grasped object and the position coordinates of the receiving area.

[0056] In one specific embodiment of the present invention, a communication method between two computers is used to realize the complete robotic arm grasping process. One Windows system computer (computer ①) is equipped with the human-computer interaction module, the vision module, the tactile module, and the robotic arm control module. The other Linux system computer (computer ②) is connected to the robotic arm and is used to receive information sent by computer ① and control the robotic arm to execute the grasping tasks issued by computer ①. Data transmission between computer ① and computer ② is performed using socket communication. In this embodiment, the robotic arm control method includes the following steps:

[0057] 1) The operator sends natural language grasping task instructions to the robotic arm via instant messaging software to the human-machine interface module. The human-machine interface module uses the UIE model to extract key information from the task instructions and sends it to the robotic arm control module. For example, the message "Please grab the heaviest large bottle and put it on the right plate" can extract three key pieces of information: the category of the object to be grasped is "large bottle," the weight class is "heaviest," and the storage area is "right plate."

[0058] 2) The robotic arm control module sends the extracted category information of the grasped object to the vision module. The vision module collects environmental images of the robotic arm working through the connected camera, and then uses the YoloX object detection model to mark objects belonging to the "large bottle" category in the images acquired by the camera, and returns the center position coordinates of the objects to the robotic arm control module.

[0059] 3) By placing large and small bottles of different weights on the surface of the robotic arm tactile sensor in the tactile module, the conductivity matrix of the corresponding robotic arm tactile sensor surface is obtained based on the control circuit of the robotic arm tactile sensor and returned to the robotic arm control module.

[0060] 4) The robotic arm control module processes the "heaviest" (weight level information), the center position coordinates of the marked object, and the conductivity matrix information obtained by the tactile module to obtain the final position coordinates of the grasped object.

[0061] 5) The robotic arm control module integrates the position coordinate information of the grasped object obtained in step 4) with the position coordinate information of the "rightplate" of the receiving area, and performs coordinate transformation on these two position coordinates to obtain two spatial position coordinates with the base of the robotic arm as the origin. One is the spatial position coordinate of the grasped object, and the other is the spatial position coordinate of the receiving area.

[0062] 6) A C++ grasping program was designed based on the examples and APIs of the libranka library provided by Franka Emika, meeting the scenario of this embodiment. This program aims to control the Franka Emika Panda robot to complete grasping tasks. When executing the main grasping program, only two spatial coordinates need to be passed in to control the robotic arm and complete the operation commands sent by the operator. Example: `. / grasp 172.16.0.2x1 y1 z1 x2 y2 z2`, where 172.16.0.2 is the IP address of the robotic arm, x1 y1 z1 are the spatial coordinates of the object to be grasped, and x2 y2 z2 are the spatial coordinates of the containment area.

Claims

1. A robotic arm control system, characterized in that, include: Human-computer interaction module, vision module, tactile module, and robotic arm control module; The human-machine interaction module is used to receive natural language commands issued by the operator through instant messaging software, then extract key information from the natural language commands and send it to the robotic arm control module. The key information includes the type, weight class and containment area of ​​the object to be grasped. The robotic arm control module is used to send the category information of the grasped object to the vision module, and generate the spatial position coordinates of the grasped object based on the coordinate information of the center position of the object of the same category returned by the vision module, the conductivity matrix returned by the tactile module, and the weight level information; combined with the spatial position coordinates of the receiving area, the robotic arm is controlled to complete the grasping task. The vision module is used to acquire environmental images of the robotic arm performing tasks. By performing target detection on the images, based on the received category information of the grasped object, it outputs the coordinate information of the center position of the object with the same category as the grasped object and sends it to the robotic arm control module. The tactile module employs a control circuit incorporating a robotic arm tactile sensor. This sensor is placed on the surface of the robotic arm. By placing objects of different weights (of the same type as the grasping object) on the sensor's surface, the control circuit acquires the conductivity matrix of the sensor's surface and sends it to the robotic arm control module. The robotic arm tactile sensor comprises, from bottom to top, a bottom-layer rigid resin plate, a high-resistance layer, a low-resistance layer, and a neoprene rubber layer. The bottom-layer rigid resin plate is non-conductive. Multiple electrodes are uniformly placed at the boundaries of the bottom-layer rigid resin plate. A conductive graphite spray is applied to the surface of the bottom-layer rigid resin plate to form a conductive graphite spray plane. Silver paste is applied to the contact area between the electrodes and the conductive graphite spray plane. The conductive graphite spray plane and the electrodes together form the high-resistance layer. The low-resistance layer is an array of uniformly spaced low-resistance fabric sheets. The neoprene rubber layer is composed of neoprene rubber foam, with the fabric sheets of the low-resistance layer adhered to the bottom of the neoprene rubber foam. The robotic arm tactile sensor operates as follows: When the neoprene layer is not under pressure, if a current excitation is applied to two adjacent electrodes, the current flows through the high-resistivity layer and does not pass through the low-resistivity fabric sheet; when the neoprene layer is subjected to downward pressure, part of the low-resistivity fabric sheet comes into contact with the high-resistivity layer, forming an equivalent circuit of a parallel resistor, thereby increasing the conductivity of the contact area between the high-resistivity layer and the low-resistivity layer; by sequentially applying current excitation to the electrode pairs composed of adjacent electrodes in the robotic arm tactile sensor and measuring the voltage value of each electrode, the conductivity matrix of the object contacted by the robotic arm tactile sensor is obtained based on the principle of resistivity tomography.

2. The robotic arm control system according to claim 1, characterized in that, The instant messaging software used is WeChat.

3. The robotic arm control system according to claim 1, characterized in that, The control circuit further includes: a power supply module, a multiplexer, and a data acquisition and microcontroller module; the power supply module is connected to the multiplexer, the multiplexer is connected to the data acquisition and microcontroller module, and the data acquisition and microcontroller module is connected to the robotic arm tactile sensor; the power supply module is used to output a constant current; the multiplexer is used to connect the selected adjacent electrode pairs on the robotic arm tactile sensor to inject a constant current; the data acquisition and microcontroller module includes a data acquisition submodule and a microcontroller unit; the data acquisition submodule is connected to each electrode of the robotic arm tactile sensor respectively, and is used to acquire the voltage signals output by each electrode of the robotic arm tactile sensor; the microcontroller unit is used to control the multiplexer to inject a constant current into the electrode pairs formed by adjacent electrodes on the robotic arm tactile sensor according to the adjacent excitation mode.

4. The robotic arm control system according to claim 2, characterized in that, The key information extracted from the natural language instructions is obtained using the UIE general information extraction model.

5. The robotic arm control system according to claim 3, characterized in that, The control circuit also includes a voltage signal processing module, which uses MATLAB to convert the voltage signal output by the data acquisition submodule into a conductivity matrix.

6. The robotic arm control system according to claim 4, characterized in that, The object detection in the image is performed using the YoloX object detection model.

7. The robotic arm control system according to claim 5, characterized in that, The voltage signal processing module is also used to map the conductivity matrix into a corresponding tactile perception image.

8. A robotic arm control method based on the robotic arm control system as described in claim 6, characterized in that, include: 1) The operator sends a natural language grasping task instruction to the robotic arm via instant messaging software to the human-machine interaction module. The human-machine interaction module uses the UIE model to extract the key information of the task instruction and sends it to the robotic arm control module. The key information includes the type, weight class and containment area of ​​the grasped object. 2) The robotic arm control module sends the category information of the grasped object to the vision module. The vision module collects environmental images of the robotic arm working through the connected camera, and then uses the YoloX object detection model to mark the objects belonging to the category in the image and returns the center position coordinates to the robotic arm control module. 3) By placing objects of different weights and the same type of grasping objects on the surface of the robotic arm tactile sensor in the tactile module, the conductivity of the objects on the surface of the robotic arm tactile sensor is measured based on the control circuit of the robotic arm tactile sensor, and the corresponding conductivity matrix is ​​returned to the robotic arm control module. 4) The robotic arm control module obtains the final coordinates of the grasped object based on the coordinate information returned by the vision module, the conductivity matrix information returned by the tactile module, and the weight level information. 5) The robotic arm control module controls the robotic arm to move to complete the grasping based on the position coordinates of the grasped object and the position coordinates of the receiving area.

Images