A robot perception and grasping method and device based on visual-tactile multi-modal fusion

Abstract

This invention discloses a robot perception and grasping method and apparatus based on visual-tactile multimodal fusion, relating to the field of intelligent robot grasping technology. It mainly includes locating the target to be grasped, with the robot driving a robotic arm to grasp the target based on the target location result; during grasping, tactile signals are collected and used to fuse visual-tactile information to obtain a target classification result, and the grasping strategy is optimized using the classification result. Specifically, this invention provides a target localization method and a scheme for optimizing the grasping strategy using visual-tactile information; compared with existing technologies, this invention can effectively improve the robot's target recognition ability and contact state perception ability, thereby achieving accurate and stable grasping.

Description

Technical Field

[0001] This invention relates to the field of intelligent grasping technology for robots, and more specifically to a robot perception method and apparatus that integrates visual and tactile multimodal sensing. Background Technology

[0002] Intelligent grasping by robots is a core technology in the fields of industrial automation and service robots, and its performance directly determines the flexibility and reliability of operations. Traditional grasping systems mostly rely on single perception, which has obvious bottlenecks. For example, pure vision grasping is easily affected by lighting and occlusion, and it is difficult to perceive tactile information such as the softness and hardness of objects, making it prone to unstable grasping; pure tactile grasping is inefficient and cannot achieve non-contact recognition and positioning.

[0003] Meanwhile, existing flexible pressure sensors are complex and costly to manufacture, making them difficult to adapt to mechanical grippers; object classification and positioning methods are independent, making it difficult to achieve visual-touch depth fusion and resulting in insufficient accuracy; traditional point cloud processing has weak noise resistance and is difficult to quickly extract target coordinates in cluttered scenes.

[0004] Therefore, developing an intelligent grasping system that is accurate in positioning, stable in grasping, and cost-controllable is an urgent need to solve current pain points and promote technological upgrades. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a robot perception and grasping method and apparatus based on visual-tactile multimodal fusion, so as to overcome the above problems or at least partially solve the above problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a robot perception and grasping method based on visual-tactile multimodal fusion, the steps of which include locating the target to be grasped, the robot driving a robotic arm to grasp the target based on the target location result; during grasping, tactile signals are collected to fuse visual-tactile information to obtain a target classification result, and the grasping strategy is optimized using the classification result; The steps for locating the target to be captured include: Reference point cloud and test point cloud for capturing the scene; Based on the reference point cloud, the changing points in the test point cloud are identified to obtain the initial difference point cloud; The initial difference point cloud was divided into several clusters using the DBSCAN clustering algorithm; Each cluster of point clouds is filtered to obtain the point cloud of the target object; A directed bounding box is constructed based on principal component analysis of the target object's point cloud. The center coordinates and rotational orientation of the directed bounding box are used as the pose in the camera coordinate system, and the pose in the camera coordinate system is transformed to the robot arm coordinate system.

[0008] Preferably, the reference point cloud is obtained through the following steps: For targetless grasping scenarios, point cloud data is continuously collected N times and directly merged to obtain the background point cloud; The background point cloud is divided into voxel grids, and the scene point cloud is obtained based on the centroid coordinates of each voxel grid. Based on the working range of the robotic arm, a retention range in the depth direction of the point cloud is determined, and the scene point cloud is filtered using the retention range to obtain a reference point cloud.

[0009] The test point cloud is collected once for a target-grabbing scenario, and the retained interval is used directly for filtering.

[0010] Preferably, identifying change points in the test point cloud includes the following steps: For each point in the test point cloud, calculate the distance between it and the nearest neighbor point in the reference point cloud. If the distance is greater than a distance threshold, it is determined to be a changed point; otherwise, it is a background point. The distance threshold is:

[0011] In the formula, For testing the midpoint of the point cloud Distance threshold, As the baseline threshold, For adjustment coefficients, For testing the midpoint of the point cloud The density value of the voxel. Use small positive numbers to prevent division by zero. To test the maximum voxel density value in the point cloud, This is the minimum voxel density value in the test point cloud.

[0012] Preferably, in the DBSCAN clustering algorithm, the neighborhood radius parameter of DBSCAN is adaptively adjusted based on the density field information, and the expression is:

[0013] In the formula, Let p be the radius of the neighborhood of point p. As the reference radius, This is the normalized result of the density value of the voxel at point p.

[0014] Preferably, each cluster of point clouds is filtered to obtain the point cloud of the target object, including: Based on the neighborhood density filtering of each cluster of point clouds, core points with more than a preset threshold of neighborhood points in each cluster and non-core points located in the neighborhood of core points are retained, while low-density points isolated from the neighborhood of all core points in each cluster are removed as noise.

[0015] Preferably, the robot gripper is equipped with a tactile sensor for collecting tactile signals; The tactile sensor comprises an electrode layer, a fiber membrane layer, and a PDMS elastic membrane in sequence. One side of the PDMS elastic membrane with a microsphere array faces the fiber membrane layer, and the other side directly contacts the target to be clamped.

[0016] Preferably, when collecting tactile signals, the time point at which the target to be grasped comes into contact with the tactile sensor is used as a reference, and multiple sampling points before and after the current time point are extracted to form tactile signals.

[0017] Preferably, the target classification result is obtained by fusing visual and tactile information, including: The visual image of the target to be captured is obtained and input into the first residual network to extract visual features; The tactile signal is converted into a two-dimensional image through the Gram angle sum field and the Gram angle difference field, and then input into the second residual network to extract tactile features; The visual and tactile features are projected onto a low-rank matrix and a Hadamard product is performed. Then, the fused features are obtained by SumPooling aggregation through a one-dimensional non-overlapping window. The fused features are input into the classifier to obtain the target classification result.

[0018] Preferably, optimizing the crawling strategy using the classification results includes: The robotic arm grasps the target to be grasped based on an initial grasping force; Based on the classification results and the pre-built database, the crawling intensity is adjusted. The pre-built database stores the mapping relationship between the classification results and the adjusted crawling intensity. The adjusted gripping force is superimposed on the initial gripping force to achieve the gripping of the target.

[0019] Secondly, embodiments of the present invention provide a robot perception and grasping system based on visual-tactile multimodal fusion, used to implement the robot perception and grasping method based on visual-tactile multimodal fusion as described in any of the preceding claims, wherein: The robot has a robotic arm, which is connected to a two-finger mechanical gripper. The robotic arm is equipped with a vision sensor, and the two-finger mechanical gripper is equipped with a tactile sensor. The robot is connected to a host computer; the host computer is used to locate the target to be grasped based on the information collected by the visual sensor and drive the robotic arm to grasp it; at the same time, based on the tactile signals collected by the tactile sensor, the visual and tactile information is fused to obtain the target classification result, and the grasping strategy is optimized using the classification result.

[0020] This invention provides a robot perception and grasping method and apparatus based on visual-touch multimodal fusion, which, compared with the prior art, offers at least the following advantages: 1. This application can effectively break through the technical bottleneck of traditional single perception system, and innovatively introduce visual and tactile information fusion to achieve deep collaboration between visual and tactile perception, making up for the defects of pure visual perception being easily affected by light and occlusion and pure tactile perception being inefficient. 2. In view of the different information content of visual and tactile images, this application designs residual network models of different depths. Visual three-channel images are input into a standard ResNet-34 network to extract visual features, and tactile images with less information are input into a lightweight residual network to extract tactile features, thereby effectively improving the accuracy of target object classification. 3. By combining point cloud differential and density filtering positioning technologies, the environmental perception capability and the anti-noise capability of target object positioning can be enhanced, thereby quickly extracting the target object position information. With the coordinated control of the robotic arm and gripper, the grasping stability can be effectively improved and the risk of object damage can be reduced. 4. The sensor fabrication process of this application has significant advantages in terms of low cost and simplicity. Specifically, it uses clay as the negative template substrate and combines PDMS mixture casting, vacuum drying and curing, and deionized water demolding. It does not require complex and precise processing equipment, and the process is simple and easy to operate, which can greatly reduce the production cost of the sensor. At the same time, the sensor with the microstructure layer exhibits high sensitivity to pressure, such as excellent repeatability, high selectivity, and long-term stability. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the robot perception and grasping method based on visual-touch multimodal fusion provided in this embodiment of the invention. Figure 2 This is a flowchart of the target localization algorithm provided in the embodiments of the present invention; Figure 3The images shown are physical photos and structural diagrams of the flexible pressure sensor provided in the embodiments of the present invention; wherein, (a) and (b) are physical photos, and (c) is a structural diagram of the sensor; Figure 4 The following are performance test results of the flexible pressure sensor provided in the embodiments of the present invention, wherein (a) is the sensor sensitivity test diagram, (b) is the sensor step response test diagram, (c) is the sensor IV curve, and (d) is the sensor response at different compression frequencies. Figure 5 This is a schematic diagram of the visual feature extraction network structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the tactile feature extraction network structure provided in an embodiment of the present invention; Figure 7 This is a structural diagram of the robot perception and grasping system with visual-touch multimodal fusion provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of system integration control provided in an embodiment of the present invention; Figure 9 This is an example diagram of a system software interface provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention discloses a robot perception and grasping method based on visual-tactile multimodal fusion. The method mainly includes locating the target to be grasped, then driving the robotic arm to grasp the target based on the target location result. During grasping, tactile signals are collected based on a preset initial gripping force, which are used to fuse visual and tactile information to obtain a target classification result, and the grasping strategy is optimized using the classification result. The specific process is as follows: Figure 1 As shown, the above process will be explained below through specific embodiments.

[0025] Example 1: In one embodiment, locating the target to be grasped includes the following steps: Reference point cloud and test point cloud for capturing the scene; Based on the reference point cloud, the changing points in the test point cloud are identified to obtain the initial difference point cloud; The initial difference point cloud was divided into several clusters using the DBSCAN clustering algorithm; Each cluster of point clouds is filtered to obtain the point cloud of the target object; A directed bounding box is constructed based on principal component analysis of the target object's point cloud. The center coordinates and rotational orientation of the directed bounding box are used as the pose in the camera coordinate system, and the pose in the camera coordinate system is transformed to the robot arm coordinate system.

[0026] Figure 2 This is a schematic diagram of a target location capture process; In one alternative implementation, a vision sensor is mounted to the robotic arm in an eye-on-hand mode.

[0027] When acquiring reference point clouds, for scenarios without target capture, point cloud data is acquired N times consecutively, and a multi-frame fusion strategy is used to generate scene point clouds as references, where N≥3; the multi-frame fusion strategy is as follows: All points in the Nth point cloud data are directly merged to obtain the original background point cloud set. ; With the voxel side length set to v = 0.003, the 3D space is divided into a regular voxel mesh. For each non-empty voxel, the centroid coordinates of all original points are calculated. The centroids of all voxels constitute the fused scene point cloud. In this embodiment, the formula for calculating the centroid coordinates of each voxel is as follows:

[0028]

[0029]

[0030] Furthermore, the retention interval in the depth direction of the point cloud is determined based on the working range of the robotic arm. , In a preferred embodiment, considering the actual installation distance between the vision sensor and the robotic arm, let... , Then, the scene point cloud within the reserved interval is filtered to remove points outside the set depth range, thus obtaining a reference point cloud. .

[0031] When collecting test point clouds, point clouds are collected only once for scenes with targeted capture, and the retained interval is directly used for filtering to obtain the final test point cloud. .

[0032] In one alternative implementation, the steps of identifying change points in the test point cloud based on the density field adaptive three-dimensional point cloud difference method include: For test point cloud Each point in Using KD-Tree in Search for its nearest neighbor. And calculate the Euclidean distance between them. When the distance is greater than the distance threshold, i.e. At that time, the judgment Points that are considered changes are considered as background points; all points that are considered changes constitute the initial difference point cloud. .

[0033] Among them, the distance threshold in the density field adaptive 3D point cloud difference method Based on the dynamically set density field information, in this embodiment, the density of a certain voxel is calculated by counting the number of original points within that voxel. Therefore, the density field D of the fused point cloud can be expressed as:

[0034] Where i represents the voxel number in the X-axis direction; j represents the voxel number in the Y-axis direction; and k represents the voxel number in the Z-axis direction. Represents any voxel in a voxel grid. This represents the number of primitive points within a voxel.

[0035] The distance threshold for:

[0036] In the formula, For testing the midpoint of the point cloud Distance threshold, As the baseline threshold, based on the camera noise standard deviation σ set up, This is an adjustment coefficient used to control the maximum influence of density on the threshold. For testing the midpoint of the point cloud The density value of the voxel. Use small positive numbers to prevent division by zero. To test the maximum voxel density value in the point cloud, This is the minimum voxel density value in the test point cloud.

[0037] In one alternative implementation, the initial difference point cloud is based on density pairs. Noise filtering is performed by using the DBSCAN clustering algorithm to cluster the point set into several clusters; the neighborhood radius parameter of DBSCAN is adaptively adjusted based on the density field information. The expression is:

[0038] In the formula, Let p be the radius of the neighborhood of point p. As the reference radius, This is the normalized result of the density value of the voxel at point p.

[0039] In one optional implementation, each cluster of point clouds is further filtered to obtain the point cloud of the target object. In this embodiment, the number of points within the neighborhood radius is first counted, and if the number of points within the neighborhood is greater than a preset threshold... Points that are considered core points are retained; points whose number of neighbors is less than a preset threshold are excluded. Points outside the core neighborhood are considered noise points and removed. In some implementations, a threshold is set based on a comprehensive consideration of preserving complete edge information and the point cloud density of the point cloud extraction device. Set to 20; the number of points in the neighborhood is less than or equal to the preset threshold. However, points within the neighborhood of other core points are preserved, resulting in the point cloud of the target object. .

[0040] Finally, a directed bounding box is constructed based on principal component analysis of the target object's point cloud. The center coordinates and rotational orientation of the directed bounding box are used as the pose in the camera coordinate system, and the pose in the camera coordinate system is transformed to the robot arm coordinate system.

[0041] In one alternative implementation, the target object point cloud is... Principal component analysis (PCA) is applied to calculate the three principal component directions of the point cloud, which serve as the three principal axes of the directed bounding box. The point cloud is projected onto these three principal axes, and the minimum and maximum values ​​of the projections are calculated to determine the boundaries of the bounding box along each principal axis. Based on these boundaries and principal axis directions, the directed bounding box is generated, and its center coordinates and rotational attitude are output as the pose of the target object in the camera coordinate system. Then, the coordinates in the camera coordinate system are transformed to the robotic arm coordinate system using a hand-eye calibration matrix. The specific transformation process is as follows:

[0042] in Represents the coordinates in the robot arm's coordinate system. 3 obtained for hand-eye calibration A rotation matrix of 3, The coordinates are in the camera coordinate system. The translation vector obtained from hand-eye calibration.

[0043] Example 2: In one embodiment, this application utilizes a tactile sensor to collect tactile signals, wherein the tactile sensor is mounted on the inside of a two-finger mechanical gripper, directly contacting the target to be grasped. In this embodiment, the tactile sensor sequentially includes an electrode layer, a fiber membrane layer, and a PDMS elastic membrane. One side of the PDMS elastic membrane with a microsphere array faces the fiber membrane layer, while the other side directly contacts the target to be grasped.

[0044] In some implementations, the tactile sensor is manufactured as follows: A positive mold with a hemispherical boss array of individual spheres and a diameter of 600 μm was pressed into plastic clay. After demolding, the clay matrix was placed in a vacuum drying oven and dried and cured at 60°C to obtain a negative template with corresponding hemispherical array depressions on the surface. The prepared polydimethylsiloxane (PDMS) mixture was slowly poured onto the surface of a cured clay negative template. After pouring, the entire assembly was left to cure in a vacuum drying oven at 60°C for 4 hours. Then, the assembly was immersed in deionized water, and the clay template was dissolved by the deionized water to achieve demolding, resulting in a PDMS elastic film with a regular microsphere array protrusion structure on the surface. Compared with traditional methods, this method is lower in cost and simpler and more convenient. Further, using dimethylformamide (DMF) as a solvent, 10 g of a polyacrylonitrile (PAN) spinning solution with a mass fraction of 8 wt% was prepared. The spinning solution was transferred to a disposable syringe, and a No. 20 metal needle was added and fixed in an electrospinning machine. Nonwoven fabric was attached to the receiving plate of the electrospinning machine, and after electrospinning for 8-10 hours, an electrospun polyacrylonitrile (PAN) fiber film was obtained. Finally, the electrospun polyacrylonitrile (PAN) fiber film was placed in a vacuum drying oven and dried at 60°C for 12 hours to remove residual solvent. Then, 750 mg each of carbon black and carbon nanotubes, 10 g of deionized water, and 10 g of ethylene glycol were weighed, stirred under magnetic stirring for 3 min, and then 100 mg of sodium carboxymethyl cellulose was added to obtain a conductive solution. The polyacrylonitrile (PAN) fiber film was immersed in the solution for 4 h and ultrasonically treated at room temperature for 30 min. It was then dried at 60°C for 2 h to obtain a PAN / MWCNTs / CB / CMC (PMCC) fiber membrane.

[0045] Finally, a high-sensitivity pressure sensor with a sandwich structure was fabricated in the following order from top to bottom: PDMS elastic membrane, fiber membrane layer, and electrode layer. The sensor was then encapsulated using a smooth, microstructure-free PDMS film. The resulting flexible pressure sensor is shown below. Figure 3 As shown in the figure, the sensor performance test results are as follows: Figure 4 As shown. Figure 4As shown in (a), the sensor exhibits high sensitivity in the low-pressure region of 0-5 kPa, enabling it to more sensitively reflect the physical characteristics of the object being applied during the tactile information acquisition process in the visual-tactile fusion stage. Furthermore, the sensor possesses a fast response / recovery speed, such as... Figure 4 (b) and the electrical signal output is stable, such as Figure 4 (c) Simultaneously, it can maintain a stable output signal under different gripper closing speeds, such as Figure 4 The middle (d) can provide stable tactile signal feedback during the robotic arm's grasping process.

[0046] Example 3: In one embodiment, visual and tactile information is fused to obtain a classification result, and the classification result is used to optimize the crawling strategy. In this embodiment, the steps include: S1. First, acquire the visual image of the target to be captured and input it into the first residual network to extract visual features; in some implementations, the first residual network adopts a ResNet-34 network, with the structure as follows: Figure 5 As shown; S2. Further, tactile signals are acquired and converted into a two-dimensional image through Gram angle sum and field and Gram angle difference field. This image is then input into a second residual network to extract tactile features. The second residual network is a signal-image-based simplified residual network based on ResNet-34 but with reduced network depth. It has fewer network layers than the standard ResNet-34 to accommodate the relatively small amount of information in tactile images. The specific structure is as follows: Figure 6 As shown, it specifically includes a signal imaging module and a simplified ResNet feature extraction module connected in sequence.

[0047] In some implementations, when collecting tactile signals, the time point at which the target to be grasped comes into contact with the tactile sensor is used as a reference, and multiple sampling points before and after the current time point are extracted to form a tactile signal. For example, the contact time point t between the object and the gripper is identified; based on the contact time point t, the tactile signal values ​​of 100 sampling points before time t and 200 sampling points after time t are extracted to form a tactile data segment with a length of 300 sampling points.

[0048] The signal imaging module is used to convert multi-channel one-dimensional tactile timing signals into two-dimensional images through Gram angle sum field and Gram angle difference field, as follows:

[0049]

[0050]

[0051]

[0052] In the formula, Represents polar coordinate angles. , 0.25, Image is the converted two-dimensional image.

[0053] The simplified ResNet feature extraction module retains the initial convolutional and max-pooling layers of ResNet-34; the number of original residual blocks in the first and second stages are 3 and 4 respectively, and the first 2 residual blocks are retained in each stage; the number of original residual blocks in the third stage is 6, and the first 3 residual blocks are retained; the number of original residual blocks in the fourth stage is 3, and the first residual block is retained; fully connected layers are removed. Meanwhile, after simplifying the third stage of ResNet, a lightweight tactile texture enhancement attention module (TTEA) is embedded. This module only enhances high-frequency texture regions in the feature map to adapt to the characteristics of tactile images with low information content and local texture salience.

[0054] S3. Project the tactile signal, visual features, and tactile features onto a low-rank matrix respectively, perform a Hadamard product, and then aggregate the fused features through a one-dimensional non-overlapping window using SumPooling. This embodiment introduces the bilinear pooling method from the field of visual question answering (VQA) to achieve efficient fusion of the visual feature vector and the tactile feature vector. This method decomposes the bilinear weights into the product of two low-rank matrices through matrix factorization, reducing the number of parameters while achieving feature-level adaptive fusion of visual and tactile features, outputting a fused feature vector. The specific transformation process is as follows: In traditional bilinear pooling, for a given feature vectors of visual and tactile modalities... and The output of traditional bilinear pooling is defined as:

[0055] in, It is a projection matrix. The core idea of ​​the bilinear model's output MFB is to project the matrix... It can be decomposed into two low-rank matrices, as shown in the formula:

[0056] At this point, the output of bilinear pooling is converted to:

[0057] in, The number of factors or latent dimensions in a low-rank decomposition. , , The Hadamard product represents the element-wise multiplication of two vectors. It is a vector consisting entirely of 1s.

[0058] To obtain the overall output Two third-order tensors can be reshaped using a simple reshaping operation. and Remodeled into a two-dimensional matrix and Therefore, the above formula can be rewritten as follows:

[0059] The function SumPooling(x,k) represents summing and pooling x using a one-dimensional non-overlapping window of size k.

[0060] S4. Input the fused features into the classifier to obtain the target classification result. Table 1 shows a comparison of the classification results between the view-touch fusion network and the single-modal network: Table 1

[0061] As shown in Table 1, the visual-touch fusion method provided in this application is superior to the single-modal classification results in terms of accuracy.

[0062] In this embodiment, the crawling strategy is optimized using the classification results, including: The robotic arm grasps the target based on a preset initial grasping force; the preset initial grasping force is not required to be able to clamp the object, but it must ensure that it makes contact with the target without causing damage to it; Based on the classification results and the pre-built database, the optimal crawling intensity is determined. The pre-built database stores the mapping relationship between the classification results and the optimal crawling intensity. The target to be grasped is grasped based on the optimal grasping force.

[0063] In one exemplary embodiment, the optimal gripping force for different fruits is obtained through pre-experimentation before grasping. After obtaining the object's coordinates, the object is moved to the target position to prepare for grasping. First, the object is initially grasped with an initial gripping force to collect tactile information as a basis for visual-tactile fusion. Then, based on the visual-tactile fusion classification results, the optimal gripping force is further switched to complete the gripping strategy optimization.

[0064] Based on the same inventive concept, embodiments of the present invention provide a robot perception and grasping system that integrates visual and tactile multimodal sensing, with the structure as follows: Figure 7 As shown, in this system: The robot has a robotic arm, which is connected to a two-finger mechanical gripper. The robotic arm is equipped with a vision sensor, and the two-finger mechanical gripper is equipped with a tactile sensor. The robot is connected to a host computer; the host computer is used to locate the target to be grasped based on the information collected by the visual sensor and drive the robotic arm to grasp it; at the same time, based on the tactile signals collected by the tactile sensor, the visual and tactile information is fused to obtain the target classification result, and the grasping strategy is optimized using the classification result.

[0065] The vision-touch multimodal fusion robot perception and grasping system provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0066] In some implementation schemes, the integrated control methods between various hardware components in the system are as follows: Figure 8 As shown in the figure. This embodiment uses self-designed and developed software to achieve separate acquisition, preprocessing, and visualization of visual and tactile data; efficient fusion and classification of visual and tactile data; target object positioning; and issuance of robotic arm movement and grasping commands. In some implementation schemes, the software interface is as follows: Figure 9 As shown in Table 2, the fruit sorting results are as follows: Table 2

[0067] Therefore, this application can effectively improve the accuracy and stability of target capture.

[0068] Any content or technical means not mentioned in the embodiments of this invention can be obtained by referring to the prior art. This disclosure does not limit the scope of the invention and therefore will not be elaborated further.

[0069] The embodiments of the present invention have been described in detail above, and the principles and implementation methods of the present invention have been explained. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, computer software program products, or electronic devices, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0072] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot perception and grasping method based on visual-tactile multi-modal fusion, characterized in that, The robot locates the target to be grasped, and drives the robotic arm to grasp it based on the target location result. During grasping, tactile signals are collected to fuse visual and tactile information to obtain target classification results, and the grasping strategy is optimized using the classification results. The steps for locating the target to be captured include: Reference point cloud and test point cloud for capturing the scene; Based on the reference point cloud, the changing points in the test point cloud are identified to obtain the initial difference point cloud; The initial difference point cloud was divided into several clusters using the DBSCAN clustering algorithm; Each cluster of point clouds is filtered to obtain the point cloud of the target object; A directed bounding box is constructed based on principal component analysis of the target object's point cloud. The center coordinates and rotational orientation of the directed bounding box are used as the pose in the camera coordinate system, and the pose in the camera coordinate system is transformed to the robot arm coordinate system.

2. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, The reference point cloud is obtained through the following steps: For targetless grasping scenarios, point cloud data is continuously collected N times and directly merged to obtain the background point cloud; The background point cloud is divided into voxel grids, and the scene point cloud is obtained based on the centroid coordinates of each voxel grid. Based on the working range of the robotic arm, a retention range in the depth direction of the point cloud is determined, and the scene point cloud is filtered using the retention range to obtain a reference point cloud.

3. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, Identifying change points in the test point cloud includes the following steps: For each point in the test point cloud, calculate the distance between it and the nearest neighbor point in the reference point cloud. If the distance is greater than a distance threshold, it is determined to be a changed point; otherwise, it is a background point. The distance threshold is: In the formula, For testing the midpoint of the point cloud Distance threshold, As the baseline threshold, For adjustment coefficients, For testing the midpoint of the point cloud The density value of the voxel. Use small positive numbers to prevent division by zero. To test the maximum voxel density value in the point cloud, This is the minimum voxel density value in the test point cloud.

4. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, In the DBSCAN clustering algorithm, the neighborhood radius parameter of DBSCAN is adaptively adjusted based on the density field information, and the expression is: In the formula, Let p be the radius of the neighborhood of point p. As the reference radius, This is the normalized result of the density value of the voxel at point p.

5. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, Each cluster of point clouds is filtered to obtain the point cloud of the target object, including: Based on the neighborhood density filtering of each cluster of point clouds, core points with more than a preset threshold of neighborhood points in each cluster and non-core points located in the neighborhood of core points are retained, while low-density points isolated from the neighborhood of all core points in each cluster are removed as noise.

6. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, The robot's grippers are equipped with tactile sensors for collecting tactile signals; The tactile sensor comprises an electrode layer, a fiber membrane layer, and a PDMS elastic membrane in sequence. One side of the PDMS elastic membrane with a microsphere array faces the fiber membrane layer, and the other side directly contacts the target to be clamped.

7. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1 or 6, characterized in that, When collecting tactile signals, the time point when the target to be grasped comes into contact with the tactile sensor is used as the reference, and multiple sampling points before and after the current time point are extracted to form tactile signals.

8. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, The target classification results are obtained by fusing visual and tactile information, including: The visual image of the target to be captured is obtained and input into the first residual network to extract visual features; The tactile signal is converted into a two-dimensional image through the Gram angle sum field and the Gram angle difference field, and then input into the second residual network to extract tactile features; The visual and tactile features are projected onto a low-rank matrix and a Hadamard product is performed. Then, the fused features are obtained by SumPooling aggregation through a one-dimensional non-overlapping window. The fused features are input into the classifier to obtain the target classification result.

9. The robot perception and grasping method based on visual-tactile multimodal fusion as described in claim 1, characterized in that, Optimizing the crawling strategy using the classification results includes: The robotic arm grasps the target based on an initial grasping force; Based on the classification results and the pre-built database, the crawling intensity is adjusted. The pre-built database stores the mapping relationship between the classification results and the adjusted crawling intensity. The adjusted gripping force is superimposed on the initial gripping force to achieve the gripping of the target.

10. A robot perception and grasping system that integrates visual and tactile multimodal sensing, characterized in that, A robot perception and grasping method for implementing the visual-touch multimodal fusion as described in any one of claims 1-9 includes: The robot has a robotic arm, which is connected to a two-finger mechanical gripper. The robotic arm is equipped with a vision sensor, and the two-finger mechanical gripper is equipped with a tactile sensor. The robot is connected to a host computer; the host computer is used to locate the target to be grasped based on the information collected by the visual sensor and drive the robotic arm to grasp it; at the same time, based on the tactile signals collected by the tactile sensor, the visual and tactile information is fused to obtain the target classification result, and the grasping strategy is optimized using the classification result.

Images