Robotic grasping hallucination verification method and system based on haptic-visual manifold

Abstract

The application discloses a robot grasping illusion verification method and system based on a tactile-visual manifold. The method is based on a visual grasping success manifold prior parameter set and a tactile stable contact manifold and an alignment prior parameter set. The online visual perception original data and the online tactile perception time series data are respectively subjected to manifold projection processing to obtain visual manifold points, visual state confidence, tactile manifold points and tactile state confidence. The tactile manifold points are mapped to the visual manifold space to obtain tactile alignment points. Based on the visual manifold points and the tactile alignment points, a geometric alignment distance is calculated. Based on the visual state confidence and the tactile state confidence, a semantic consistency coefficient is calculated. The geometric alignment distance and the semantic consistency coefficient are fused to generate a bimodal manifold alignment score. According to the bimodal manifold alignment score and a preset alignment threshold, a robot grasping illusion verification result is generated, which can accurately identify false positive grasping illusions at the moment when the gripper is closed, thereby improving the verification accuracy.

Description

Technical Field

[0001] This application relates to the fields of robot grasping control and multimodal perception technology, and in particular to a robot grasping illusion verification method and system based on tactile-visual manifolds. Background Technology

[0002] Body-bound robots are widely used in industrial precision sorting, electronic component assembly, flexible material handling, and surgical instrument delivery. Grasping is the fundamental action for these robots to complete various tasks. In these scenarios, robots typically rely on visual perception to identify targets, estimate poses, and plan the grasping process, determining whether the grasping was successful after the grippers close. Because the objects being grasped are often small, reflective, have complex textures, or are occluded, the accuracy of the grasping state directly affects the reliability of subsequent handling, assembly, or delivery processes.

[0003] In existing technologies, successful grasping is mainly determined by visual information, while tactile information is mostly used for grasping force closure, pose fine-tuning, or slip detection. Force thresholds and trajectory abrupt changes are usually used for judgment after failure occurs. The confirmation of grasping results still relies mainly on visual or simple physical thresholds, making it difficult to identify false positive grasping illusions at the moment the gripper closes in place, resulting in low accuracy of grasping illusion verification. Summary of the Invention

[0004] The main purpose of this application is to propose a method and system for verifying robot grasping illusions based on tactile-visual manifolds, which can accurately identify false positive grasping illusions at the moment the gripper closes, thereby improving the accuracy of verification.

[0005] The first aspect of this application provides a method for verifying illusions in robot grasping based on a tactile-visual manifold, the method comprising: Acquire raw online visual perception data at the moment the robot gripper closes into position, as well as online tactile perception time-series data within a set time window after the gripper closes; Based on the pre-constructed prior parameter set of the successful visual capture manifold, the original online visual perception data is processed by manifold projection to obtain visual manifold points and visual state confidence. Based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, the online tactile perception time series data is processed by manifold projection to obtain tactile manifold points and tactile state confidence. The tactile manifold points are mapped to the visual manifold space to obtain tactile alignment points; Calculate the geometric alignment distance based on the visual manifold point and the tactile alignment point; Based on the confidence scores of the visual and tactile states, a semantic consistency coefficient is calculated. By fusing the geometric alignment distance and the semantic consistency coefficient, a bimodal manifold alignment score is generated. Based on the bimodal manifold alignment score and the preset alignment threshold, a robot grasping illusion verification result is generated.

[0006] The first aspect of this application provides a robot grasping illusion verification method based on tactile-visual manifolds. This method performs manifold projection processing on the original online visual perception data and the online tactile perception time-series data, respectively, based on a prior parameter set of successful visual grasping manifolds and a prior parameter set of stable tactile contact manifolds and alignment, to obtain visual manifold points, visual state confidence scores, tactile manifold points, and tactile state confidence scores. The tactile manifold points are mapped to the visual manifold space to obtain tactile alignment points. The geometric alignment distance is calculated based on the visual manifold points and the tactile alignment points. A semantic consistency coefficient is calculated based on the visual state confidence scores and the tactile state confidence scores. The geometric alignment distance and the semantic consistency coefficient are fused to generate a bimodal manifold alignment score. Based on the bimodal manifold alignment score and a preset alignment threshold, a robot grasping illusion verification result is generated. This method can accurately identify false positive grasping illusions at the moment the gripper closes, improving verification accuracy.

[0007] To achieve the above objectives, a second aspect of this application provides a robot grasping illusion verification system based on a tactile-visual manifold, the system comprising: The acquisition module is used to acquire the raw online visual perception data at the moment when the robot gripper closes into position, as well as the online tactile perception time sequence data within a set time window after the gripper closes. The first module is used to perform manifold projection processing on the online visual perception raw data based on a pre-built set of prior parameters of the successful visual capture manifold, so as to obtain visual manifold points and visual state confidence. The second module is used to perform manifold projection processing on the online tactile perception time series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, to obtain tactile manifold points and tactile state confidence. The third module is used to map the tactile manifold points to the visual manifold space to obtain tactile alignment points; The fourth module is used to calculate the geometric alignment distance based on the visual manifold point and the tactile alignment point; The fifth module is used to calculate the semantic consistency coefficient based on the confidence scores of the visual state and the tactile state. The sixth module is used to fuse the geometric alignment distance and the semantic consistency coefficient to generate a bimodal manifold alignment score; The seventh module is used to generate a robot grasping illusion verification result based on the bimodal manifold alignment score and a preset alignment threshold.

[0008] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising: at least one processor and a memory for communicatively connecting to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for verifying robot grasping illusions based on a tactile-visual manifold.

[0009] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the aforementioned robot grasping illusion verification method based on a tactile-visual manifold.

[0010] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating a robot grasping illusion verification method based on a tactile-visual manifold provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a robot grasping illusion verification system based on a tactile-visual manifold, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0013] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0014] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0015] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0016] Based on this, the embodiments of this application provide a robot grasping illusion verification method and system based on tactile-visual manifold, which aims to accurately identify false positive grasping illusions at the moment the gripper closes, thereby improving verification accuracy.

[0017] The robot grasping illusion verification method, system, electronic device and medium based on tactile-visual manifold provided in this application are specifically described through the following embodiments. First, the robot grasping illusion verification method based on tactile-visual manifold in this application embodiment is described.

[0018] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0019] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0020] The robot grasping illusion verification method based on tactile-visual manifold provided in this application relates to the fields of robot grasping control and multimodal perception technology. This robot grasping illusion verification method based on tactile-visual manifold can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the robot grasping illusion verification method based on tactile-visual manifold, but is not limited to the above forms.

[0021] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0022] Therefore, referring to Figure 1 This application provides a robot grasping illusion verification method based on tactile-visual manifold, the method including the following steps S110 to S180.

[0023] Step S110: Obtain the raw online visual perception data at the moment the robot gripper closes in place, and the online tactile perception timing data within a set time window after the gripper closes.

[0024] In this step, the aforementioned robotic gripper refers to the mechanical structure in the end effector of a robotic body used to grip target objects. It can be a parallel two-finger gripper, a three-finger adaptive gripper, a flexible pneumatic gripper, or a vacuum suction cup gripper, etc. Specifically, in industrial precision sorting scenarios, the aforementioned robotic gripper can be a parallel two-finger electric gripper, which uses a servo motor to drive two parallel fingers to move towards each other to grip surface-mount components; in surgical instrument transfer scenarios, the aforementioned robotic gripper can be a precision three-finger gripper with force feedback function, used to transfer instruments such as scalpels and needle holders.

[0025] The aforementioned moment of complete closure refers to the instant when the gripper moves from an open state to a fully closed state or reaches a preset gripping force threshold. Specifically, data acquisition at this moment can be triggered by the closing signal output by the gripper controller. For example, when the position feedback value of the gripper servo motor reaches the preset closing position threshold, or when the gripping force detected by the gripper fingertip force sensor reaches a preset force threshold (e.g., 5N to 20N), the gripper is determined to be in complete closure. For example, in an SMD component sorting scenario, when the distance between the fingers of the parallel two-finger gripper moves from the initial open position of 12mm to the closed position of 0.5mm, the gripper controller outputs a closing trigger signal, and the system simultaneously starts data acquisition from the visual perception module and the tactile perception module at the instant this signal is triggered.

[0026] The aforementioned raw online visual perception data refers to the unprocessed, raw perception data acquired by the visual perception module at the instant the gripper closes into position. Specifically, this raw online visual perception data may include RGB color images, depth images, point cloud data of the gripper's working area, and relative pose data between the gripper and the target object. For example, in an electronic component assembly scenario, the raw online visual perception data may include a 1280×960 resolution RGB image acquired by an industrial camera, a depth image acquired by a 3D structured light camera, and a 6D pose transformation matrix of the gripper relative to the target component calculated from the hand-eye calibration matrix.

[0027] The aforementioned set time window refers to the time interval used to collect tactile sensing data after the gripper is closed in place. Its duration can be configured according to the specific application scenario. Specifically, the duration of this set time window can be set from 50ms to 500ms. For example, it can be set to 100ms in an SMD component sorting scenario and 200ms in a surgical instrument transfer scenario. The aforementioned online tactile sensing timing data refers to the sequence of tactile signals continuously collected by the tactile sensing module at a preset sampling frequency within the set time window. Specifically, this online tactile sensing timing data can include the pressure value sequence of the gripper fingertip pressure sensor, the shear force sequence of the sliding sensor, and the contact temperature sequence of the temperature sensor. For example, in an SMD component sorting scenario, the tactile sensing module collects 100 tactile sampling points within a 100ms set time window at a sampling frequency of 1000Hz. Each sampling point includes the normal pressure value and tangential shear force value of the two fingertips, forming a 100×4 tactile sensing timing data matrix.

[0028] By simultaneously acquiring raw visual perception data and tactile perception timing data at the moment the gripper closes in place, a spatiotemporal alignment benchmark for the visual and tactile modalities can be established at the same time point when the grasping action is completed. This provides a data foundation for subsequent bimodal manifold alignment verification, thereby avoiding alignment errors caused by asynchronous acquisition timing of the two modalities.

[0029] Step S120: Based on the pre-built prior parameter set of the visual capture success manifold, perform manifold projection processing on the original online visual perception data to obtain visual manifold points and visual state confidence.

[0030] In this step, the aforementioned prior parameter set of the visual grasping success manifold refers to a set of parameters pre-constructed through offline training to characterize the intrinsic topological structure of visual perception data in the successful grasping state. Specifically, this prior parameter set of the visual grasping success manifold may include visual projection mapping parameters, successful state geodesic Gaussian distribution parameters, and manifold metric tensor parameters. Among them, the visual projection mapping parameters are used to map the visual Euclidean feature vector to manifold points on the visual grasping success manifold, the successful state geodesic Gaussian distribution parameters are used to describe the probability distribution shape of the successful grasping state on the manifold, and the manifold metric tensor parameters are used to define the rules for calculating the length of geodesics on the manifold.

[0031] The aforementioned manifold projection processing refers to the nonlinear transformation process of mapping raw online visual perception data from Euclidean space to Riemannian manifold space. Specifically, this manifold projection processing can include three stages: feature extraction, feature vectorization, and manifold mapping. The feature extraction stage converts the raw image and pose data into structured feature representations; the feature vectorization stage converts the structured features into Euclidean feature vectors of fixed dimensions; and the manifold mapping stage uses visual projection mapping to project the Euclidean feature vectors onto manifold points on the manifold.

[0032] The aforementioned visual manifold points refer to the projected coordinates of the original online visual perception data on the successful visual grasping manifold, representing the geometric position of the current visual perception data in the successful grasping state space. The aforementioned visual state confidence is a probability measure of whether a visual manifold point belongs to a successful grasping state, with a value ranging from 0 to 1. The closer the value is to 1, the higher the degree of matching between the current visual perception data and the successful grasping state.

[0033] By performing manifold projection processing on the raw online visual perception data based on a pre-built set of prior parameters of the successful visual grasping manifold, high-dimensional heterogeneous visual perception data can be compressed into a low-dimensional manifold space, preserving the intrinsic topological structure information of the successful grasping state, and outputting quantified visual state confidence, providing geometric and semantic benchmarks for visual modalities for subsequent bimodal alignment calculations.

[0034] Step S130: Based on the pre-constructed tactile stable contact manifold and alignment prior parameter set, perform manifold projection processing on the online tactile perception time series data to obtain tactile manifold points and tactile state confidence.

[0035] In this step, the aforementioned tactile stable contact manifold and alignment prior parameter set refers to a set of parameters pre-constructed through offline training to characterize the intrinsic topological structure of tactile perception data under stable contact states and the alignment relationship between the tactile manifold and the visual manifold. Specifically, this tactile stable contact manifold and alignment prior parameter set may include tactile projection mapping parameters, stable contact geodesic Gaussian distribution parameters, tactile manifold metric tensor parameters, and Riemann isometric alignment mapping parameters, etc. Among them, the tactile projection mapping parameters are used to map the tactile Euclidean eigenvectors to manifold points on the tactile stable contact manifold, the stable contact geodesic Gaussian distribution parameters are used to describe the probability distribution shape of the stable contact state on the tactile manifold, and the Riemann isometric alignment mapping parameters are used to map the tactile manifold points to the visual manifold space.

[0036] The manifold projection process described above is consistent in principle with the manifold projection process in step S120, the difference being the different prior parameter sets and input data types used. The manifold projection process in this step targets online tactile perception time-series data, using tactile projection mapping parameters to project tactile feature vectors onto a stable tactile contact manifold.

[0037] The aforementioned tactile manifold point refers to the projected coordinates of online tactile perception time-series data onto the stable tactile contact manifold, representing the geometric position of the current tactile perception data in the stable contact state space. The aforementioned tactile state confidence is a probability measure of whether the tactile manifold point belongs to a stable contact state, with a value ranging from 0 to 1. The closer the value is to 1, the higher the degree of matching between the current tactile perception data and the stable contact state.

[0038] By performing manifold projection processing on online tactile perception temporal data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, the temporal tactile signal can be compressed into a low-dimensional manifold space, preserving the intrinsic topological information of the stable contact state, while outputting quantized tactile state confidence, providing a geometric and semantic benchmark for tactile modality for subsequent bimodal alignment calculation.

[0039] Step S140: Map the tactile manifold points to the visual manifold space to obtain the tactile alignment points.

[0040] In this step, the aforementioned visual manifold space refers to the Riemannian manifold space defined by the successful visual grasping manifold, where each point corresponds to a geometric representation of visual perception data in the successful grasping state. The aforementioned tactile alignment point refers to the corresponding point in the visual manifold space after the tactile manifold point has undergone Riemann isometric alignment mapping, and it represents the geometric position of the tactile perception data in the visual manifold space.

[0041] Specifically, the process of mapping tactile manifold points to visual manifold space is achieved through Riemann isometric alignment mapping, which is constrained by metric preservation conditions to ensure that the geodesic distance between any two points on the tactile manifold remains unchanged after mapping to visual manifold space.

[0042] By mapping tactile manifold points to the visual manifold space, the manifold representations of the visual and tactile modalities can be directly compared in a unified geometric space, eliminating the problems of inconsistent dimensions and metrics between the two modal manifold spaces, and providing a unified coordinate reference for subsequent geometric alignment distance calculation.

[0043] Step S150: Calculate the geometric alignment distance based on the visual manifold point and the tactile alignment point.

[0044] In this step, the aforementioned geometric alignment distance refers to the Riemann geodesic distance between a visual manifold point and a tactile alignment point in the visual manifold space, which characterizes the degree of alignment between the visual and tactile modalities at the geometric topological level. Specifically, this geometric alignment distance is calculated by integrating the metric tensor along the geodesic line connecting the visual manifold point and the tactile alignment point on the visual manifold.

[0045] By calculating the geometric alignment distance between visual manifold points and tactile alignment points, the consistency of bimodal perception data can be quantitatively characterized from a geometric topological dimension. When the geometric alignment distance is large, it indicates that there is a deviation in the geometric representation between the visual modality and the tactile modality, and the possibility of grasping illusion is high.

[0046] Step S160: Calculate the semantic consistency coefficient based on the visual state confidence and tactile state confidence.

[0047] In this step, the semantic consistency coefficient refers to the measure of the correlation between visual state confidence and tactile state confidence at the probabilistic semantic level, representing the consistency between the visual and tactile modalities in state confidence judgment. Specifically, this semantic consistency coefficient is obtained by calculating the joint probability density of visual state confidence and tactile state confidence using a Gaussian pairwise correlation function.

[0048] By calculating the semantic consistency coefficient between visual state confidence and tactile state confidence, the correlation consistency of bimodal state judgment can be quantitatively characterized from the probabilistic semantic dimension. When the semantic consistency coefficient is low, it indicates that there is a contradiction between the visual modality and the tactile modality in state judgment, and the possibility of grasping illusion is high.

[0049] Step S170: Fuse geometric alignment distance and semantic consistency coefficient to generate bimodal manifold alignment score.

[0050] In this step, the aforementioned bimodal manifold alignment score refers to a quantitative indicator used to characterize the overall alignment degree of bimodal sensing data, calculated by combining the geometric alignment distance and the semantic consistency coefficient. Specifically, this bimodal manifold alignment score is obtained by multiplying the semantic consistency coefficient by an exponential decay term constructed based on the geometric alignment distance. The exponential decay term is used to convert the geometric alignment distance into a decay weight between 0 and 1, where a larger geometric alignment distance results in a smaller decay weight.

[0051] By fusing geometric alignment distance and semantic consistency coefficient to generate bimodal manifold alignment scores, the consistency of bimodal perception data can be comprehensively evaluated from two dimensions: geometric topology and probabilistic semantics. This avoids the one-sidedness of single-dimensional evaluation and improves the accuracy and robustness of illusion recognition.

[0052] Step S180: Generate the robot grasping illusion verification result based on the bimodal manifold alignment score and the preset alignment threshold.

[0053] In this step, the aforementioned preset alignment threshold refers to a pre-configured score threshold used to determine whether the grasp is genuine. Specifically, this preset alignment threshold can be configured according to the accuracy requirements of the specific application scenario. For example, it can be set to 0.75 in an SMD component sorting scenario and 0.85 in a surgical instrument delivery scenario. The aforementioned robot grasping illusion verification result refers to a judgment conclusion generated based on the comparison between the bimodal manifold alignment score and the preset alignment threshold, used to indicate whether the grasp is genuine.

[0054] Specifically, when the bimodal manifold alignment score is greater than or equal to a preset alignment threshold, an indication is generated to capture the true verification result; when the bimodal manifold alignment score is less than the preset alignment threshold, an indication is generated to capture the illusion verification result.

[0055] Furthermore, the above-mentioned robot grasping illusion verification results can also include grading information of grasping illusion. Specifically, grasping illusion can be divided into multiple levels according to the score range of the bimodal manifold alignment score. For example, when the bimodal manifold alignment score is in the range of 0.50 to 0.75, it is judged as a level 1 illusion (slight deviation, corresponding to one-sided contact or slight slippage of the gripper), triggering a fine adjustment of the gripper posture and re-verification; when the bimodal manifold alignment score is in the range of 0.25 to 0.50, it is judged as a level 2 illusion (moderate deviation, corresponding to insufficient gripping force or partial slippage of the gripper), triggering a re-closure of the gripper and re-verification; when the bimodal manifold alignment score is below 0.25, it is judged as a level 3 illusion (severe deviation, corresponding to the gripper completely missing its grip), triggering the abandonment of the current grasping and replanning of the grasping pose.

[0056] In this embodiment, the following steps are taken: First, online visual perception raw data is acquired at the instant the robot gripper closes in place, along with online tactile perception time-series data within a set time window after gripper closure. Second, manifold projection processing is performed on the online visual perception raw data based on a pre-constructed manifold prior parameter set for successful visual grasping, yielding visual manifold points and visual state confidence scores. Third, manifold projection processing is performed on the online tactile perception time-series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, yielding tactile manifold points and tactile state confidence scores. Fourth, tactile manifold points are mapped to the visual manifold space to obtain tactile alignment points. Fifth, geometric alignment distance is calculated based on the visual manifold points and tactile alignment points. Sixth, semantic consistency coefficient is calculated based on the visual state confidence score and tactile state confidence score. Seventh, the geometric alignment distance and semantic consistency coefficient are fused to generate a bimodal manifold alignment score. Finally, a robot grasping illusion verification result is generated based on the bimodal manifold alignment score and a preset alignment threshold. Thus, by simultaneously acquiring visual and tactile perception data at the moment the gripper closes in place, and constructing independent visual grasping success manifolds and tactile stable contact manifolds respectively, and using tactile perception as an independent physical truth benchmark, the degree of bimodal matching is quantitatively characterized from both geometric topology and probabilistic semantic dimensions through the geometric alignment distance and semantic consistency coefficient of the bimodal manifold space. At the moment of grasping completion, false positive grasping illusions such as visual judgment of successful grasping but actual gripper gap, one-sided contact, insufficient gripping force, or slippage can be identified. This solves the technical problems in existing technologies where grasping success judgment mainly relies on visual information, making it difficult to identify false positive grasping illusions at the moment the gripper closes in place, resulting in low accuracy of grasping illusion verification. It achieves real-time blocking of grasping illusions in advance, significantly improving the grasping reliability in precision operation scenarios.

[0057] In some embodiments of this application, step S120 above involves manifold projection processing of the online visual perception raw data based on a pre-built set of prior parameters of the visual capture success manifold, to obtain visual manifold points and visual state confidence, including steps S210 to S240 below.

[0058] Step S210: Convert the original online visual perception data into visual Euclidean feature vectors.

[0059] In this step, the aforementioned visual Euclidean feature vector refers to a fixed-dimensional vector located in Euclidean space, obtained after feature extraction and vectorization of the original online visual perception data. Specifically, this visual Euclidean feature vector can be constructed by the logarithmic mapping of the relative pose transformation matrix between the gripper and the target object, the pose increment, and the concatenation of dimensionality-reduced features from the point cloud data.

[0060] Specifically, the process of converting raw online visual perception data into visual Euclidean feature vectors may include the following sub-steps: First, transform the relative pose of the gripper and the target object. Lie group The logarithmic mapping on the logarithm yields a 6-dimensional Lie algebra vector. Secondly, the difference between the current pose transformation matrix and the previous capture pose transformation matrix is ​​calculated to obtain the pose increment. Then, the point cloud data of the gripper's working area... Principal component analysis was performed for dimensionality reduction, and the top 12 principal component components were extracted to obtain the dimensionality-reduced features. Finally, the above three features are concatenated into a visual Euclidean feature vector, the calculation formula of which is as follows: ; in, This represents the visual Euclidean eigenvector. This represents a vector concatenation operation. Represents the pose transformation matrix exist Logarithmic mappings on Lie groups Indicates the pose increment. Representing point cloud data The 12-dimensional features after dimensionality reduction by principal component analysis.

[0061] Step S220: Using visual projection mapping, project the visual Euclidean feature vector onto the visually captured manifold to obtain the visual manifold points.

[0062] In this step, the aforementioned visual projection mapping refers to the nonlinear mapping function in the prior parameter set of the successfully captured manifold used to map Euclidean eigenvectors to points on the Riemannian manifold. Specifically, this visual projection mapping can be implemented using an exponential mapping method, that is, firstly, the Euclidean eigenvectors are transformed into tangent vectors in the tangent space of the manifold through a multi-layer fully connected network, and then the tangent vectors are mapped to points on the manifold through an exponential mapping.

[0063] The aforementioned visual grasping success manifold refers to a Riemannian manifold pre-constructed through offline training to characterize the intrinsic topological structure of visual perception data under successful grasping states. Specifically, each point on this visual grasping success manifold corresponds to a visual perception data representation under a successful grasping state, and the geodesic distance between two points on the manifold reflects the topological difference between the two successful grasping states.

[0064] The aforementioned visual manifold points refer to the projected coordinates of the visual Euclidean eigenvectors onto the successfully captured manifold after visual projection mapping. Specifically, the formula for calculating visual manifold points is as follows: ; in, Represents visual manifold points, Represents visual projection mapping, This represents the visual Euclidean eigenvector. Represents an exponential mapping on a manifold. The reference point for the successful visual capture manifold (i.e., the mean vector of the geodesic Gaussian distribution of the successful state). The weight matrix represents the visual projection mapping. This represents a non-linear activation function (such as ReLU or tanh function). Represents the characteristic transformation matrix, This represents the bias vector.

[0065] Step S230: Calculate the probability of successful capture based on the geodesic Gaussian distribution of the successful state.

[0066] In this step, the aforementioned geodesic Gaussian distribution of successful states refers to a pre-constructed geodesic Gaussian distribution model used to describe the probability distribution of successful grasping states on the visual grasping success manifold. Specifically, this geodesic Gaussian distribution of successful states uses the reference point of the visual grasping success manifold as a reference point. The mean is the covariance tensor of the success states. For covariance, the manifold metric tensor is used. For metric purposes, the probability density of any point on the manifold belonging to a successful grasp state is defined.

[0067] The aforementioned successful grasp probability refers to the probability density value of the visual manifold point under a geodesic Gaussian distribution in a successful state. Specifically, the formula for calculating the successful grasp probability is as follows: ; in, This indicates the probability of a successful capture. This represents the natural exponential function. Represents the geodesic distance function on the manifold. Represents visual manifold points, The vector representing the mean of the geodesic Gaussian distribution of the successful state. Represents the manifold metric tensor. This represents the covariance tensor of the successful state. The above geodesic distance function... The calculation formula is as follows: ; in, and Represent two points on a manifold. Represents the manifold metric tensor. Denotes the covariance tensor. Indicates connection and Geodesic parameterization curves ( (This is the arc length parameter, with a value ranging from 0 to 1). This represents the tangent vector of the geodesic.

[0068] Step S240: Normalize the success rate of grabbing to obtain the visual state confidence score.

[0069] In this step, the normalization process mentioned above refers to mapping the successful grasp probability to a standardized range of 0 to 1, so that it can be directly used as the visual state confidence score in subsequent calculations. Specifically, when the successful grasp probability is already in the range of 0 to 1, the normalization process can directly use this value; when the successful grasp probability exceeds the range of 0 to 1, max-min normalization or the Sigmoid function can be used to map it to the range of 0 to 1.

[0070] The aforementioned visual state confidence score refers to a quantitative indicator obtained after normalization, used to characterize the degree of matching between the current visual perception data and the successful grasping state.

[0071] In this embodiment, the online visual perception raw data is converted into visual Euclidean feature vectors. The visual Euclidean feature vectors are then projected onto the successful visual grasping manifold using visual projection mapping to obtain visual manifold points. The successful grasping probability is calculated based on the geodesic Gaussian distribution of the successful state. The successful grasping probability is then normalized to obtain the visual state confidence. This achieves a compressed mapping from high-dimensional heterogeneous visual perception data to low-dimensional manifold representation, preserving the intrinsic topological structure information of the successful grasping state. This provides the geometric coordinates and semantic confidence of the visual modality for subsequent bimodal alignment calculations.

[0072] In some embodiments of this application, step S130 above involves manifold projection processing of online tactile perception time-series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set to obtain tactile manifold points and tactile state confidence, including steps S310 to S340 below.

[0073] Step S310: Convert the online tactile perception time-series data into tactile Euclidean feature vectors.

[0074] In this step, the aforementioned tactile Euclidean feature vector refers to a fixed-dimensional vector located in Euclidean space, obtained by concatenating statistical and histogram feature extractions from online tactile perception time-series data. Specifically, this tactile Euclidean feature vector can be constructed by concatenating the statistical features and distribution histogram features of the tactile time-series signal.

[0075] Specifically, the process of converting online tactile perception time-series data into tactile Euclidean feature vectors can include the following sub-steps: First, extract statistical features from the signals of each channel in the online tactile perception time-series data, including 10 statistical quantities such as mean, variance, skewness, kurtosis, maximum value, minimum value, zero-crossing rate, energy, root mean square value, and signal variation coefficient, to obtain a statistical feature vector. Secondly, a distribution histogram is calculated for each channel signal in the online tactile perception time-series data. The signal value range is divided into 8 intervals, and the number of sampling points in each interval is counted to obtain the histogram feature vector. Finally, the two features mentioned above are concatenated into a tactile Euclidean feature vector, calculated using the following formula: ; in, This represents the Euclidean eigenvector of tactile sensation. This represents a vector concatenation operation. Represents tactile timing signals The 10-dimensional statistical eigenvectors, Represents the distribution of tactile temporal signals The 8-dimensional histogram feature vector.

[0076] Step S320: Using tactile projection mapping, project the tactile Euclidean eigenvectors onto the tactile stable contact manifold to obtain tactile manifold points.

[0077] In this step, the aforementioned tactile projection mapping refers to the nonlinear mapping function used by the tactile stable contact manifold and the alignment prior parameter set to map tactile Euclidean eigenvectors to manifold points on the tactile stable contact manifold. Specifically, this tactile projection mapping is structurally similar to the visual projection mapping, also using an exponential mapping form, the difference being the mapping parameters and the manifold dimension.

[0078] The aforementioned tactile stable contact manifold refers to a Riemannian manifold pre-constructed through offline training to characterize the intrinsic topological structure of tactile perception data under stable contact states. Specifically, each point on the tactile stable contact manifold corresponds to a tactile perception data representation under a stable contact state, and the geodesic distance between two points on the manifold reflects the topological difference between the two stable contact states.

[0079] The aforementioned tactile manifold point refers to the projected coordinates of the tactile Euclidean eigenvector onto the stable tactile contact manifold after tactile projection mapping. Specifically, the formula for calculating the tactile manifold point is as follows: ; in, Represents tactile manifold points, Represents haptic projection mapping, This represents the Euclidean eigenvector of tactile sensation. Represents an exponential mapping on a manifold. The reference point representing the stable contact manifold of tactile sensation (i.e., the mean vector of the stable contact geodesic Gaussian distribution). The weight matrix represents the haptic projection mapping. Represents a non-linear activation function. Represents the characteristic transformation matrix, This represents the bias vector.

[0080] Step S330: Calculate the stable contact probability based on the geodesic Gaussian distribution of stable contact.

[0081] In this step, the aforementioned stable contact geodesic Gaussian distribution refers to a pre-constructed geodesic Gaussian distribution model used to describe the probability distribution of stable contact states on the tactile stable contact manifold. Specifically, this stable contact geodesic Gaussian distribution is based on the reference point of the tactile stable contact manifold. The mean is used to stabilize the contact covariance tensor. For covariance, the tensor is measured using a tactile manifold. For metric purposes, the probability density of any point on the manifold belonging to a stable contact state is defined.

[0082] The aforementioned steady-state contact probability refers to the probability density value of the tactile manifold point under a steady-state contact geodesic Gaussian distribution. Specifically, the formula for calculating the steady-state contact probability is as follows: ; in, Indicates the probability of stable contact. This represents the natural exponential function. Represents the geodesic distance function on the manifold. Represents tactile manifold points, This represents the mean vector of a stable contact geodesic Gaussian distribution. Represents the tactile manifold metric tensor. This represents the stable contact covariance tensor.

[0083] Step S340: Normalize the stable contact probability to obtain the tactile state confidence.

[0084] In this step, the normalization process described above is consistent in principle with the normalization process in step S240, the difference being that the object of the normalization process is the stable contact probability. Specifically, when the stable contact probability is already in the range of 0 to 1, the normalization process can directly use this value; when the stable contact probability exceeds the range of 0 to 1, max-min normalization or the Sigmoid function can be used to map it to the range of 0 to 1.

[0085] The aforementioned tactile state confidence level refers to a quantitative indicator obtained after normalization, used to characterize the degree of matching between the current tactile perception data and the stable contact state.

[0086] In this embodiment, online tactile perception time-series data is converted into tactile Euclidean feature vectors. Tactile projection mapping is used to project the tactile Euclidean feature vectors onto a stable tactile contact manifold to obtain tactile manifold points. The stable contact probability is calculated based on the stable contact geodesic Gaussian distribution. The stable contact probability is normalized to obtain the tactile state confidence. This achieves a compressed mapping from time-series tactile signals to low-dimensional manifold representations, preserving the intrinsic topological information of the stable contact state and providing the geometric coordinates and semantic confidence of the tactile modality for subsequent bimodal alignment calculations.

[0087] In some embodiments of this application, before mapping the tactile manifold points to the visual manifold space in step S140 to obtain the tactile alignment points, the method further includes a step of training the Riemann isometric alignment mapping, including the following steps S410 to S430.

[0088] Step S410: Obtain multiple sets of training sample sets collected simultaneously. Each set of training samples includes visual perception training data and tactile perception training data collected simultaneously, as well as corresponding sample labels.

[0089] In this step, the training sample set refers to the offline dataset used to train the Riemann isometric alignment mapping. Specifically, each training sample set includes visual perception training data and tactile perception training data collected synchronously during the same grasping process, as well as sample labels used to indicate whether the grasp was truly successful.

[0090] The aforementioned synchronously acquired visual perception training data and tactile perception training data refer to the perception data collected by the visual sensor and tactile sensor respectively at the same moment when the gripper closes into position. Specifically, the visual perception training data includes the relative pose transformation matrix and point cloud data between the gripper and the target object, while the tactile perception training data includes the timing signal of the tactile sensor array within a set time window after the gripper closes.

[0091] The aforementioned sample labels refer to annotation information used to indicate the authenticity of the grasp. Specifically, sample labels can include two categories: successful grasp and grasp illusion. The grasp illusion category can be further subdivided into subcategories such as gripper gap, one-sided contact, insufficient gripping force, and slippage.

[0092] Step S420: Based on the sample label indicating that the sample was successfully captured, construct feature pairs between the visual perception training data and the tactile perception training data to obtain the feature training set.

[0093] In this step, the aforementioned feature pairs refer to paired data consisting of visual manifold points corresponding to visual perception training data and tactile manifold points corresponding to tactile perception training data simultaneously collected during the same grasping process. Specifically, the process of constructing feature pairs includes: first, selecting samples from the training sample set whose labels indicate successful grasping; then, extracting visual and tactile features from each selected sample to obtain visual Euclidean feature vectors and tactile Euclidean feature vectors; next, projecting the visual and tactile Euclidean feature vectors onto the corresponding manifold spaces using pre-constructed visual and tactile projection maps to obtain visual and tactile manifold points; finally, forming feature pairs from the visual and tactile manifold points corresponding to the same grasping process, and the set of all feature pairs constitutes the feature training set.

[0094] The aforementioned feature training set refers to a dataset consisting of multiple feature pairs used to train the Riemann isometric alignment mapping.

[0095] Step S430: Using the preset measurement conditions as constraints, train the Riemann isometric alignment map using the feature training set.

[0096] In this step, the aforementioned preset metric preservation condition refers to the constraint applied during training to ensure that the Riemann isometric alignment mapping maintains the geodesic distance on the manifold as much as possible. Specifically, the preset metric preservation condition requires that the deviation between the geodesic distance between any two points on the tactile stable contact manifold after the Riemann isometric alignment mapping and the geodesic distance between the corresponding two points on the visually grasped manifold is less than a preset metric deviation threshold.

[0097] The aforementioned Riemann isometric alignment mapping refers to a nonlinear mapping function that maps tactile manifold points from tactilely stable contact manifolds to visual manifold spaces. Specifically, this Riemann isometric alignment mapping is implemented using an exponential mapping form parameterized by the Jacobian matrix, and its calculation formula is as follows: ; in, Indicates the tactile alignment point (i.e., the tactile manifold point). (Projection points in visual manifold space after Riemann isometric alignment mapping). This represents the Riemann isometric alignment map. Represents an exponential mapping on the visual manifold. The reference point representing the successful visual capture of the manifold. Denotes the Jacobian matrix of the Riemann isometric alignment map. Represents a logarithmic mapping on a tactile manifold. The reference point representing the stable tactile contact manifold, Represents the tactile manifold point.

[0098] Specifically, the process of training the Riemann isometric alignment map may include the following sub-steps: First, initialize the Jacobian matrix. The matrix is ​​either an identity matrix or a random orthogonal matrix; then, a set of feature pairs is selected from the feature training set. Calculate tactile manifold points Logarithmic mapping on tactile manifolds Then through the Jacobian matrix The transformation yields the tangent vector in the tangent space of the visual manifold. Finally, through exponential mapping Obtain tactile alignment point Next, the haptic alignment point is calculated. ′ and visual manifold point The geodesic distance between feature pairs is used as the alignment error, and the metric-preserving constraint loss (i.e., the deviation between the geodesic distance between feature pairs on the tactile manifold and the geodesic distance between corresponding feature pairs on the visual manifold) is calculated. Finally, the Jacobian matrix is ​​updated using the weighted sum of the alignment error and the metric-preserving constraint loss as the total loss through gradient descent. Until the total loss converges.

[0099] In this embodiment, multiple sets of synchronously acquired training sample sets are obtained. Based on the sample label indication, feature pairs are constructed to obtain a feature training set for samples that have been successfully captured. The Riemann equidistant alignment mapping is trained using the feature training set under the constraint of a preset metric preservation condition. This ensures that the mapping from the tactile manifold space to the visual manifold space is as equidistant as possible in the sense of Riemann metric, so that the geometric alignment distance between the tactile alignment point and the visual manifold point can truly reflect the topological differences between the bimodal perception data, and avoids the interference of metric distortion introduced by nonlinear mapping on the accuracy of illusion verification.

[0100] In some embodiments of this application, the calculation of geometric alignment distance based on visual manifold points and tactile alignment points in step S150 above may include the following steps S510 to S540.

[0101] Step S510: On the visual manifold, determine the geodesic connecting the visual manifold point and the tactile alignment point.

[0102] In this step, the aforementioned geodesic line refers to the line connecting the visual manifold points on the successfully captured visual manifold. Alignment with tactile senses The shortest path curve. Specifically, a geodesic is a locally shortest curve on a manifold, with zero geodesic curvature at every point on it. The geodesic can be determined by numerically solving the geodesic equation.

[0103] Specifically, the process of determining geodesics may include the following sub-steps: First, at the visual manifold point... Alignment with tactile senses Linear interpolation is performed between them to obtain the initial path; then, the visual manifold metric tensor is used. To measure this, the geodesic curvature at each point on the initial path is calculated. Then, the positions of each point on the path are adjusted using the gradient descent method to gradually reduce the geodesic curvature to zero. Finally, when the geodesic curvature of all points on the path is less than the preset curvature threshold, the adjusted path is output as a geodesic line.

[0104] Step S520: Integrate the metric tensor along the geodesic to obtain the geodesic integral result.

[0105] In this step, the geodesic integral result refers to the value obtained by performing path integration on the metric tensor along the geodesic. Specifically, the geodesic integral result reflects the path length of the geodesic in the sense of manifold metric, and its calculation formula is as follows: ; Specifically, the process of integrating the metric tensor along the geodesic includes: first, parameterizing the geodesic into an arc length parameter. function ,in The value of is in the range of 0 to 1; then, the tangent vector at each point on the geodesic line is calculated. Next, the visual manifold metric tensor is used. Calculate the norm of the tangent vector; finally, perform numerical integration along the geodesic with respect to the norm of the tangent vector to obtain the geodesic integral result.

[0106] Step S530: Using the successful state covariance tensor, the geodesic tangent vector in the geodesic integral result is weighted to obtain the covariance-weighted Riemann geodesic distance.

[0107] In this step, the aforementioned successful state covariance tensor refers to the tensor in the prior parameter set of the successful visual grasping manifold used to describe the variance of the successful grasping state distribution in each direction of the manifold. Specifically, the successful state covariance tensor... By weighting the geodesic tangent vectors, the contribution of geodesic distance is attenuated in directions with large variance of the successful grasp state distribution, and amplified in directions with small variance of the distribution, thus more accurately reflecting the topological differences between the visual manifold point and the tactile alignment point.

[0108] The covariance-weighted Riemann geodesic distance mentioned above refers to the geodesic distance after being weighted by the successful state covariance tensor. Specifically, the formula for calculating the covariance-weighted Riemann geodesic distance is as follows: ; in, This represents the covariance-weighted Riemannian geodesic distance. and This represents two points on the manifold (in this step, they are visual manifold points). Alignment with tactile senses ), Represents the manifold metric tensor (in this step, the visual manifold metric tensor). ), This represents the covariance tensor (in this step, the covariance tensor represents the successful state). ), Indicates connection and Geodesic parameterization curves, This represents the tangent vector of the geodesic.

[0109] Step S540: Use the covariance-weighted Riemann geodesic distance as the geometric alignment distance.

[0110] In this step, the aforementioned geometric alignment distance refers to a quantitative indicator used to characterize the geometric topological difference between the visual manifold point and the tactile alignment point on the manifold where visual grasping was successful. Specifically, the covariance-weighted Riemann geodesic distance calculated in step S530 is directly used as the geometric alignment distance. Its expression is as follows: ; in, Indicates the geometric alignment distance. This represents the covariance-weighted Riemannian geodesic distance function. Represents visual manifold points, Indicates the tactile alignment point. Represents the visual manifold metric tensor. This represents the covariance tensor of the successful state.

[0111] In this embodiment, a geodesic connecting the visual manifold point and the tactile alignment point is determined on the visual manifold. The metric tensor is integrated along the geodesic to obtain the geodesic integral result. The geodesic tangent vector in the geodesic integral result is weighted using the successful state covariance tensor to obtain the covariance-weighted Riemann geodesic distance. The covariance-weighted Riemann geodesic distance is used as the geometric alignment distance, thereby quantitatively characterizing the spatial alignment degree between visual perception and tactile perception within the geometric topological framework of the Riemann manifold. The covariance weighting mechanism enables the geometric alignment distance to adaptively highlight deviations in the direction related to the successful grasping state, thereby improving the sensitivity of illusion verification.

[0112] In some embodiments of this application, the semantic consistency coefficient is calculated based on the visual state confidence and the tactile state confidence in step S160, including the following steps S610 to S630.

[0113] Step S610: Perform a standard normal inverse cumulative distribution transformation on the visual state confidence to obtain the visual transformation value.

[0114] In this step, the aforementioned standard normal inverse cumulative distribution transformation refers to mapping the probability values ​​in the interval from 0 to 1 to quantiles under the standard normal distribution. Specifically, the standard normal inverse cumulative distribution transformation uses the inverse cumulative distribution function of the standard normal distribution. This achieves the goal of increasing the confidence level of visual states. The mapping is to the quantiles under the standard normal distribution.

[0115] The aforementioned visual transformation value refers to the standard normal quantile obtained after transforming the visual state confidence score through a standard normal inverse cumulative distribution. Specifically, the formula for calculating the visual transformation value is as follows: ; in, Represents the visual transformation value. This represents the inverse cumulative distribution function of the standard normal distribution. This indicates the confidence level of the visual state.

[0116] Step S620: Perform a standard normal inverse cumulative distribution transformation on the tactile state confidence to obtain the tactile transformation value.

[0117] In this step, the standard normal inverse cumulative distribution transformation described above is consistent in principle with the transformation in step S610, the difference being that the object being processed is the confidence level of tactile state. Specifically, the formula for calculating the tactile transformation value is as follows: ; in, Indicates the tactile transformation value. This represents the inverse cumulative distribution function of the standard normal distribution. This indicates the confidence level of the tactile state.

[0118] Step S630: Using the preset view-touch dependency coefficient as a parameter, calculate the joint probability density using the Gaussian pairwise association function to obtain the semantic consistency coefficient.

[0119] In this step, the aforementioned preset view-touch dependency coefficient refers to a parameter used to describe the statistical dependence strength between the visual modality and the tactile modality. Specifically, the view-touch dependency coefficient... The value range is from -1 to 1, when A value close to 1 indicates a high positive correlation between the visual and tactile modalities. When the value is close to 0, it indicates that the two modes are independent. A value close to -1 indicates a high degree of negative correlation between the two modes. The view-touch dependence coefficient... The correlation coefficient between the confidence levels of visual and tactile states in truly successful capture samples can be determined by offline statistical analysis.

[0120] The Gaussian pairwise association function mentioned above refers to a binary Gaussian joint probability density function with the view-touch dependency coefficient as a parameter. Specifically, the formula for calculating the semantic consistency coefficient is as follows: ; in, Represents the semantic consistency coefficient. This represents the Gaussian pairwise correlation function. This represents the inverse cumulative distribution function of the standard normal distribution. Indicates the confidence level of the visual state. Indicates the confidence level of tactile state. This represents the preset view-touch dependency coefficient.

[0121] In this embodiment, visual and tactile state confidence scores are transformed by standard normal inverse cumulative distribution to obtain visual and tactile transformation values, respectively. Using a preset visual-tactile dependency coefficient as a parameter, a Gaussian pairwise correlation function is used to calculate the joint probability density to obtain the semantic consistency coefficient. This quantitatively characterizes the consistency between the visual and tactile modal judgment results at the probabilistic semantic level. The Gaussian pairwise correlation function can capture the nonlinear statistical dependency between the two modalities, enabling the semantic consistency coefficient to sensitively detect semantic contradictions where visual judgment is successful but tactile judgment fails.

[0122] In some embodiments of this application, the geometric alignment distance and semantic consistency coefficient are fused in step S170 above to generate a bimodal manifold alignment score, including steps S710 to S730 below.

[0123] Step S710: Calculate the attenuation amount based on the geometric alignment distance and the preset alignment scale parameters.

[0124] In this step, the aforementioned preset alignment scale parameter refers to a hyperparameter used to control the rate at which the geometric alignment distance affects the alignment fraction decay of the bimodal manifold. Specifically, the preset alignment scale parameter... The larger the value of , the more gradual the effect of the geometric alignment distance on the alignment score of the bimodal manifold; preset alignment scale parameter The smaller the value of , the steeper the effect of the geometric alignment distance on the alignment score of the bimodal manifold. Preset alignment scale parameter. It can be configured according to the accuracy requirements of specific application scenarios.

[0125] The aforementioned attenuation refers to the value after normalizing the geometric alignment distance using a preset alignment scale parameter. Specifically, the formula for calculating the attenuation is as follows: ; in, Indicates the amount of attenuation. Indicates the geometric alignment distance. This indicates the preset alignment scale parameter.

[0126] Step S720: Construct an exponential decay term based on the decay amount.

[0127] In this step, the exponential decay term refers to a function term with the natural exponential function as its base and the decay amount as its exponent. Specifically, the formula for calculating the exponential decay term is as follows: ; in, Represents the exponentially decaying term. This represents the natural exponential function. Indicates the amount of attenuation. Indicates the geometric alignment distance. This indicates the preset alignment scale parameter.

[0128] Step S730: Generate bimodal manifold alignment scores based on semantic consistency coefficients and exponential decay terms.

[0129] In this step, the bimodal manifold alignment score refers to the fusion score combining geometric alignment distance and semantic consistency coefficient. Specifically, the formula for calculating the bimodal manifold alignment score is as follows: ; in, Represents the alignment fraction of a bimodal manifold. Represents the semantic consistency coefficient. This represents the natural exponential function. Indicates the geometric alignment distance. This indicates the preset alignment scale parameter.

[0130] Specifically, bimodal manifold alignment fractions Based on semantic consistency coefficient and exponential decay term The product constitutes the semantic consistency coefficient. The consistency of bimodal judgment results is measured from a probabilistic semantic dimension, while the exponential decay term measures the closeness of bimodal manifold representations from a geometric topological dimension. When the visual manifold point and the tactile alignment point are close in position on the manifold (small geometric alignment distance, exponential decay term close to 1) and the visual state confidence and tactile state confidence are semantically consistent (semantic consistency coefficient close to 1), the bimodal manifold alignment score is close to 1, indicating a realistic grasp; when the geometric alignment distance is large or the semantic consistency coefficient is low, the bimodal manifold alignment score is low, indicating the existence of grasping illusion.

[0131] In this embodiment, the attenuation amount is calculated based on the geometric alignment distance and the preset alignment scale parameter. An exponential attenuation term is constructed based on the attenuation amount. A bimodal manifold alignment score is generated based on the semantic consistency coefficient and the exponential attenuation term. This integrates the alignment metric of the geometric topology dimension and the consistency metric of the probabilistic semantic dimension into a unified quantitative score. The exponential attenuation mechanism ensures the sensitivity of the bimodal manifold alignment score to geometric deviations. The product fusion mechanism ensures that the bimodal manifold alignment score will decrease significantly when any modality is abnormal, thus improving the robustness of the illusion detection.

[0132] To make the technical solution of this application clearer, the robot grasping illusion verification method provided in the embodiments of this application will be fully described below in conjunction with the SMD component sorting scenario.

[0133] In SMD component sorting scenarios, the robot is equipped with a two-finger parallel gripper, with a tactile sensor array (4 channels) installed at the gripper's fingertips, and a depth camera as a vision sensor mounted on the robot's wrist. The robot needs to pick up SMD components with a 0402 package size from the tray and transfer them to the inspection station. Since the SMD components are only 1.0 mm × 0.5 mm in size, the gripper is prone to grasping illusions during the closing process, such as missing the gripper, one-sided contact, insufficient gripping force, or slippage. If these illusions are not identified in time, subsequent processes will be interrupted or the components will be damaged.

[0134] In this scenario, the offline preparation phase of the robot grasping illusion verification system includes the following steps: First, 5000 sets of synchronized visual perception training data and tactile perception training data are collected, of which 3500 sets are labeled as successful grasping and 1500 sets are labeled as grasping illusions (including 800 sets of empty grasping, 400 sets of unilateral contact, and 300 sets of insufficient gripping force or slippage); then, using the 3500 sets of training samples of successful grasping, visual grasping success manifolds and tactile stable contact manifolds are constructed respectively, and visual projection mapping parameters (weight matrix) are determined. Dimensions are 64×24, and the feature transformation matrix is... 24×24 dimension, bias vector (Dimension 24), haptic projection mapping parameters (weight matrix) Dimensions are 48×18, and the feature transformation matrix is... 18×18 dimension, bias vector (Dimension 18) Visual manifold reference point (Dimension 64), tactile manifold reference point (Dimension 48), Visual manifold metric tensor tactile manifold metric tensor Successful state covariance tensor and stable contact covariance tensor Next, constrained by a preset metric preservation condition, the Riemann isometric alignment mapping is trained using 3500 sets of feature pairs to obtain the Jacobian matrix. (Dimensions are 64×48), the preset metric bias threshold is set to 0.05, the learning rate is set to 0.001, and training is performed for 200 epochs until the total loss converges; finally, the preset alignment scale parameter is configured. The view-touch dependency coefficient is 1.0. The value is 0.65, and the preset alignment threshold is 0.75.

[0135] In this scenario, the online operation phase of the robot-based hallucination verification system includes the following steps: The first step is that when the robot gripper closes into position, the data acquisition module triggers the depth camera to collect raw visual perception data at the instant the gripper closes, including the relative pose transformation matrix between the gripper and the SMD components. (4×4 homogeneous transformation matrix) and point cloud data of the gripper working area (1024 three-dimensional points), and simultaneously acquire the timing signal of the tactile sensor array within a 100-millisecond time window after the gripper closes. (100×4 matrix) and distribution data .

[0136] The second step involves the visual manifold projection module processing the raw visual perception data. First, it processes the pose transformation matrix. conduct Logarithmic mappings on Lie groups yield 6-dimensional Lie algebra vectors. Calculate pose increment (6-dimensional) point cloud data Principal component analysis was performed to reduce the dimensionality and obtain 12-dimensional features. The 24-dimensional visual Euclidean feature vector is obtained by concatenation. Then, visual projection mapping is used. Will Projecting onto the successfully captured visual manifold yields 64-dimensional visual manifold points. Next, the probability of successful capture is calculated based on the geodesic Gaussian distribution of the successful state. Finally, the probability of successful capture is normalized to obtain the visual state confidence score. .

[0137] The third step involves the tactile manifold projection module processing the tactile perception timing data. First, it processes the tactile timing signal. Extracting 10-dimensional statistical features (Including mean, variance, skewness, kurtosis, maximum value, minimum value, zero-crossing rate, energy, root mean square value, and signal variation coefficient), for distributed data Calculate 8D histogram features The 18-dimensional tactile Euclidean eigenvectors are obtained by concatenation. Then, haptic projection mapping is used. Will Projecting onto a tactilely stable contact manifold yields 48-dimensional tactile manifold points. Next, the probability of stable contact is calculated based on the geodesic Gaussian distribution of stable contact. Finally, the stable contact probability is normalized to obtain the tactile state confidence score. .

[0138] The fourth step involves the manifold alignment mapping module utilizing a pre-trained Riemann isometric alignment mapping. tactile manifold points Logarithmic mapping on tactile manifolds Transformed into a tangent vector, via the Jacobian matrix Transform to the tangent space of the visual manifold, and then apply an exponential mapping on the visual manifold. Obtain tactile alignment point .

[0139] Fifth, the geometric alignment calculation module determines the points connecting the visual manifolds on the visual manifold. Alignment with tactile senses The geodesics, along which the metric tensor is... Integrate using the success state covariance tensor We weight the geodesic tangent vectors to obtain the covariance-weighted Riemann geodesic distance, and use it as the geometric alignment distance. .

[0140] Step 6: The semantic consistency calculation module calculates the confidence level of the visual state. Confidence of tactile state Visual transformation values ​​were obtained by performing standard normal inverse cumulative distribution transformations respectively. and tactile transformation value With view-touch dependency coefficient Using a Gaussian pairwise association function with a parameter of 0.65, the joint probability density is calculated to obtain the semantic consistency coefficient. .

[0141] Step 7: The alignment score generation module aligns the scores according to the geometric alignment distance. and preset alignment scale parameters Calculate the attenuation for 1.0 Construct an exponential decay term based on the decay amount. According to the semantic consistency coefficient and exponential decay term According to the formula Generate bimodal manifold aligned fractions .

[0142] Step 8: The hallucination verification result generation module aligns the scores of the bimodal manifold. Compare with the preset alignment threshold of 0.75. When When the value is greater than or equal to 0.75, an instruction is generated to retrieve the actual verification result, allowing the robot to perform subsequent handling procedures; when... When the value is less than 0.75, a verification result indicating the capture of hallucinations is generated, and based on... The score range in which the illusion is located will classify the illusion as a level one illusion ( If the value is in the range of 0.50 to 0.75, it corresponds to one-sided contact or slight slippage of the gripper, triggering a fine-tuning of the gripper posture followed by re-verification; secondary illusion ( A value between 0.25 and 0.50 indicates insufficient gripping force or partial slippage of the grippers, triggering a re-closing and re-verification of the grippers. (This also indicates a level three illusion.) If the value is below 0.25, the corresponding gripper will completely miss its target, triggering the abandonment of the current grip and the replanning of the gripping pose.

[0143] In this scenario, when the gripper actually grasps the SMD component, the visual manifold point... Alignment with tactile senses Proximity in visual manifold, geometric alignment distance The confidence level for visual states is 0.15. It is 0.92 (corresponding to) (1.41), tactile state confidence level It is 0.88 (corresponding to) (1.17), semantic consistency coefficient The bimodal manifold alignment score is 0.85. for Since 0.84 is greater than 0.75, the system generates and captures the actual verification results.

[0144] When the gripper is empty, the visual manifold point Alignment with tactile senses Significant deviation in position on the visual manifold, geometric alignment distance. The confidence level for visual states is 2.83. It is 0.92 (corresponding to) The confidence level was 1.41, but the confidence level of the tactile state was... It is 0.12 (corresponding to) (-1.17), semantic consistency coefficient The alignment score for the bimodal manifold is 0.08. for Since 0.001 is less than 0.25, the system judges it as a level 3 illusion, triggering the abandonment of the current grasp and the replanning of the grasp pose.

[0145] When the gripper contacts the SMD component on one side, the visual manifold point Alignment with tactile senses There is a moderate deviation in position on the visual manifold, geometric alignment distance. The confidence level for visual states is 0.92. It is 0.85 (corresponding to) (1.04), tactile state confidence level It is 0.45 (corresponding to) (with a value of -0.13), semantic consistency coefficient The alignment score for the bimodal manifold is 0.32. for Since 0.21 is less than 0.25, the system judges it as a level 3 illusion, triggering the abandonment of the current grasp and the replanning of the grasp pose.

[0146] When insufficient gripping force of the grippers causes micro-slippage of SMD components, visual manifold points Alignment with tactile senses There is a slight deviation in position on the visual manifold, and the geometric alignment distance is... The confidence level for visual states is 0.58. It is 0.88 (corresponding to) (1.17), tactile state confidence level It is 0.62 (corresponding to) (0.31), semantic consistency coefficient The bimodal manifold alignment score is 0.58. for Since 0.49 falls within the range of 0.25 to 0.50, the system determines it to be a level two hallucination, triggering the gripper to close again and re-verify.

[0147] In some embodiments of this application, the above-described robot-grasping illusion verification method may further include an incremental manifold update step.

[0148] Specifically, after the illusion verification result generation module generates an instruction to capture the real verification result, it can use the original visual perception data, tactile perception time-series data, and corresponding visual manifold points and tactile manifold points in the current capture process as new samples to incrementally update the prior parameter set of the visual capture success manifold and the prior parameter set of the tactile stable contact manifold and alignment.

[0149] The aforementioned incremental manifold update refers to locally adjusting manifold parameters using real, successfully captured samples accumulated during online operation, without retraining the entire dataset offline. Specifically, the incremental manifold update process may include the following sub-steps: First, add the visual and tactile manifold points of the newly added samples to the visual manifold sample set and the tactile manifold sample set, respectively; then, update the weight matrices of the visual projection mapping and the tactile projection mapping using the incremental principal component analysis algorithm; finally, update the visual manifold reference point using the weighted moving average method on the manifold. and tactile manifold reference point Finally, the success state covariance tensor is updated using the incremental covariance estimation method. and stable contact covariance tensor .

[0150] For example, in the SMD component sorting scenario, the system triggers an incremental manifold update after every 100 successful grabs. The system uses the 100 newly added successful grab samples to locally adjust the prior parameter set of the visual grab success manifold and the prior parameter set of the tactile stable contact manifold and alignment manifold. This allows the manifold parameters to adapt to changes in component batches, gripper wear, and other working conditions, maintaining the long-term stability of the illusion verification.

[0151] In this embodiment, the incremental manifold update step enables continuous learning of new real successful capture samples during online operation, adaptively adjusting manifold parameters to cope with operating condition drift and scene changes, thereby maintaining the long-term stability of illusion verification accuracy.

[0152] like Figure 2 As shown, some embodiments of this application provide a robot grasping illusion verification system based on a tactile-visual manifold, specifically: The acquisition module 210 is used to acquire the raw online visual perception data at the moment when the robot gripper closes into position, as well as the online tactile perception timing data within a set time window after the gripper closes. The first module 220 is used to perform manifold projection processing on the original online visual perception data based on a pre-built set of prior parameters of the successful visual capture manifold, so as to obtain visual manifold points and visual state confidence. The second module 230 is used to perform manifold projection processing on online tactile perception time series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set to obtain tactile manifold points and tactile state confidence. The third module 240 is used to map tactile manifold points to visual manifold space to obtain tactile alignment points; The fourth module 250 is used to calculate the geometric alignment distance based on the visual manifold points and the tactile alignment points; The fifth module 260 is used to calculate the semantic consistency coefficient based on visual state confidence and tactile state confidence. Module 6, 270, is used to fuse geometric alignment distance and semantic consistency coefficient to generate bimodal manifold alignment score; Module 7, 280, is used to generate the robot grasping illusion verification result based on the bimodal manifold alignment score and the preset alignment threshold.

[0153] It should be noted that the robot grasping illusion verification system based on tactile-visual manifold provided in this embodiment is based on the same inventive concept as the robot grasping illusion verification method based on tactile-visual manifold described above. Therefore, the relevant content of the robot grasping illusion verification method based on tactile-visual manifold described above is also applicable to the content of the robot grasping illusion verification system based on tactile-visual manifold. Therefore, it will not be repeated here.

[0154] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described robot grasping illusion verification method based on tactile-visual manifold.

[0155] like Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the above-described method for verifying the illusion of robot grasping based on a tactile-visual manifold, as described in this application.

[0156] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0157] The electronic devices according to embodiments of this application will now be described in detail.

[0158] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform a robot grasping illusion verification method based on a tactile-visual manifold according to an embodiment of this application.

[0159] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0160] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described robot grasping illusion verification method based on tactile-visual manifold.

[0161] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0163] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0166] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any related variations, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

[0173] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A method for verifying illusions in robot grasping based on tactile-visual manifolds, characterized in that, The method includes: Acquire raw online visual perception data at the moment the robot gripper closes into position, as well as online tactile perception time-series data within a set time window after the gripper closes; Based on the pre-constructed prior parameter set of the successful visual capture manifold, the original online visual perception data is processed by manifold projection to obtain visual manifold points and visual state confidence. Based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, the online tactile perception time series data is processed by manifold projection to obtain tactile manifold points and tactile state confidence. The tactile manifold points are mapped to the visual manifold space to obtain tactile alignment points; Calculate the geometric alignment distance based on the visual manifold point and the tactile alignment point; Based on the confidence scores of the visual and tactile states, a semantic consistency coefficient is calculated. By fusing the geometric alignment distance and the semantic consistency coefficient, a bimodal manifold alignment score is generated. Based on the bimodal manifold alignment score and the preset alignment threshold, a robot grasping illusion verification result is generated.

2. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, The process involves manifold projection processing of the online visual perception raw data based on a pre-built set of prior parameters for successful visual grasping, to obtain visual manifold points and visual state confidence scores, including: The raw online visual perception data is converted into visual Euclidean feature vectors. Using the visual projection mapping in the prior parameter set of the visually captured manifold, the visual Euclidean feature vector is projected onto the visually captured manifold to obtain the visual manifold point; Based on the successful state geodesic Gaussian distribution in the prior parameter set of the visual capture success manifold, the success capture probability is calculated, and the success capture probability represents the probability that the visual manifold point belongs to the successful capture state. The probability of successful capture is normalized to obtain the confidence level of the visual state.

3. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, The process involves manifold projection processing of the online tactile perception time-series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set to obtain tactile manifold points and tactile state confidence scores, including: The online tactile perception time-series data is converted into a tactile Euclidean feature vector; By using the tactile stable contact manifold and the tactile projection mapping in the alignment prior parameter set, the tactile Euclidean eigenvectors are projected onto the tactile stable contact manifold to obtain the tactile manifold points; Based on the stable contact manifold and the stable contact geodesic Gaussian distribution in the alignment prior parameter set, the stable contact probability is calculated, which represents the probability that the tactile manifold point belongs to a stable contact state. The stable contact probability is normalized to obtain the confidence level of the tactile state.

4. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, Before mapping the tactile manifold points to the visual manifold space to obtain the tactile alignment points, the method further includes: Multiple sets of training samples collected simultaneously are acquired. Each set of samples in the training sample set includes visual manifold points, tactile manifold points, and sample labels. The sample labels are used to indicate the grasping status of the corresponding sample. The grasping status includes grasping success and grasping failure. Based on the sample labels indicating that the samples were successfully captured, feature pairs between the corresponding visual manifold points and the corresponding tactile manifold points are constructed to obtain a feature training set. With a preset metric-preserving condition as a constraint, a Riemann isometric alignment mapping is trained using the feature training set. The preset metric-preserving condition includes that the first local geometric distance between any two neighboring points on the tactile manifold corresponding to the feature training set is equal to the second local geometric distance between two image points obtained on the visual manifold corresponding to the feature training set after the Riemann isometric alignment mapping of any two neighboring points. The first local geometric distance is determined by the metric tensor of the tactile manifold corresponding to the feature training set, and the second local geometric distance is determined by the metric tensor of the visual manifold corresponding to the feature training set. The Riemann isometric alignment mapping is used to map the tactile manifold points to the visual manifold space.

5. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, The calculation of the geometric alignment distance based on the visual manifold point and the tactile alignment point includes: On the visual manifold corresponding to the prior parameter set of the visually captured manifold, determine the geodesic connecting the visual manifold point and the tactile alignment point; Integrate the metric tensor in the prior parameter set of the visually captured manifold along the geodesic line to obtain the geodesic integral result; Using the successful state covariance tensor in the successful manifold prior parameter set captured by vision, the geodesic tangent vector in the geodesic integral result is weighted to obtain the covariance-weighted Riemann geodesic distance; The covariance-weighted Riemann geodesic distance is used as the geometric alignment distance.

6. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, The step of calculating the semantic consistency coefficient based on the visual state confidence and the tactile state confidence includes: The visual state confidence score is transformed by a standard normal inverse cumulative distribution to obtain the visual transformation value. The tactile state confidence is transformed by a standard normal inverse cumulative distribution to obtain the tactile transformation value. Using a preset visual-touch dependency coefficient as a parameter, a Gaussian pairwise correlation function is used to calculate the joint probability density of the visual transformation value and the tactile transformation value to obtain the semantic consistency coefficient.

7. The robot grasping illusion verification method based on tactile-visual manifold according to claim 1, characterized in that, The process of fusing the geometric alignment distance and the semantic consistency coefficient to generate a bimodal manifold alignment score includes: The attenuation amount is calculated based on the geometric alignment distance and the preset alignment scale parameter; Based on the aforementioned attenuation amount, an exponential attenuation term is constructed; The bimodal manifold alignment score is generated based on the semantic consistency coefficient and the exponential decay term.

8. A robot grasping illusion verification system based on a tactile-visual manifold, characterized in that, The system includes: The acquisition module is used to acquire the raw online visual perception data at the moment when the robot gripper closes into position, as well as the online tactile perception time sequence data within a set time window after the gripper closes. The first module is used to perform manifold projection processing on the online visual perception raw data based on a pre-built set of prior parameters of the successful visual capture manifold, so as to obtain visual manifold points and visual state confidence. The second module is used to perform manifold projection processing on the online tactile perception time series data based on a pre-constructed tactile stable contact manifold and alignment prior parameter set, to obtain tactile manifold points and tactile state confidence. The third module is used to map the tactile manifold points to the visual manifold space to obtain tactile alignment points; The fourth module is used to calculate the geometric alignment distance based on the visual manifold point and the tactile alignment point; The fifth module is used to calculate the semantic consistency coefficient based on the confidence scores of the visual state and the tactile state. The sixth module is used to fuse the geometric alignment distance and the semantic consistency coefficient to generate a bimodal manifold alignment score; The seventh module is used to generate a robot grasping illusion verification result based on the bimodal manifold alignment score and a preset alignment threshold.

9. An electronic device, characterized in that, It includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a robot grasping illusion verification method based on a tactile-visual manifold as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a robot grasping illusion verification method based on a tactile-visual manifold as described in any one of claims 1 to 7.

Images