Digital twin interaction control system based on fusion of vision and touch of robot

Abstract

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

Description

Technical Field

[0001] The present invention relates to the field of robot perception and control, and in particular to a digital twin interactive control system based on the fusion of robot vision and touch. Background Art

[0002] In existing robot control technology, visual recognition and tactile feedback, as two key perception methods, have been widely used in complex operation tasks. Traditional methods mostly use a single modality to drive tasks. For example, image information is obtained through a camera to identify the position of an object, and then a preset path is used to complete grasping or interactive operations. However, the method that relies solely on vision is easily affected by factors such as occlusion and lighting changes, resulting in reduced recognition accuracy. Although tactile perception can supplement environmental contact information, its data dimension is complex and the response is local, making it difficult to support stable decision-making alone. In particular, it exhibits problems of response lag and insufficient control accuracy in unstructured environments or fine-grained operations.

[0003] To address the above issues, some studies have begun to attempt to fuse visual and tactile data, but most of them are limited to simple splicing of data levels or static fusion of decision layers, failing to achieve deep collaboration between the two types of perception data in the time domain and semantic layer. In addition, existing interactive control methods generally lack a digital twin mechanism that maps the real robot state in real time, making it impossible to intuitively perceive the dynamic correspondence between simulated feedback and actual response during user interactive control, making it difficult to form a stable and effective control closed loop.

[0004] Therefore, there is currently a lack of an interactive control system that integrates visual and tactile multimodal perception capabilities and combines digital twin mapping, motion simulation prediction, control intention analysis and dynamic feedback optimization. There is an urgent need to improve the operating accuracy, response sensitivity and control robustness of robot systems in complex tasks. Summary of the Invention

[0005] One purpose of the present invention is to propose a digital twin interactive control system based on the fusion of robot vision and touch. The present invention integrates robot vision and tactile information to construct a digital twin synchronous control mechanism to achieve an interactive optimization method with fast operation response, high execution accuracy and stable control process. Through multimodal perception modeling, fusion state vector generation and dynamic feedback mechanism, the robot's perception ability and action decision-making ability in complex operating environments are improved, effectively solving the problems of response lag, large precision error and lack of tactile feedback in traditional control methods, and has good scalability and system real-time performance.

[0006] A digital twin interactive control system based on robot vision and tactile fusion according to an embodiment of the present invention includes:

[0007] A data acquisition module, used to collect visual image data and tactile sensing data;

[0008] A preprocessing module, used to preprocess the collected visual image data and tactile sensing data respectively;

[0009] Multimodal perception modeling module, used to perform spatial perception modeling and tactile state modeling, and extract multidimensional features of target objects and contact states;

[0010] A fusion state vector building module, used to generate a fusion state vector;

[0011] The digital twin mapping module is used to build a digital twin that is consistent with the robot's current state in real time;

[0012] The simulation interactive control module is used to perform interactive motion control in the digital twin and generate control instruction parameter groups through user input intention, simulation feedback and motion optimization;

[0013] The action execution and feedback module is used to control the robot to perform actual actions based on the generated control instruction parameter group and continuously update the fusion state vector to form a closed-loop control process;

[0014] The dynamic evaluation and adaptive adjustment module is used to evaluate the control execution effect in real time and dynamically optimize and adjust the fusion state vector construction strategy based on the posture rate, tactile response and accuracy error.

[0015] Optionally, modules can be connected using the following methods:

[0016] S1. Collect visual image data and tactile sensor data of the robot body, wherein the visual image data is acquired by an image acquisition device deployed on the robot body, and the tactile sensor data is acquired by force and pressure sensors;

[0017] S2, formatting and preprocessing the visual image data and the tactile sensing data respectively;

[0018] S3. Perform spatial perception modeling on the formatted and pre-processed visual image data to extract the position, posture, and edge contour features of the target object; perform tactile state modeling on the formatted and pre-processed tactile sensing data to extract the contact point pressure distribution, friction trend, and contact stability index;

[0019] S4. Based on the output results of spatial perception modeling and tactile state modeling, a fusion state vector is constructed using channel alignment and timing synchronization mechanisms;

[0020] S5. Input the fused state vector into the digital twin mapping module to drive the generation of a digital twin that is highly consistent with the current state of the robot and maintains real-time synchronization with the robot's motion state;

[0021] S6, performing a simulation interaction operation in the digital twin, inputting a control intention through a user control interface, predicting an action and optimizing parameters based on digital simulation feedback, and generating a control instruction parameter set;

[0022] S7, controlling the robot to perform an actual action based on the control instruction parameter set, continuously collecting visual image data and tactile sensing data to update a fusion state vector, and realizing a closed-loop interactive control process of state synchronization, action response, and control feedback;

[0023] S8, in the closed-loop interactive control process, real-time evaluation of the execution effect of the control instruction parameter set, dynamic adjustment of the construction method of the fusion state vector according to the posture change rate, tactile response sensitivity, and action accuracy error.

[0024] Optionally, the format preprocessing of the visual image data includes image denoising, color normalization, and spatial resolution uniformity processing, and the format preprocessing of the tactile sensing data includes filtering, amplitude normalization, and timestamp alignment operation.

[0025] Optionally, the S3 includes the following specific steps:

[0026] S31, performing edge extraction operation on the color channel in the format preprocessed visual image data, analyzing the brightness change of each pixel point in the horizontal direction and the vertical direction in the visual image data, identifying the edge pixels with brightness change amplitude exceeding the set threshold, and constructing the edge set composed of these edge pixels as the edge contour feature of the target object;

[0027] S32, performing three-dimensional coordinate solution processing based on the depth channel in the format preprocessed visual image data, according to the depth value corresponding to each pixel in the visual image data, combining the known camera internal parameter, projecting each pixel point on the two-dimensional image plane to the robot workspace, obtaining the three-dimensional space point corresponding to the pixel, and constructing the point cloud model of the target object with all three-dimensional coordinate points;

[0028] S33, performing position extraction processing on the constructed point cloud model, obtaining the three-dimensional position vector of the target object in the robot workspace coordinate system by averaging all three-dimensional coordinate points in the point cloud model in each coordinate axis direction;

[0029] S34, performing pose extraction processing on the constructed point cloud model, using principal component analysis method to statistically analyze the spatial distribution in the point cloud, determining the principal axis distribution trend of the point set in the point cloud in each direction, and then converting the principal axis direction into a pose quaternion for representing the pose information of the target object in the three-dimensional space, the pose information being used to describe the rotation state of the target object relative to a fixed reference coordinate system, the pose quaternion including a scalar part and a vector part, the scalar part representing a cosine term of a rotation angle, and the vector part including a rotation component around an x-axis direction, a rotation component around a y-axis direction, and a rotation component around a z-axis direction;

[0030] S35, performing combination processing on the force sensation data in the formatted preprocessed tactile sensing data, arranging the contact forces in three directions and the contact moments in three directions into a group of six-dimensional vectors in channel order, the six-dimensional vectors being used to represent the actual contact state of the robot end and the target object at the current time, as the force basis information of the contact points;

[0031] S36, performing expansion processing on the pressure sensation data in the formatted preprocessed tactile sensing data, arranging the unit pressure values of each subunit in the pressure sensation data into a one-dimensional vector in row priority order, the unit area pressure values of the corresponding contact point positions at the current time being represented in the one-dimensional vector, and being used to constitute the contact point pressure distribution;

[0032] S37, based on the spatial difference relationship in the contact point pressure distribution, analyzing the pressure change amplitude between adjacent subunits, obtaining the maximum pressure gradient at the current time, and then calculating the overall force intensity at the same time according to the six-dimensional contact force data, and using the ratio of the maximum pressure gradient to the overall force intensity as a contact stability index, which is used to measure the change degree of the local fluctuation relative to the overall force state in the current contact process;

[0033] S38, according to the change of the contact point pressure distribution at different time points, calculating the pressure change rate of each subunit in the adjacent time interval, and combining the tangent direction of each subunit on the actual contact surface, projecting the pressure change rate along the tangent direction and taking the average to obtain the friction trend parameter at the current time, the friction trend parameter being used to represent whether there is a sliding trend on the surface of the contact point.

[0034] Optionally, the S4 includes the following specific steps:

[0035] S41, obtaining the three-dimensional position vector, the pose quaternion and the edge contour feature data output by the space perception modeling;

[0036] S42, combining the three-dimensional position vector and the pose quaternion to form a space pose subvector, and performing normalization processing on the dimensional data of the space pose subvector to adapt to a unified data scale;

[0037] S43, compress the edge contour feature data into a fixed-dimension contour encoding vector, and splice it with a spatial pose sub-vector to construct a visual state vector;

[0038] S44, obtain the contact point pressure distribution, friction trend parameters and contact stability indicators output by the tactile state modeling;

[0039] S45, extract the principal components of the contact point pressure distribution, select the pressure components with the largest contribution, combine the friction trend parameters and contact stability indicators to form a tactile state vector, and perform normalization processing to make it consistent with the visual state vector in numerical scale and structural dimension;

[0040] S46, perform channel alignment processing on the visual state vector and the tactile state vector, and splice to construct a fusion state original vector;

[0041] S47, construct a sliding time window, and perform weighted average calculation on the fusion state original vectors generated at the current and previous time, remove high-frequency disturbance noise, and output the smoothed fusion state vector;

[0042] S48, take the fusion state vector as the only input state vector for digital twin synchronous driving at the current time, and store it in the local state cache pool.

[0043] Optionally, the S5 includes the following specific steps:

[0044] S51, input the fusion state vector into the digital twin mapping module, the digital twin mapping module includes a spatial synchronization unit, a pose mapping unit and a tactile simulation unit, for driving to generate a digital twin body highly consistent with the current state of the robot;

[0045] The spatial synchronization unit is used to drive the position and structure contour of the digital twin body to keep real-time consistent with the robot body;

[0046] The pose mapping unit is used to map the attitude quaternion of the robot body to the digital twin body, ensuring spatial orientation synchronization;

[0047] The tactile simulation unit is used to simulate the tactile feedback state, so that the digital twin body presents a simulation effect consistent with the actual contact response of the robot;

[0048] S52, analyze the three-dimensional position vector in the fusion state vector by the spatial synchronization unit, and real-time position the geometric center point of the digital twin body, so that the digital twin body keeps the same position state as the robot body in the virtual space;

[0049] S53. The posture mapping unit parses the posture quaternion in the fusion state vector, aligns the spatial orientation and main axis direction of the digital twin with the actual posture of the robot body, converts the posture quaternion into a direction parameter, and uses it to update the spatial orientation state of the digital twin;

[0050] S54, the spatial synchronization unit further analyzes the edge contour features in the fusion state vector and updates the boundary structure of the digital twin, so that the virtual object presents contour changes consistent with the real object in the graphic display;

[0051] S55, the tactile simulation unit analyzes the contact point pressure distribution, friction trend parameters, and contact stability index in the fusion state vector, and adjusts the contact feedback performance of the digital twin in real time;

[0052] S56. Fusing the output results of each unit based on a unified time axis to achieve position synchronization, posture synchronization, and tactile synchronization to form an integrated digital twin, and performing periodic refresh within a preset time interval;

[0053] S57. Cache the integrated digital twin in the digital twin rendering engine and synchronously broadcast the status data to achieve continuous linkage and state consistency between the robot body and the digital twin.

[0054] Optionally, S6 includes the following specific steps:

[0055] S61. Setting a control intention input area in the user control interface of the digital twin to receive the target action type, target position vector, target attitude quaternion, and tactile response setting parameters input by the operator through graphical interaction, and parsing the control intention to convert it into a structured control request instruction;

[0056] S62. Read the fusion state vector of the current digital twin;

[0057] S63. Based on the current state of the digital twin described by the fused state vector, call the digital simulation engine to perform physical prediction simulation on the structured control request instruction, and output simulation feedback data including target state change, motion path estimation, and tactile response prediction results;

[0058] S64, performing a difference analysis between the simulation feedback data and the target parameters input by the operator, and adjusting the parameters of the intended action by the action optimization unit. The action optimization unit constructs a multi-objective cost function including accuracy error, contact stability, and execution efficiency based on the fused state vector and the target parameters input by the operator, and jointly optimizes the target position vector, the target attitude quaternion, and the tactile response setting parameters to generate a control instruction parameter group that meets the error tolerance and action feasibility;

[0059] The control instruction parameter group includes a desired three-dimensional position vector, a desired posture quaternion and a tactile response setting parameter;

[0060] S65. The simulated predicted action process and expected response result are displayed in real time in the user control interface, and the operator is allowed to fine-tune the control intention. The adjusted control intention enters steps S61-S65 again until the user confirms.

[0061] Optionally, S8 includes the following specific steps:

[0062] S81, while executing the control instruction parameter group, continuously collecting visual image data and tactile sensor data;

[0063] S82. Construct a fusion state vector at the current moment using the feature results output by the spatial perception modeling and the tactile state modeling. Compare the vector with the fusion state vector constructed at the previous moment, calculate the amplitude of the three-dimensional position change, the angular velocity of the posture change, and the amplitude of the tactile response change, and obtain three execution evaluation indicators: the posture change rate, the tactile response sensitivity, and the motion accuracy error.

[0064] S83. Perform dynamic stability assessment on the attitude change rate to determine whether there is a risk of directional control drift at the robot end effector;

[0065] Evaluate the tactile response timeliness to determine whether there is a hysteresis response or sudden imbalance in the contact behavior;

[0066] Evaluate the error threshold of the motion accuracy error to determine whether the execution result meets the tolerance requirements of position and posture control;

[0067] S84. Triggering an adaptive adjustment operation of the fusion state vector construction mechanism based on the combined judgment result of the three execution evaluation indicators, wherein the adaptive adjustment operation adjusts the synchronization buffer length in the channel alignment mechanism;

[0068] S85. Use the adjusted construction mechanism to generate the fusion state vector at the next moment and input it into the digital twin mapping module to ensure that the digital twin is synchronized with the robot body in terms of three-dimensional position, posture orientation, and tactile simulation feedback;

[0069] S86. During each round of fusion state vector construction and synchronization, record the fusion state vector change data, control instruction execution effect and adjustment behavior to form a closed-loop control log;

[0070] S87. If an abnormal posture change rate, deviation in tactile response sensitivity, or movement accuracy error exceeding a preset threshold is detected in multiple consecutive control cycles, a local alarm is triggered to suspend the current control process, and the current control instruction parameter group and fusion state vector are cached in the state recovery buffer, waiting for user intervention or re-execution after the recovery condition is triggered.

[0071] The beneficial effects of the present invention are:

[0072] The present invention effectively solves the problems of single perception mode instability, incomplete control feedback loop, and asynchrony between simulation and real state in the existing technology by constructing a digital twin interactive control system based on the fusion of robot vision and touch. The system realizes deep coordination of visual and tactile information at the timing and channel levels by setting a formatting preprocessing module for visual images and tactile sensing data, a multimodal perception modeling module, a fusion state vector construction module, a digital twin mapping module, a simulation interactive control module and an action execution feedback module, thereby improving the integrity and stability of state perception.

[0073] In addition, the present invention introduces a digital twin synchronization mechanism and an action prediction feedback mechanism. After the user inputs the control intention, simulation and cost optimization are performed before the physical execution stage, and the control parameter group is adjusted in advance, which significantly reduces the execution error and action uncertainty. The dynamic construction mechanism of the fusion state vector enables the system to have real-time adjustment capabilities during the task execution process. It can actively optimize the information fusion strategy according to indicators such as the posture change rate, tactile response sensitivity and action accuracy error during the execution process, and maintain the dynamic stability of the perception and control link. Finally, by constructing a closed-loop interactive control process, the system realizes real-time feedback, adaptive adjustment and high-precision execution of robot operations, and improves the operational reliability, response sensitivity and control accuracy under complex interactive tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0075] Figure 1 This is a schematic diagram of the overall structure of a digital twin interactive control system based on the fusion of robot vision and touch proposed in the present invention;

[0076] Figure 2 This is a schematic diagram of the construction and state synchronization structure of the digital twin mapping module of a digital twin interactive control system based on the fusion of robot vision and touch proposed in the present invention. DETAILED DESCRIPTION

[0077] The application will be described in further detail below with reference to the drawings. These drawings are simplified schematic diagrams and only show the basic structure of the application in a schematic manner, and thus only show the components relevant to the application.

[0078] Reference Figure 1-2 A digital twin interaction control system based on fusion of robot vision and haptics, comprising:

[0079] A data acquisition module for acquiring visual image data and haptic sensing data;

[0080] A preprocessing module for preprocessing the acquired visual image data and haptic sensing data respectively;

[0081] A multi-modal perception modeling module for performing spatial perception modeling and haptic state modeling to extract multi-dimensional features of target objects and contact states;

[0082] A fusion state vector construction module for generating a fusion state vector;

[0083] A digital twin mapping module for constructing a digital twin body consistent with the current state of the robot in real time;

[0084] A simulation interaction control module for performing interactive action control in the digital twin body, generating control instruction parameter sets through user input intent, simulation feedback and action optimization;

[0085] An action execution and feedback module for controlling the robot to perform actual actions based on the generated control instruction parameter sets, and continuously updating the fusion state vector to form a closed-loop control process;

[0086] A dynamic evaluation and adaptive adjustment module for real-time evaluation of control execution effect, and dynamic optimization and adjustment of the fusion state vector construction strategy according to the pose rate, haptic response and precision error.

[0087] The application realizes efficient fusion and interactive control of visual and haptic information by constructing a system structure including a data acquisition module, a preprocessing module, a multi-modal perception modeling module, a fusion state vector construction module, a digital twin mapping module, a simulation interaction control module, an action execution and feedback module, and a dynamic evaluation and adaptive adjustment module, and has complete state perception, action prediction and execution feedback loop capabilities, and improves the intelligent response level and operation robustness of the robot in complex task scenarios.

[0088] In this embodiment, the modules are realized by the following methods:

[0089] S1, collect visual image data and tactile sensing data of the robot body, the visual image data is obtained by an image collection device arranged on the robot body, and the tactile sensing data is collected by a force and pressure sensor;

[0090] S2, respectively format and pretreat the visual image data and the tactile sensing data;

[0091] S3, spatial perception modeling is performed on the formatted and pretreated visual image data, position, posture and edge contour features of the target object are extracted, and tactile state modeling is performed on the formatted and pretreated tactile sensing data, contact point pressure distribution, friction trend and contact stability indicators are extracted;

[0092] S4, based on the output results of the spatial perception modeling and the tactile state modeling, a fusion state vector is constructed by using a channel alignment and time sequence synchronization mechanism;

[0093] S5, the fusion state vector is input into a digital twin mapping module, a digital twin body highly consistent with the current state of the robot is generated, and real-time synchronization with the action state of the robot is maintained;

[0094] S6, simulation interaction operation is performed in the digital twin body, control intention is input through a user control interface, action prediction and parameter optimization are performed based on digital simulation feedback, and a control instruction parameter group is generated;

[0095] S7, based on the control instruction parameter group, the robot performs actual action, and the visual image data and the tactile sensing data are continuously collected to update the fusion state vector, realizing a closed-loop interactive control process of state synchronization, action response and control feedback;

[0096] S8, in the closed-loop interactive control process, the execution effect of the control instruction parameter group is evaluated in real time, and the construction mode of the fusion state vector is dynamically adjusted according to the posture change rate, the tactile response sensitivity and the action accuracy error.

[0097] The present application realizes information driving and dynamic adjustment of the whole process from perception to control by establishing strict step sequence and logical closed loop among system modules, including data collection, modeling, fusion, mapping, simulation control and adaptive feedback, solves the problems of existing system feedback lag, response disconnection and the like, improves the real-time performance and response reliability of the system, and guarantees the stability of the robot in a dynamic environment and the continuity of task execution.

[0098] In the embodiment, the formatted pretreatment of the visual image data includes image denoising, color normalization and spatial resolution uniformization processing, and the formatted pretreatment of the tactile sensing data includes filtering, amplitude normalization and timestamp alignment operation.

[0099] In this embodiment, S3 includes the following specific steps:

[0100] S31, performing an edge extraction operation on the color channel in the formatted preprocessed visual image data, analyzing the brightness changes of each pixel in the visual image data in the horizontal and vertical directions, identifying edge pixels whose brightness changes exceed a set threshold, and using the edge set composed of these edge pixels as the edge contour feature of the target object;

[0101] S32. Perform three-dimensional coordinate calculation based on the depth channel in the formatted preprocessed visual image data. Based on the depth value corresponding to each pixel in the visual image data and in combination with known camera intrinsic parameters, back-project each pixel on the two-dimensional image plane into the robot workspace to obtain the three-dimensional space point corresponding to the pixel, and construct a point cloud model of the target object from all three-dimensional space points.

[0102] S33, performing position extraction processing on the constructed point cloud model, and obtaining the three-dimensional position vector of the target object in the robot workspace coordinate system by averaging all three-dimensional coordinate points in the point cloud model in the directions of each coordinate axis;

[0103] S34. Performing posture extraction processing on the constructed point cloud model, using a principal component analysis method to perform statistics on the spatial distribution in the point cloud, determining the principal axis distribution trend of the point set in each direction in the point cloud, and then converting the principal axis direction into a posture quaternion for representing the posture information of the target object in three-dimensional space. The posture information is used to describe the rotation state of the target object relative to a fixed reference coordinate system. The posture quaternion includes a scalar part and a vector part. The scalar part represents the cosine term of the rotation angle, and the vector part includes a rotation component around the x-axis, a rotation component around the y-axis, and a rotation component around the z-axis.

[0104] S35. Combining the force data in the formatted and pre-processed tactile sensing data, arranging the contact forces and contact torques in the three directions in the order of the channels into a set of six-dimensional vectors. The six-dimensional vectors are used to represent the actual contact state between the robot end and the target object at the current moment, serving as basic force information at the contact point.

[0105] S36: Expand the pressure data in the formatted and pre-processed tactile sensing data, and arrange the unit pressure value of each sub-unit in the pressure data into a one-dimensional vector in row-priority order. The one-dimensional vector represents the unit area pressure value corresponding to the contact point position at the current moment, so as to form a contact point pressure distribution.

[0106] S37, based on the spatial difference relationship in the contact point pressure distribution, analyze the pressure change amplitude between adjacent sub-units, obtain the maximum pressure gradient at the current moment, and calculate the overall stress intensity according to the six-dimensional contact force data at the same moment, adopt the ratio of the maximum pressure gradient and the overall stress intensity as the contact stability index, which is used to measure the change degree of local fluctuation relative to the overall stress state in the current contact process;

[0107] S38, according to the change of the contact point pressure distribution at different time points, calculate the pressure change rate of each sub-unit in the adjacent time interval, and combine the tangential direction of each sub-unit on the actual contact surface, project the pressure change rate along the tangential direction and take the average to obtain the friction trend parameter at the current moment, the friction trend parameter is used to represent whether there is a sliding trend on the surface of the contact point.

[0108] In the multi-modal perception modeling process, the position, posture and surface contact state of the target object are accurately described by combining point cloud modeling, posture extraction and pressure development, a complete space and tactile feature representation system is constructed, and the environmental understanding and fine interaction ability of the robot are improved.

[0109] In the embodiment, the S4 includes the following specific steps:

[0110] S41, obtain the three-dimensional position vector, attitude quaternion and edge contour feature data output by the spatial perception modeling;

[0111] S42, combine the three-dimensional position vector and the attitude quaternion to form a spatial attitude sub-vector, and normalize each dimension data of the spatial attitude sub-vector to adapt to the unified data scale;

[0112] S43, compress the edge contour feature data into a fixed-dimensional contour coding vector, and splice the spatial attitude sub-vector to construct a visual state vector;

[0113] S44, obtain the contact point pressure distribution, friction trend parameter and contact stability index output by the tactile state modeling;

[0114] S45, perform principal component extraction on the contact point pressure distribution, select a plurality of pressure components with the largest contribution degree, combine the friction trend parameter and the contact stability index to form a tactile state vector, and perform normalization processing to make it consistent with the visual state vector in numerical scale and structural dimension;

[0115] S46, perform channel alignment processing on the visual state vector and the tactile state vector, and splice to construct a fusion state original vector;

[0116] S47, construct a sliding time window, and perform weighted average calculation on the fusion state original vector generated at the current and previous time, remove high-frequency disturbance noise, and output the fusion state vector after smoothing processing;

[0117] S48, the fusion state vector is used as the only input state vector for digital twin synchronous driving at the current time, and is stored in a local state cache pool.

[0118] The application proposes a channel alignment and timing sliding fusion mechanism, constructs a stable and dynamically updated fusion state vector, fully combines visual and tactile information, reduces the fluctuation interference of perception data, effectively improves the expression consistency of fusion information, provides high-quality input basis for digital twin mapping, and enhances the synchronization accuracy and state simulation reality of the twin.

[0119] In the embodiment, the S5 includes the following specific steps:

[0120] S51, input the fusion state vector to a digital twin mapping module, the digital twin mapping module includes a space synchronization unit, a posture mapping unit and a tactile simulation unit, and is used to drive generation of a digital twin body highly consistent with the current state of the robot;

[0121] The space synchronization unit is used to drive the position and structure contour of the digital twin body to keep real-time consistent with the robot body;

[0122] The posture mapping unit is used to map the attitude quaternion of the robot body to the digital twin body, and ensure the space orientation synchronization;

[0123] The tactile simulation unit is used to simulate the tactile feedback state, so that the digital twin body presents a simulation effect consistent with the actual contact response of the robot;

[0124] S52, the three-dimensional position vector in the fusion state vector is analyzed by the space synchronization unit, the geometric center point of the digital twin body is positioned in real time, and the digital twin body keeps the same position state as the robot body in the virtual space;

[0125] S53, the attitude quaternion in the fusion state vector is analyzed by the posture mapping unit, the space orientation and main shaft direction of the digital twin body are kept consistent with the actual attitude of the robot body, the attitude quaternion is converted into direction parameters and used to update the space orientation state of the digital twin body;

[0126] S54, the edge contour feature in the fusion state vector is further analyzed by the space synchronization unit, the boundary structure of the digital twin body is updated, and the virtual object presents consistent contour change with the real object in the graphic display;

[0127] S55, the contact point pressure distribution, the friction tendency parameter and the contact stability index in the fusion state vector are analyzed by the haptic simulation unit to adjust the contact feedback performance of the digital twin in real time;

[0128] S56, the output results of each unit are fused based on the unified time axis, so that the position synchronization, the attitude synchronization and the haptic synchronization constitute an integrated digital twin, and periodic refreshing is performed within a preset time interval;

[0129] S57, the integrated digital twin is cached in the digital twin rendering engine, the state data is broadcasted synchronously, and continuous linkage and state consistency between the robot body and the digital twin are maintained.

[0130] The present application realizes accurate replication of the physical robot state in the virtual environment by constructing a digital twin mapping module including space synchronization, attitude mapping and haptic simulation, and completes linkage of each module through a unified time axis and a state queue to ensure real-time linkage consistency between the virtual and real, and provides clear and visual operation feedback for interactive control.

[0131] In the embodiment, the S6 includes the following specific steps:

[0132] S61, a control intention input area is set in the user control interface of the digital twin, target action types, target position vectors, target attitude quaternions and haptic response setting parameters input by the operator through graphical interaction are received, the control intention is analyzed and converted into a structured control request instruction;

[0133] S62, the fusion state vector of the current digital twin is read;

[0134] S63, according to the current state of the digital twin described by the fusion state vector, the digital simulation engine is called to perform physical prediction simulation on the structured control request instruction, and simulation feedback data including target state change, action path estimation and haptic response prediction results are output;

[0135] S64, the simulation feedback data and the target parameters input by the operator are differentially analyzed, and the action optimization unit adjusts the parameters of the action to be executed, the action optimization unit constructs a multi-objective cost function including accuracy error, contact stability and execution efficiency based on the fusion state vector and the target parameters input by the operator, and jointly optimizes the target position vector, the target attitude quaternion and the haptic response setting parameter to generate a control instruction parameter group meeting the error tolerance and action feasibility;

[0136] The control instruction parameter group includes an expected three-dimensional position vector, an expected attitude quaternion and a haptic response setting parameter;

[0137] S65, real-time display the simulation predicted action process and expected response result in the user control interface, and allow the operator to control intention fine-tuning, the adjusted control intention enters step S61-S65 again until the user confirms.

[0138] The application introduces a user control intention analysis and digital simulation feedback linkage mechanism in the simulation interaction process, uses a multi-objective optimization function to finely adjust the control instruction, realizes the closed-loop verification of control intention, simulation feedback and actual feasibility, and enhances the flexibility of human-computer interaction and the execution precision of control strategy.

[0139] In the embodiment, the S8 comprises the following specific steps:

[0140] S81, while executing the control instruction parameter group, continuously collecting visual image data and tactile sensing data;

[0141] S82, the feature results output by the space perception modeling and the tactile state modeling are combined to construct the fusion state vector at the current moment, and compared with the fusion state vector constructed at the last moment, the three-dimensional position change amplitude, the attitude change angular velocity and the tactile response change amplitude are calculated, and three execution evaluation indexes of attitude change rate, tactile response sensitivity and action accuracy error are obtained;

[0142] S83, the dynamic stability of the attitude change rate is evaluated, and whether the robot end effector exists the direction control drift risk is judged;

[0143] The response timeliness of the tactile response sensitivity is evaluated, and whether the contact behavior exists the hysteresis response or the mutation imbalance phenomenon is judged;

[0144] The error threshold of the action accuracy error is evaluated, and whether the execution result meets the tolerance requirement of position and attitude control is judged;

[0145] S84, according to the joint judgment result of the above three execution evaluation indexes, the adaptive adjustment operation of the fusion state vector construction mechanism is triggered, and the adaptive adjustment operation adjusts the length of the synchronization buffer in the channel alignment mechanism;

[0146] S85, the adjusted construction mechanism is used for the fusion state vector generation at the next moment, and input to the digital twin mapping module, so as to ensure that the digital twin keeps synchronization with the robot body in three-dimensional position, attitude orientation and tactile simulation feedback;

[0147] S86, in each round of fusion state vector construction and synchronization process, the fusion state vector change data, the control instruction execution effect and the adjustment behavior are recorded to form a closed-loop control log;

[0148] S87. If an abnormal posture change rate, deviation in tactile response sensitivity, or movement accuracy error exceeding a preset threshold is detected in multiple consecutive control cycles, a local alarm is triggered to suspend the current control process, and the current control instruction parameter group and fusion state vector are cached in the state recovery buffer, waiting for user intervention or re-execution after the recovery condition is triggered.

[0149] The present invention uses a continuous perception and execution evaluation mechanism to obtain the posture change rate, tactile response sensitivity and motion accuracy error in real time, combines it with a dynamic adaptive adjustment strategy to optimize the fusion state vector construction method, and triggers alarms and state recovery under abnormal conditions, effectively improving the system's robustness, fault tolerance and safety level.

[0150] Example 1:

[0151] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the robot collaborative assembly task in a certain intelligent manufacturing workshop. The workshop uses a multi-degree-of-freedom robotic arm to perform the plug-in and assembly operations of precision components. In actual application, it often faces problems such as uncertain position of target components, insufficient tactile feedback during plug-in, and accumulation of operational errors, which can easily lead to plug-in failure, equipment wear and even operation interruption, seriously restricting operation efficiency and system stability.

[0152] In the above-mentioned actual environment, the digital twin interactive control system based on the fusion of robot vision and touch proposed in the present invention is deployed in the assembly work unit. First, the image information of the components to be plugged in is obtained through the visual acquisition device installed at the end of the robot. At the same time, high-precision force and pressure sensors are installed at the end of the robotic arm to perceive the changes in contact status during the assembly process in real time. After image acquisition, this system performs image denoising, color normalization and resolution unification to ensure the stability of spatial perception; the tactile sensing data is also filtered, amplitude normalized and time aligned to eliminate signal anomalies caused by environmental interference during the acquisition process.

[0153] In the fusion perception modeling stage, on the one hand, the system constructs a three-dimensional point cloud model of the target object and extracts its position vector, posture quaternion and edge contour features; on the other hand, the tactile sensing data is constructed into a six-dimensional contact force vector and a one-dimensional contact pressure distribution vector, and the contact stability index and friction trend parameters are further calculated. Subsequently, the system constructs a smooth fusion state vector through channel alignment and sliding time window mechanism, drives the digital twin to generate and synchronize the robot operation status in real time. In the simulation interaction interface, the operator inputs the target action intention and combines the simulation feedback results to automatically complete parameter optimization and generate a control instruction group containing the expected position, posture and tactile response. After the control instruction group is issued by the system, it is executed by the robot and the control effect is fed back in real time, forming a closed loop control update link.

[0154] By continuously observing the attitude change rate, tactile response sensitivity, and motion error change trends in the fusion state vector, the system can stably identify attitude drift risks, tactile hysteresis, and other problems in multiple control cycles, automatically optimize the fusion strategy, and dynamically adjust the channel alignment buffer to achieve high-precision and stable control of the operation process.

[0155] In order to verify the performance of the present invention in practice, a comparison was made with the traditional method.

[0156] Table 1 Summary of key performance comparison between the present invention and traditional methods

[0157] Performance indicators Method of the present invention Traditional methods Improvement rate Fusion status update delay (ms) 45.0 120.0 62.5% reduction Digital twin synchronization accuracy (%) 95.7 85.2 12.3% increase Haptic feedback response delay (ms) 32.0 75.0 57.3% reduction Attitude tracking error (°) 0.8 2.4 66.7% reduction 3D position recognition error (cm) 0.5 2.1 76.2% reduction Action execution success rate (%) 96.5 81.3 18.7% increase User operation response time (s) 0.9 1.8 50% shorter System stable operation time (h) 1200.0 760.0 57.9% longer

[0158] As can be seen from Table 1, the present invention has achieved significant performance improvements in many key technical indicators. From the perspective of the fusion state update delay indicator, the delay of the method of the present invention is 45ms, while the delay of the traditional method reaches 120ms, which is a 62.5% reduction in delay time. This shows that the multimodal data processing mechanism and timing synchronization strategy of the present invention improve the real-time performance of data fusion and make state updates faster and more efficient.

[0159] In terms of digital twin synchronization accuracy, the method of the present invention achieved 95.7%, an increase of 12.3% over the 85.2% of the traditional method, fully demonstrating the effectiveness of the fusion state vector construction and digital twin mapping module proposed in the present invention. This high-precision state synchronization is mainly due to the deep fusion of visual and tactile information and the efficient spatial and tactile modeling methods, which ensures the consistency and accuracy between virtual and real states.

[0160] In terms of tactile feedback response delay, the response delay of the present invention is only 32ms, which is 57.3% lower than the 75ms of the traditional method. This improvement is due to the use of efficient tactile data normalization, filtering and amplitude alignment technology in the present invention, which ensures that tactile information is quickly and accurately transmitted to the decision-making module, enabling the system to quickly perceive and respond to environmental changes.

[0161] The tracking error of the proposed method is only 0.8°, significantly lower than the 2.4° of the traditional method, representing a 66.7% reduction. Furthermore, the 3D position recognition error of the proposed method is only 0.5cm, while the traditional method achieves 2.1cm, a 76.2% reduction. These significant improvements in accuracy are primarily due to the advanced edge extraction, 3D point cloud positioning, and posture extraction methods used in the proposed spatial perception modeling, which significantly improve the accuracy of position and posture recognition.

[0162] The proposed method achieved a 96.5% success rate for action execution, an 18.7% improvement over the 81.3% achieved by conventional methods. This is due to the introduction of simulation prediction and control instruction parameter optimization mechanisms, which ensure the accuracy and effectiveness of robot actions and reduce the probability of execution failure caused by parameter mismatches in conventional methods.

[0163] In addition, in terms of user operation response time indicators, the response time of the system of the present invention is 0.9s, while that of the traditional method reaches 1.8s, and the user response time is shortened by 50%. This reflects the close coordination of the present invention in the user interaction interface, digital simulation feedback and control strategy optimization, which can quickly respond to user intentions and enhance the interaction fluency between the system and the user.

[0164] The stable operation time of the system of the present invention is significantly extended. This is because the dynamic evaluation and adaptive adjustment module introduced in the present invention can monitor the system operation status in real time, actively adjust the fusion strategy, avoid errors and state deviations accumulated during long-term operation, and ensure that the system operates stably and reliably for a long time.

[0165] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A digital twin interactive control system based on the fusion of robot vision and touch, characterized in that: include: A data acquisition module, used to collect visual image data and tactile sensing data; A preprocessing module, used to preprocess the collected visual image data and tactile sensing data respectively; Multimodal perception modeling module, used to perform spatial perception modeling and tactile state modeling, and extract multidimensional features of target objects and contact states; A fusion state vector building module, used to generate a fusion state vector; The digital twin mapping module is used to build a digital twin that is consistent with the robot's current state in real time; The simulation interactive control module is used to perform interactive motion control in the digital twin and generate control instruction parameter groups through user input intention, simulation feedback and motion optimization; The action execution and feedback module is used to control the robot to perform actual actions based on the generated control instruction parameter group and continuously update the fusion state vector to form a closed-loop control process; The dynamic evaluation and adaptive adjustment module is used to evaluate the control execution effect in real time and dynamically optimize and adjust the fusion state vector construction strategy based on the posture rate, tactile response and accuracy error.

2. A digital twin interactive control system based on robot vision and tactile fusion according to claim 1, characterized in that: The modules are implemented as follows: S1. Collect visual image data and tactile sensor data of the robot body, wherein the visual image data is acquired by an image acquisition device deployed on the robot body, and the tactile sensor data is acquired by force and pressure sensors; S2, formatting and preprocessing the visual image data and the tactile sensing data respectively; S3. Perform spatial perception modeling on the formatted and pre-processed visual image data to extract the position, posture, and edge contour features of the target object; perform tactile state modeling on the formatted and pre-processed tactile sensing data to extract the contact point pressure distribution, friction trend, and contact stability index; S4. Based on the output results of spatial perception modeling and tactile state modeling, a fusion state vector is constructed using channel alignment and timing synchronization mechanisms; S5. Input the fused state vector into the digital twin mapping module to drive the generation of a digital twin that is highly consistent with the current state of the robot and maintains real-time synchronization with the robot's motion state; S6. Perform simulation interaction operations in the digital twin, input control intentions through the user control interface, perform action prediction and parameter optimization based on digital simulation feedback, and generate a control instruction parameter group; S7, based on the control instruction parameter group, controls the robot to perform actual actions, while continuously collecting visual image data and tactile sensor data to update the fusion state vector, thus realizing a closed-loop interactive control process of state synchronization, action response and control feedback; S8. In the closed-loop interactive control process, the execution effect of the control instruction parameter group is evaluated in real time, and the construction method of the fusion state vector is dynamically adjusted according to the posture change rate, tactile response sensitivity and movement accuracy error.

3. A digital twin interactive control system based on robot vision and tactile fusion according to claim 2, characterized in that: The formatting preprocessing of the visual image data includes image denoising, color normalization and spatial resolution unified processing, and the formatting preprocessing of the tactile sensing data includes filtering, amplitude normalization and timestamp alignment operations.

4. The digital twin interactive control system based on robot vision and tactile fusion according to claim 2 is characterized in that: The S3 includes the following specific steps: S31, performing an edge extraction operation on the color channel in the formatted preprocessed visual image data, analyzing the brightness changes of each pixel in the visual image data in the horizontal and vertical directions, identifying edge pixels whose brightness changes exceed a set threshold, and using the edge set composed of these edge pixels as the edge contour feature of the target object; S32. Perform three-dimensional coordinate calculation based on the depth channel in the formatted preprocessed visual image data. Based on the depth value corresponding to each pixel in the visual image data and in combination with known camera intrinsic parameters, back-project each pixel on the two-dimensional image plane into the robot workspace to obtain the three-dimensional space point corresponding to the pixel, and construct a point cloud model of the target object from all three-dimensional space points. S33, performing position extraction processing on the constructed point cloud model, and obtaining the three-dimensional position vector of the target object in the robot workspace coordinate system by averaging all three-dimensional coordinate points in the point cloud model in the directions of each coordinate axis; S34, performing posture extraction processing on the constructed point cloud model, using principal component analysis to perform statistics on the spatial distribution of the point cloud, determining the principal axis distribution trend of the point set in each direction in the point cloud, and then converting the principal axis direction into a posture quaternion to represent the posture information of the target object in three-dimensional space. The posture information is used to describe the rotation state of the target object relative to the fixed reference coordinate system; S35. Combining the force data in the formatted and pre-processed tactile sensing data, arranging the contact forces and contact torques in the three directions in the order of the channels into a set of six-dimensional vectors. The six-dimensional vectors are used to represent the actual contact state between the robot end and the target object at the current moment, serving as basic force information at the contact point. S36: Expand the pressure data in the formatted and pre-processed tactile sensing data, and arrange the unit pressure value of each sub-unit in the pressure data into a one-dimensional vector in row-priority order. The one-dimensional vector represents the unit area pressure value corresponding to the contact point position at the current moment, so as to form a contact point pressure distribution. S37. Based on the spatial difference relationship in the pressure distribution of the contact points, analyze the pressure variation between adjacent subunits to obtain the maximum pressure gradient at the current moment. Then, calculate the overall force intensity based on the six-dimensional contact force data at the same moment. Use the ratio of the maximum pressure gradient to the overall force intensity as the contact stability index to measure the degree of change of the local fluctuation relative to the overall force state during the current contact process. S38. Based on the changes in the pressure distribution of the contact point at different time points, the pressure change rate of each sub-unit in adjacent time intervals is calculated, and combined with the tangential direction of each sub-unit on the actual contact surface, the pressure change rate is projected along the tangential direction and averaged to obtain the friction trend parameter at the current moment. The friction trend parameter is used to indicate whether there is a slip trend on the contact point surface.

5. The digital twin interactive control system based on robot vision and tactile fusion according to claim 2 is characterized in that: The S4 includes the following specific steps: S41, obtaining the three-dimensional position vector, attitude quaternion and edge contour feature data output by the spatial perception modeling; S42, combining the three-dimensional position vector and the attitude quaternion to form a spatial attitude sub-vector, and normalizing the data of each dimension of the spatial attitude sub-vector to adapt to a unified data scale; S43, compressing the edge contour feature data into a contour encoding vector of fixed dimension, and concatenating it with the spatial posture sub-vector to construct a visual state vector; S44, obtaining contact point pressure distribution, friction trend parameters, and contact stability index output by tactile state modeling; S45. Extract the principal components of the contact point pressure distribution, select the pressure components with the largest contribution, combine the friction trend parameter and the contact stability index, form a tactile state vector, and perform normalization processing to make it consistent with the visual state vector in terms of numerical scale and structural dimension; S46, performing channel alignment processing on the visual state vector and the tactile state vector, and splicing them to construct a fusion state original vector; S47: Construct a sliding time window, perform weighted average calculation on the original fusion state vectors generated at the current and previous moments, remove high-frequency disturbance noise, and output a smoothed fusion state vector; S48. Use the fused state vector as the only input state vector for digital twin synchronous driving at the current moment and store it in the local state cache pool.

6. The digital twin interactive control system based on robot vision and tactile fusion according to claim 2 is characterized in that: The S5 includes the following specific steps: S51. Inputting the fused state vector into a digital twin mapping module, wherein the digital twin mapping module includes a spatial synchronization unit, a posture mapping unit, and a tactile simulation unit, and is used to drive the generation of a digital twin that is highly consistent with the current state of the robot; S52, the spatial synchronization unit analyzes the three-dimensional position vector in the fused state vector and locates the geometric center point of the digital twin in real time, so that the digital twin maintains the same position state as the robot body in the virtual space; S53. The posture mapping unit parses the posture quaternion in the fusion state vector, aligns the spatial orientation and main axis direction of the digital twin with the actual posture of the robot body, converts the posture quaternion into a direction parameter, and uses it to update the spatial orientation state of the digital twin; S54, the spatial synchronization unit further analyzes the edge contour features in the fusion state vector and updates the boundary structure of the digital twin, so that the virtual object presents contour changes consistent with the real object in the graphic display; S55, the tactile simulation unit analyzes the contact point pressure distribution, friction trend parameters, and contact stability index in the fusion state vector, and adjusts the contact feedback performance of the digital twin in real time; S56. Fusing the output results of each unit based on a unified time axis to achieve position synchronization, posture synchronization, and tactile synchronization to form an integrated digital twin, and performing periodic refresh within a preset time interval; S57. Cache the integrated digital twin in the digital twin rendering engine and synchronously broadcast the status data to achieve continuous linkage and state consistency between the robot body and the digital twin.

7. The digital twin interactive control system based on robot vision and tactile fusion according to claim 2 is characterized in that: The S6 comprises the following specific steps: S61. Setting a control intention input area in the user control interface of the digital twin to receive the target action type, target position vector, target attitude quaternion, and tactile response setting parameters input by the operator through graphical interaction, and parsing the control intention to convert it into a structured control request instruction; S62. Read the fusion state vector of the current digital twin; S63. Based on the current state of the digital twin described by the fused state vector, call the digital simulation engine to perform physical prediction simulation on the structured control request instruction, and output simulation feedback data including target state change, motion path estimation, and tactile response prediction results; S64, performing a difference analysis between the simulation feedback data and the target parameters input by the operator, and adjusting the parameters of the intended action by the action optimization unit. The action optimization unit constructs a multi-objective cost function including accuracy error, contact stability, and execution efficiency based on the fused state vector and the target parameters input by the operator, and jointly optimizes the target position vector, the target attitude quaternion, and the tactile response setting parameters to generate a control instruction parameter group that meets the error tolerance and action feasibility; The control instruction parameter group includes a desired three-dimensional position vector, a desired posture quaternion and a tactile response setting parameter; S65. The simulated predicted action process and expected response result are displayed in real time in the user control interface, and the operator is allowed to fine-tune the control intention. The adjusted control intention enters steps S61-S65 again until the user confirms.

8. The digital twin interactive control system based on robot vision and tactile fusion according to claim 2 is characterized in that: The S8 includes the following specific steps: S81, while executing the control instruction parameter group, continuously collecting visual image data and tactile sensor data; S82. Construct a fusion state vector at the current moment using the feature results output by the spatial perception modeling and the tactile state modeling. Compare the vector with the fusion state vector constructed at the previous moment, calculate the amplitude of the three-dimensional position change, the angular velocity of the posture change, and the amplitude of the tactile response change, and obtain three execution evaluation indicators: the posture change rate, the tactile response sensitivity, and the motion accuracy error. S83. Perform dynamic stability assessment on the attitude change rate to determine whether there is a risk of directional control drift at the robot end effector; Evaluate the tactile response timeliness to determine whether there is a hysteresis response or sudden imbalance in the contact behavior; Evaluate the error threshold of the motion accuracy error to determine whether the execution result meets the tolerance requirements of position and posture control; S84. Triggering an adaptive adjustment operation of the fusion state vector construction mechanism based on the combined judgment result of the three execution evaluation indicators, wherein the adaptive adjustment operation adjusts the synchronization buffer length in the channel alignment mechanism; S85. Use the adjusted construction mechanism to generate the fusion state vector at the next moment and input it into the digital twin mapping module to ensure that the digital twin is synchronized with the robot body in terms of three-dimensional position, posture orientation, and tactile simulation feedback; S86. During each round of fusion state vector construction and synchronization, record the fusion state vector change data, control instruction execution effect and adjustment behavior to form a closed-loop control log; S87. If an abnormal posture change rate, deviation in tactile response sensitivity, or movement accuracy error exceeding a preset threshold is detected in multiple consecutive control cycles, a local alarm is triggered to suspend the current control process, and the current control instruction parameter group and fusion state vector are cached in the state recovery buffer, waiting for user intervention or re-execution after the recovery condition is triggered.

Images