Engineering machinery remote accurate control system based on touch sensing and force feedback

Abstract

The invention relates to the technical field of engineering machinery teleoperation, and discloses an engineering machinery remote accurate control system based on tactile sensing and force feedback, which comprises a far-end operation subsystem, a near-end control subsystem and a central processing unit, wherein the far-end operation subsystem comprises a high-fidelity touch sensing array, a dynamic environment sensing module and an intelligent slave end controller; the near-end control subsystem comprises a myoelectricity sensor and an intention sensing module; and the central processing unit comprises a virtual environment online prediction module and a self-adaptive hybrid force feedback module. Through the intention perception module composed of the dual-channel signal decoupling unit and the adaptive learning unit, cross interference among different muscle group signals and physiological differences among different operators are effectively eliminated; according to the method, a personalized and stable intention mapping model can be quickly established for any new user, a standardized control intention signal is output, and the learning cost and the use threshold of the system are greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of remote operation technology for engineering machinery, specifically a remote precision control system for engineering machinery based on tactile sensing and force feedback. Background Technology

[0002] Construction machinery (such as excavators, cranes, and demolition machines) plays an irreplaceable role in dangerous or harsh environments such as mining, emergency rescue, and nuclear facility processing. Remote control technology is a key means of ensuring personnel safety and expanding the operational scope. Traditional remote control systems for construction machinery mainly rely on visual feedback channels, that is, transmitting on-site images back to the operator via cameras installed on the remote machinery. However, this single visual feedback method has serious drawbacks: 1. Lack of force information: Operators cannot perceive key mechanical information such as contact force, vibration, and texture when the robotic arm interacts with the environment; this leads to rough operation, low efficiency, and is prone to causing equipment overload or task failure.

[0003] 2. Heavy workload and low precision: Operators need to concentrate fully on interpreting two-dimensional planar images to infer three-dimensional spatial relationships and mechanical states, resulting in an extremely high cognitive load. Especially in environments with visual obstruction, insufficient light, or dust, performing delicate operations (such as threading ropes, inserting pins, and fine excavation) is extremely difficult, leading to long task completion times and a high error rate.

[0004] 3. Lack of proactive guidance and obstacle avoidance assistance: Existing systems mostly use "direct mapping" control, which means that the operator's control commands are sent one-to-one to the remote machinery. The system itself does not have environmental understanding and intelligent assistance capabilities. Operators must rely entirely on their own judgment to avoid obstacles and accurately approach targets, which requires extremely high operational skills and makes it difficult to guarantee safety in complex and dynamic environments.

[0005] 4. Poor adaptability: Physiological differences among operators (such as muscle strength and wearing habits) can lead to unstable performance of control interfaces based on biosignals (such as electromyography). Furthermore, the system struggles to adapt to the impact of outdoor temperature changes on high-precision sensors, as well as dynamic changes in the environment during operation.

[0006] In recent years, although some studies have attempted to introduce force feedback into teleoperation, most have remained at the stage of simple direct force reflection, that is, directly mapping the force measured by the remote sensor onto the operating handle. This method provides limited force information, cannot distinguish between beneficial operating resistance and dangerous collision forces, and cannot provide predictive guidance. In addition, the environmental modeling of existing systems is mostly offline or has a low update rate, which cannot be synchronized with the rapidly changing work site, resulting in a disconnect between virtual predictions and real conditions, and the guidance force may fail or even be misleading. Summary of the Invention

[0007] The purpose of this invention is to provide a remote precision control system for engineering machinery based on tactile sensing and force feedback, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A remote precision control system for engineering machinery based on tactile sensing and force feedback includes a remote operation subsystem, a near-end control subsystem, and a central processing unit; wherein, The remote operation subsystem includes: A high-fidelity tactile sensor array is installed at the end effector and key force-bearing joints of the robotic arm of engineering machinery to provide real-time multi-dimensional force perception with sub-millinewton resolution, and has a built-in online temperature compensation unit. The dynamic environment perception module consists of a 3D depth vision sensor and a lidar; it is used to acquire 3D spatial information of the work site in real time and construct 3D point cloud data. The intelligent slave controller is used to execute control commands and synchronously collect data from the high-fidelity tactile sensor array and the dynamic environment perception module. The proximal control subsystem includes: Electromyography (EMG) sensors are used to collect bioelectrical signals generated when an operator performs specific actions. The intent perception module, connected to the electromyography sensor, is used to process bioelectrical signals and identify and generate control intentions. The central processing unit, which is communicatively connected to both the remote operation subsystem and the local control subsystem, includes: The virtual environment online prediction module is used to build and update a local virtual environment that is synchronized with the remote working environment in real time based on 3D point cloud data. The adaptive hybrid force feedback module is used to receive control intentions, local virtual environment information, and real-time force data from a high-fidelity tactile sensor array, and generate hybrid force feedback commands.

[0009] As a further aspect of the present invention: the intent-aware module includes: A dual-channel signal decoupling unit is used to eliminate cross-interference between the two raw bioelectrical signals acquired from the electromyography sensor; The adaptive learning unit is used to dynamically adjust the normalization parameters of each channel signal based on the signal characteristics of the current operator, so as to output a standardized control intention signal.

[0010] As a further aspect of the present invention, the method for eliminating cross-interference of bioelectrical signals in the dual-channel signal decoupling unit is as follows: S201. The first channel raw bioelectric signal and the second channel raw bioelectric signal obtained from the electromyography sensor are preprocessed to obtain the first channel envelope signal and the second channel envelope signal, respectively. S202. Based on the pre-calibrated decoupling matrix, perform linear decoupling operation on the first channel envelope signal and the second channel envelope signal to obtain the first channel decoupled signal and the second channel decoupled signal. S203. Normalize the decoupled signals of the first channel and the decoupled signals of the second channel and then output them.

[0011] As a further aspect of the present invention: the method for standardizing the output of the control intention signal in the adaptive learning unit is as follows: S211. Initialization phase: Collect the baseline values ​​of the first channel and the second channel when the current operator is in a relaxed state, as well as the first initial reference value and the second initial reference value under the maximum voluntary contraction action of each channel, and set the first current reference value to the first initial reference value and the second current reference value to the second initial reference value. S212. During the online operation phase, continuously monitor the decoupled signals of the first channel and the second channel, and dynamically update the first current reference value and the second current reference value according to preset conditions. S213. Based on the baseline values ​​of the first channel and the second channel, as well as the first current reference value and the second current reference value, perform standardization calculations on the real-time input decoupled signals of the first channel and the decoupled signals of the second channel to generate standardized control intention signals of the first channel and the second channel. S304: Outputs the standardized control intention signal from the first channel and the standardized control intention signal from the second channel after smoothing and filtering.

[0012] As a further aspect of the present invention: the method for constructing and updating the local virtual environment in the online virtual environment prediction module is as follows: S31. Receive and synchronize the 3D point cloud data from the dynamic environment perception module, and convert it to the preset world coordinate system to obtain the world coordinate system point cloud. S32. Combining the robotic arm body state data from the intelligent slave controller, filter out the point cloud belonging to the structure of the engineering machinery itself from the world coordinate system point cloud to obtain the pure environmental point cloud. S33. Perform dynamic segmentation on the pure environment point cloud to distinguish between static background point cloud and dynamic object point cloud. S34. For static background point clouds, take the real-time pose of the robotic arm end effector as the center, select local neighborhood point clouds, and use the sliding least squares method to fit the local environment surface online to obtain the environmental feature parameter set at the current moment. S35. Based on the environmental feature parameter set and the dynamic object point cloud, incrementally update the local virtual environment model. S36. Output the updated local virtual environment model and environmental feature parameter set to the adaptive hybrid force feedback module.

[0013] As a further aspect of the present invention: In step S34, the specific process of using the sliding least squares method to fit the local environment surface online and solve for the set of environmental feature parameters is as follows: S341. Local surface modeling: Construct a spherical neighborhood centered on the end effector position and with a set radius; S342. Construct a weighted least squares problem: Define the weights of environmental points within a spherical neighborhood; then, using the parameters to be identified as optimization variables, minimize the weighted residual sum of squares function. S343. Online parameter solving and updating: The parameters that minimize the weighted residual sum of squares function are solved online using the recursive least squares algorithm and used as the set of environmental feature parameters at the current time.

[0014] As a further aspect of the present invention: in the adaptive hybrid force feedback module, the method for generating hybrid force feedback commands is as follows: S41. Receive and process real-time force data from the high-fidelity tactile sensor array, and output the real contact force vector after coordinate transformation, filtering and perception mapping. S42. Receive local virtual environment information from the virtual environment online prediction module, and calculate the virtual prediction guidance force vector online based on the artificial potential field method, combined with the current end pose of the robotic arm, the target pose, and the obstacle pose. S43. Analyze the time-domain and frequency-domain characteristics of the real contact force vector, extract key characteristic forces, and perform nonlinear amplification processing to generate a tactile enhancement force vector; S44. Receive the control intention signal from the intention perception module, and perform weighted fusion and vector synthesis of the real contact force vector, virtual predicted guidance force vector and tactile enhancement force vector according to the control intention signal; and perform safety limiting on the synthesized force vector, and finally output the hybrid force feedback command.

[0015] In another aspect, this application also provides a method for a remote precision control system for engineering machinery based on tactile sensing and force feedback, comprising the following steps: S1. The remote operation subsystem collects multi-dimensional force information and three-dimensional environmental information of the interaction between the operating machinery and the environment in real time, and transmits them back to the central processing unit. S2. Collect operator bioelectric signals through the proximal control subsystem, and generate control intention signals after processing by the intention perception module; S3. The virtual environment online prediction module of the central processing unit updates the local virtual environment synchronized with the remote environment in real time based on the returned three-dimensional environment information. S4, the adaptive hybrid force feedback module integrates the control intention signal, local virtual environment information and multi-dimensional force information to calculate and generate hybrid force feedback commands; S5. Apply the hybrid force feedback command to the force feedback actuator of the near-end control subsystem, and at the same time send the control command generated based on the control intention signal to the remote operation subsystem to drive the construction machinery to perform the corresponding action, forming a remote operation closed loop.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves multidimensional force measurement with sub-millinewt resolution by integrating a high-fidelity tactile sensor array with online temperature compensation at the end effector and key joints of a robotic arm. This enables the system to capture extremely subtle mechanical features such as soil texture differences, object surface textures, and wire rope micro-slippage. Through the mapping of real contact, the operator can feel the mechanical interaction state at the remote end in an immersive way.

[0017] The intention perception module, composed of a dual-channel signal decoupling unit and an adaptive learning unit, effectively eliminates cross-interference between signals from different muscle groups and physiological differences between different operators. The system can quickly establish a personalized and stable intention mapping model for any new user and output standardized control intention signals, which greatly reduces the learning cost and usage threshold of the system and improves the robustness and accuracy of intention recognition.

[0018] By intelligently synthesizing a hybrid force feedback command consisting of real contact force, virtual predictive guidance force, and tactile enhancement force, predictive assisted control is achieved, enabling operators to perceive risks in advance and easily aim at targets; and improving the operator's sensitivity to dangers or important events. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a remote precision control system for engineering machinery based on tactile sensing and force feedback. Figure 2 This is a flowchart illustrating a method for remote and precise control of engineering machinery based on tactile sensing and force feedback. Detailed Implementation

[0020] Please see Figure 1 In this embodiment of the invention, a remote precision control system for engineering machinery based on tactile sensing and force feedback includes a remote operation subsystem, a near-end control subsystem, and a central processing unit; wherein, The remote operation subsystem includes: A high-fidelity tactile sensor array is installed at the end effectors of the robotic arms of construction machinery (such as the teeth of the excavator bucket, the jaws of the hydraulic shears, and the fingertips of the grab bucket) and at key force-bearing joints (such as the connecting joints of each section of the robotic arm and the connecting pins between the bucket and the boom). It is used to provide real-time multi-dimensional force perception with sub-millinewt resolution and has a built-in online temperature compensation unit. The online temperature compensation unit can compensate the sensing signals in real time to maintain measurement stability in the field environment. The dynamic environment perception module consists of a 3D depth vision sensor (such as a structured light camera or a binocular stereo camera) and a lidar; it is used to acquire 3D spatial information of the work site in real time and construct 3D point cloud data. The intelligent slave controller is used to execute control commands and synchronously collect data from the high-fidelity tactile sensor array and the dynamic environment perception module. The proximal control subsystem includes: Electromyography (EMG) sensors are used to collect bioelectrical signals generated when an operator performs specific actions, such as when worn on the operator's forearm or hand, using bioelectrical signals from the surface of the forearm or hand. An intent perception module, connected to an electromyography (EMG) sensor, is used to process bioelectrical signals and identify and generate control intentions; it includes: A dual-channel signal decoupling unit is used to eliminate cross-interference between the two raw bioelectrical signals acquired from the electromyography sensor; the method for eliminating cross-interference of bioelectrical signals is as follows: S201, The first channel raw bioelectric signal obtained from the electromyography sensor Second channel raw bioelectric signal Preprocessing is performed separately, for example, the raw bioelectrical signals are processed. The bandpass filter, full-wave rectification, and cutoff frequency are Low-pass smoothing filtering is applied to extract the signal envelope; the first channel envelope signal is obtained. Second channel envelope signal ; S202, Based on the decoupling matrix obtained through pre-calibration For the first channel envelope signal Second channel envelope signal Perform linear decoupling operation to obtain the decoupled signal of the first channel. The signal after decoupling from the second channel ;in, , , , The decoupling matrix The four elements, the decoupling matrix The method to obtain it is as follows: The operator is guided to sequentially perform a first calibration action that primarily stimulates the muscles corresponding to the first channel and a second calibration action that primarily stimulates the muscles corresponding to the second channel. During the duration of the first calibration operation, the average value of the envelope signal of the first channel is calculated. The average value of the second channel envelope signal ; During the second calibration operation, the average value of the envelope signal of the first channel is calculated. The average value of the second channel envelope signal ; According to the formula Calculate the cross-interference coefficient; based on the formula. Calculate the decoupling matrix ; S203, Decoupling the signal from the first channel The signal after decoupling from the second channel Output after normalization.

[0021] The adaptive learning unit dynamically adjusts the normalization parameters of each channel signal based on the current operator's signal characteristics to output a standardized control intention signal. The method for outputting the standardized control intention signal is as follows: S211. Initialization phase: Acquire the first channel baseline value when the current operator is in a relaxed state. Compared with the second channel baseline value And the first initial reference value under the maximum voluntary contraction action of each channel. Compared with the second initial reference value And set the first current reference value The first initial reference value ,Right now Second current reference value for Second initial reference value ,Right now Among them, the first initial reference value Compared with the second initial reference value The method for obtaining it is as follows: Guide the operator to execute the maximum voluntary contraction action of the corresponding channel, and take the decoupled signal of the first channel during the duration of the action. Signal after decoupling from the second channel of The quantile is used as the initial reference value, where, The preset percentile values ​​are within... between; S212. During the online operation phase, continuously monitor the signal after decoupling of the first channel. Signal after decoupling from the second channel And dynamically update the first current reference value according to preset conditions. Compared with the second current reference value Among them, the first current reference value is dynamically updated. Compared with the second current reference value The specific method is as follows: First, for the first channel, when in the most recent time window Inside, Continue to exceed And the signal variance is below the threshold. When a strong contraction event occurs, the signal of the first channel within the window is decoupled. of Quantiles as candidate reference values ; Subsequently, according to the formula Update the first current reference value ,in, A forgetting factor close to 1; the second current reference value Update using the same logic; S213, Based on the baseline value of the first channel Compared with the second channel baseline value and the first current reference value Compared with the second current reference value The decoupled signal of the first channel input in real time Signal after decoupling from the second channel Perform standardized calculations to generate the first channel standardized control intention signal. Standardized control intention signal with second channel ;in, For the amplitude limiting function, when Time return ,when Time return Otherwise return ; S304: Standardized control intention signal for the first channel Standardized control intention signal with second channel The output is after smoothing filtering.

[0022] The central processing unit, which is communicatively connected to both the remote operation subsystem and the local control subsystem, includes: The online virtual environment prediction module is used to construct and update a local virtual environment synchronized with the remote operating environment in real time based on 3D point cloud data; the method for constructing and updating the local virtual environment is as follows: S31. Receive and synchronize 3D point cloud data from the dynamic environment perception module. Then, they are uniformly transformed to the preset world coordinate system to obtain the world coordinate system point cloud. ; S32, Combine the robotic arm body state data from the intelligent slave controller. From the point cloud in the world coordinate system Point clouds belonging to the structure of the engineering machinery itself are filtered out to obtain pure environmental point clouds. ; S33, Pure environmental point clouds Perform dynamic segmentation to distinguish the static background point cloud. Point clouds of dynamic objects ; S34, Point Cloud for Static Background Based on the real-time pose of the robotic arm's end effector Centered on a local neighborhood point cloud, a sliding least squares method is used. Online fitting of the local environment surface yields the set of environmental feature parameters at the current moment. Among them, the set of environmental characteristic parameters This includes: the geometric parameters of the local environment surface, and the equivalent stiffness coefficient of the surface estimated online based on the deviation between historical tactile data and the point cloud model. Geometric parameters are used to extract the normal vectors of the environment surface. and curvature characteristics; among which, the environmental feature parameter set is solved. The specific process is as follows: S341, Local Surface Modeling: Constructing a surface model based on the end effector position. Centered on, and set the corresponding radius A spherical neighborhood (e.g., 0.5m to 1.0m) ; S342. Constructing a weighted least squares problem: Defining a spherical neighborhood. Internal environment points weight ,in, For compactly supported radial basis functions; Parameters to be identified To optimize the variables, minimize the weighted sum of squared residuals function. ,in, The equations for the parameterized local surface; S343. Online Parameter Solving and Update: The weighted residual sum of squares function is solved online using a recursive least squares algorithm. Minimum parameter As the set of environmental feature parameters at the current moment ; S35. Based on the set of environmental characteristic parameters and dynamic object point clouds For local virtual environment models Incremental updates are performed; among them, the local virtual environment model Storage and management are performed using octrees or hierarchical bounding box data structures; incremental updates include: adding environmental feature parameter sets. The newly represented surface patches are inserted or updated into the data structure, and dynamic object point clouds are processed. A simplified geometric bounding volume is used for temporary characterization and tracking; S36. Update the local virtual environment model With environmental characteristic parameter set Output to the adaptive hybrid force feedback module; Assuming the initial state is a remote scene consisting of piles of cement blocks and oil drums, then the following steps will be performed: 1) The lidar scans the entire field, generating an initial point cloud. After filtering, the ground plane, two cement blocks (stable point cloud clusters), and one oil drum (stable point cloud cluster) were identified. 2) The operator begins to maneuver the grab bucket closer to the oil drum; at this time, the module uses points on the expected path of the grab bucket as centers to perform point cloud mapping on the surface of the oil drum. Fitting; assuming the oil drum is cylindrical, SLSM can identify its radius online. The position and orientation of the central axis are determined, and this parametric cylinder model is updated accordingly. In the middle, it is marked as "target, graspable, medium stiffness"; 3) At the same time, a concrete block (obstacle) is also modeled as a cube bounding box, and its parameters (length, width, height, position, and orientation) are identified online and stored in the model; 4) When the remote grab bucket actually contacts the oil drum, the tactile sensor array transmits the contact force. The module, combining the contact point location with a virtual oil drum model, can estimate the local deformation at the contact point and fine-tune the stiffness coefficient of that area. This updated, more accurate model It was immediately used to calculate more precise virtual guiding force. For example, the normal vector of the oil drum surface. It is used to calculate the direction of the clamping force required during grasping.

[0023] The adaptive hybrid force feedback module receives control intentions, local virtual environment information, and real-time force data from a high-fidelity tactile sensor array, and generates hybrid force feedback commands; its hybrid force feedback commands The generation method is as follows: S41. Receive and process real-time force data from the high-fidelity tactile sensor array. After coordinate transformation, filtering, and sensor mapping, the true contact force vector is output. ; S42. Receive local virtual environment information from the online prediction module for the virtual environment, and based on the artificial potential field method, combine it with the current pose of the robotic arm's end effector. Target pose and obstacle pose Online calculation of virtual predictive guidance force vector Among them, the virtual predicted guiding force vector For all virtual gravity With virtual repulsion The vector sum of, where, virtual gravity The calculation formula is as follows: ,in, For example, take the gravitational gain coefficient. ; Virtual repulsion Minimum distance between the end effector of the robotic arm and the virtual obstacle Less than the safe distance It is activated when (e.g., 0.3m) is taken, that is Furthermore, its size is limited to a preset warning force limit. Within; Virtual repulsion The calculation formula is as follows: ;in, For repulsive force gain coefficient (e.g.) ), To obtain the pose from the obstacle Pointing to the current position of the robotic arm's end effector The unit direction vector; To improve stability, when When approaching 0 (about to collide), Replace with a saturated warning force Its size smoothly grows to its upper limit. (e.g., 8N), to avoid sudden changes in force; S43. Analyze the actual contact force vector. The time and frequency domain features are analyzed to extract key force features, which are then nonlinearly amplified to generate haptic enhancement force vectors. The extraction of key feature forces includes: calculating the true contact force vector. The time gradient, and the gradient value exceeding the preset threshold. Force change events are used as transient impact characteristics; then the actual contact force vector is analyzed. Bandpass filtering is performed on a specific frequency band, and the filtered signal is used as the vibration characteristic force. Nonlinear amplification processing uses saturated nonlinear functions ,in, The key feature force scalar is the input. This is the amplification gain coefficient. To control the linear interval, It is a hyperbolic tangent function, which amplifies small signals significantly but saturates large signals to prevent distortion. S44. Receive the control intention signal from the intention perception module. Based on the control intention signal, the actual contact force vector Virtual Predictive Guiding Force Vector and tactile enhancement force vector Weighted fusion and vector synthesis are performed; whereby the synthesized force vector... The calculation formula is as follows: ;in, , , These are the corresponding weighting coefficients; And the synthesized force vector Perform safety limiting and finally output hybrid force feedback command. The formula for calculating the safety limit is as follows: ;in, This is the maximum output force of the construction machinery and equipment; thus ensuring that the maximum output force of the equipment is not exceeded. (e.g., 15N) and operator comfort range; Suppose an operator attempts to remotely control a grab bucket to pick up an oil drum in a virtual environment. There is a cement block next to it. The remote grab bucket has made slight contact with the surface of the oil drum. 1) Input: The "closing gripper" and "micro-advancement" intentions derived from electromyographic signals; : Includes oil drum (target, stiffness) ), cement blocks (obstacles, The model; the normal vector of the oil drum contact point. ; : Tactile sensors detect contact force (Mainly positive pressure), and superimposed with approximately Amplitude, Slight tangential vibrations (which may indicate slippage); 2) Processing procedure: (1) After mapping Mapping ratio ); (2) Calculation: Pointing to the center of the oil drum, size ; Grab bucket distance from cement block Therefore Therefore = 0.8N (gravity); (3) Analysis The tangential vibration component is extracted. (original After mapping); after nonlinear amplification Therefore (A noticeable lateral shaking sensation); (4) Because it is in "fine crawling" mode, the weight of each is 1; Composite: ; Assuming the resultant force does not exceed the limit, That is, the vector; 3) Operator's sensation: The hand simultaneously feels: A vertical line to the palm Pressure (actual contact force); A hand pulled towards the center of the oil drum Guiding force (virtual guiding force); A noticeable left-right shaking sensation ( (Haptic enhancement), strongly indicating a risk of slippage on the surface of the oil drum; The operator then subconsciously adjusts their hand posture and increases grip strength to achieve stable grasping at a distance.

[0024] Please see Figure 2 In this embodiment of the invention, a method for remote and precise control of engineering machinery based on tactile sensing and force feedback includes the following steps: S1. The remote operation subsystem collects multi-dimensional force information and three-dimensional environmental information of the interaction between the operating machinery and the environment in real time, and transmits them back to the central processing unit. Suppose an operator needs to remotely control an excavator located in a mining area to complete the following complex tasks: dig up sand mixed with a small amount of pebbles from area A; transport the sand to a hopper in area B, 5 meters away, while navigating around a fixed monitoring pole; and dump the sand into the hopper, preventing spillage. Then, 1) Multidimensional force information acquisition: The high-fidelity tactile sensor array on the excavator bucket teeth measures the three-dimensional contact force in real time when cutting into sand; for example, when the bucket teeth cut into pebbles, the sensor array... (Horizontal) Approximately detected in the (vertical) direction The force pulse was detected, and at the same time, the force pulse generated by the rolling of the pebble was detected. The rolling torque; the sensor array at the key joints of the robotic arm synchronously measures the load of each joint, such as the load force measured at the hydraulic cylinder pin of the boom. Intelligent slave controller with Frequency acquisition of all force sensory data, packaged into a data stream ; 2) Three-dimensional environmental information acquisition: Installed on the top of the excavator cab. Linear lidar with Frequency rotation scanning, binocular stereo camera with Frequency-based imaging is used to jointly generate point cloud data of the work site. Point clouds clearly contain Sand piles in the area Geometric information of the hopper, monitoring pole, excavator itself, and distant background; 3) Data feedback: and After compression and encoding, through A dedicated network transmits data back to the central processing unit located in the control room. End-to-end communication delay. ; S2. Collect operator bioelectric signals through the proximal control subsystem, and generate control intention signals after processing by the intention perception module; Assuming we continue with the example from step S1, then... 1) Bioelectrical signal acquisition: The operator wears a device on their right forearm. The channel electromyography (EMG) sensor array collects surface EMG signals generated when the user performs actions such as "clenching a fist" (controlling bucket closing), "wrist dorsiflexion" (controlling boom lifting), and "forearm pronation" (controlling stick retraction). ; 2) Intent perception and processing: (1) Dual-channel signal decoupling unit: Taking the easily confused actions of "clenching a fist" and "forearm pronation" as examples. Original signal (Taken from the flexor muscles of the fingers) and (Taken from the pronator teres muscle) Cross-interference exists; the dual-channel signal decoupling unit uses a pre-calibrated decoupling matrix. (For example The pure intent signal is obtained through real-time calculation. (Fist clenching strength) and (Spin strength); (2) Adaptive learning unit: For a new operator (with average muscle development), the corresponding "fully clenched fist" is The maximum value is After completing the unit online learning, Dynamically adjusted to When the operator clenches their fist with moderate force... At that time, the unit standardizes it into a control intention signal. ; 3) Intent signal generation: Finally, the intent perception module outputs a set of standardized control intent signals. These respectively represent the combined intentions of "raising the boom to medium force", "keeping the stick still", "keeping the bucket still", and "slightly turning the platform to the right". S3. The virtual environment online prediction module of the central processing unit updates the local virtual environment synchronized with the remote environment in real time based on the returned three-dimensional environment information. Assuming we continue with the example from step S2, then... 1) Data reception: The virtual environment online prediction module receives the latest point cloud data. ; 2) Environment Modeling: (1) The module first removes the part of the point cloud that belongs to the excavator itself (using the known excavator model and joint angles). (2) For the remaining environmental point cloud, the sliding least squares method is used. Perform online surface fitting; Taking the front of the bucket as an example: with the center of the bucket For the center of the ball, Construct a neighborhood for the radius; within this neighborhood, approximately A point describing the surface of sandy soil. Fit a quadratic surface equation Solve for the parameters Extract the normal vector of the surface at that location. (Approximately vertical upward) and curvature; Taking monitoring poles as an example: for pole-shaped point clusters in point clouds, The model was fitted to a cylinder, and its central axis position and radius were identified. And direction, and mark as obstacles; 3) Model Update: The fitted sandy soil surface patch, cylindrical obstacle model, and hopper geometric model (pre-known, with pose updated via point cloud registration) are incrementally updated to the local virtual environment model with an octree structure. The geometric error between the model and the real-world environment is... ; 4) Output: The module will... and characteristic parameters (such as surface stiffness estimation) Output; S4, the adaptive hybrid force feedback module integrates the control intention signal, local virtual environment information and multi-dimensional force information to calculate and generate hybrid force feedback commands; Assuming we continue with the example from step S3, then... 1) Input convergence: The adaptive hybrid force feedback module receives: (1) Manipulation intention ; (2) Local virtual environment and ; (3) Real-time force data (after mapping) ); 2) Calculate the three types of forces: (1) Actual contact force :from The force pulses from the collision between the bucket and the gravel are extracted and mapped to generate a... Instantaneous feedback force.

[0025] (2) Virtual Predictive Guidance : Gravity: According to Location of the unloading port of the middle hopper Calculate a size of Gravity pointing towards the unloading port ; Repulsion: The distance between the bucket and the monitoring pole in the virtual model was detected. Calculate a size of Repulsive force in the direction away from the upright ; ; (3) Tactile enhancement :right High-frequency pulses (characteristic frequency band) generated by the collision of pebbles ), applying nonlinear amplification functions , will be faint Lateral jitter amplified to obvious lateral pulse force ; 3) Integration and Output: Based on the "Refined Mining" control mode (by... (Implicit), setting weights , , Perform vector synthesis: After synthesis, a safety limit is applied (to ensure...). Finally, a hybrid force feedback command is generated. For example, for one The force vector; S5. Apply the hybrid force feedback command to the force feedback actuator of the near-end control subsystem, and at the same time send the control command generated based on the control intention signal to the remote operation subsystem to drive the construction machinery to perform the corresponding action, forming a remote operation closed loop. Assuming we continue with the example from step S4, then... 1) Proximal force feedback application: The command is sent to the six-DOF force feedback joystick in the operator's hand; the joystick's torque motor precisely outputs the three-dimensional force; the operator's hand immediately feels it. (1) An obvious Vertical upward force (reproduces bucket excavation resistance); (2) A slight pull towards the hopper and a push away from the upright (virtual guide); (3) A clear left-right shaking sensation (enhanced pebble collision indication); 2) Issuance and execution of remote control commands: Simultaneously, the central processing unit will control the intent signals. This is resolved into a specific set of joint space control commands with velocity and acceleration feedforwards: boom cylinder displacement Platform turnaround The instruction is sent to the remote excavator's intelligent slave controller via the same 5G network. 3) Closed-loop formation: (1) The remote excavator executes the action according to the instruction, the boom begins to lift, and the platform begins to turn to the right; (2) The new mechanical state and the resulting new environmental interaction forces and point clouds are collected and transmitted back again; (3) After the operator feels the updated force feedback, he adjusts his muscle activity according to the progress of the task (such as feeling that he has bypassed the pole and the pebbles have been cleared) and generates a new control intention. (4) The system repeats itself, forming a real-time closed loop of "perception → decision → action → feedback".

[0026] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A remote precision control system for engineering machinery based on tactile sensing and force feedback, characterized in that, It includes a remote operation subsystem, a local control subsystem, and a central processing unit; among which, The remote operation subsystem includes: A high-fidelity tactile sensor array is installed at the end effector and key force-bearing joints of the robotic arm of engineering machinery to provide real-time multi-dimensional force perception with sub-millinewton resolution, and has a built-in online temperature compensation unit. The dynamic environment perception module consists of a 3D depth vision sensor and a lidar; it is used to acquire 3D spatial information of the work site in real time and construct 3D point cloud data. The intelligent slave controller is used to execute control commands and synchronously collect data from the high-fidelity tactile sensor array and the dynamic environment perception module. The proximal control subsystem includes: Electromyography (EMG) sensors are used to collect bioelectrical signals generated when an operator performs specific actions. The intent perception module, connected to the electromyography sensor, is used to process bioelectrical signals and identify and generate control intentions. The central processing unit, which is communicatively connected to both the remote operation subsystem and the local control subsystem, includes: The virtual environment online prediction module is used to build and update a local virtual environment that is synchronized with the remote working environment in real time based on 3D point cloud data. The adaptive hybrid force feedback module is used to receive control intentions, local virtual environment information, and real-time force data from a high-fidelity tactile sensor array, and generate hybrid force feedback commands.

2. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 1, characterized in that, The intent-aware module includes: A dual-channel signal decoupling unit is used to eliminate cross-interference between the two raw bioelectrical signals acquired from the electromyography sensor; The adaptive learning unit is used to dynamically adjust the normalization parameters of each channel signal based on the signal characteristics of the current operator, so as to output a standardized control intention signal.

3. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 2, characterized in that, In the dual-channel signal decoupling unit, the method for eliminating cross-interference of bioelectrical signals is as follows: S201. The first channel raw bioelectric signal and the second channel raw bioelectric signal obtained from the electromyography sensor are preprocessed to obtain the first channel envelope signal and the second channel envelope signal, respectively. S202. Based on the pre-calibrated decoupling matrix, perform linear decoupling operation on the first channel envelope signal and the second channel envelope signal to obtain the first channel decoupled signal and the second channel decoupled signal. S203. Normalize the decoupled signals of the first channel and the decoupled signals of the second channel and then output them.

4. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 2, characterized in that, In the adaptive learning unit, the method for standardizing the output of the control intention signal is as follows: S211. Initialization phase: Collect the baseline values ​​of the first channel and the second channel when the current operator is in a relaxed state, as well as the first initial reference value and the second initial reference value under the maximum voluntary contraction action of each channel, and set the first current reference value to the first initial reference value and the second current reference value to the second initial reference value. S212. During the online operation phase, continuously monitor the decoupled signals of the first channel and the second channel, and dynamically update the first current reference value and the second current reference value according to preset conditions. S213. Based on the baseline values ​​of the first channel and the second channel, as well as the first current reference value and the second current reference value, perform standardization calculations on the real-time input decoupled signals of the first channel and the decoupled signals of the second channel to generate standardized control intention signals of the first channel and the second channel. S304: Outputs the standardized control intention signal from the first channel and the standardized control intention signal from the second channel after smoothing and filtering.

5. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 1, characterized in that, The method for constructing and updating the local virtual environment in the online virtual environment prediction module is as follows: S31. Receive and synchronize the 3D point cloud data from the dynamic environment perception module, and convert it to the preset world coordinate system to obtain the world coordinate system point cloud. S32. Combining the robotic arm body state data from the intelligent slave controller, filter out the point cloud belonging to the structure of the engineering machinery itself from the world coordinate system point cloud to obtain the pure environmental point cloud. S33. Perform dynamic segmentation on the pure environment point cloud to distinguish between static background point cloud and dynamic object point cloud. S34. For static background point clouds, take the real-time pose of the robotic arm end effector as the center, select local neighborhood point clouds, and use the sliding least squares method to fit the local environment surface online to obtain the environmental feature parameter set at the current moment. S35. Based on the environmental feature parameter set and the dynamic object point cloud, incrementally update the local virtual environment model. S36. Output the updated local virtual environment model and environmental feature parameter set to the adaptive hybrid force feedback module.

6. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 1, characterized in that, In step S34, the specific process of using the sliding least squares method to fit the local environment surface online and solve for the set of environmental feature parameters is as follows: S341. Local surface modeling: Construct a spherical neighborhood centered on the end effector position and with a set radius; S342. Construct a weighted least squares problem: Define the weights of environmental points within a spherical neighborhood; then, using the parameters to be identified as optimization variables, minimize the weighted residual sum of squares function. S343. Online parameter solving and updating: The parameters that minimize the weighted residual sum of squares function are solved online using the recursive least squares algorithm and used as the set of environmental feature parameters at the current time.

7. The remote precision control system for engineering machinery based on tactile sensing and force feedback according to claim 1, characterized in that, In the adaptive hybrid force feedback module, the hybrid force feedback command is generated as follows: S41. Receive and process real-time force data from the high-fidelity tactile sensor array, and output the real contact force vector after coordinate transformation, filtering and perception mapping. S42. Receive local virtual environment information from the virtual environment online prediction module, and calculate the virtual prediction guidance force vector online based on the artificial potential field method, combined with the current end pose of the robotic arm, the target pose, and the obstacle pose. S43. Analyze the time-domain and frequency-domain characteristics of the real contact force vector, extract key characteristic forces, and perform nonlinear amplification processing to generate a tactile enhancement force vector; S44. Receive the control intention signal from the intention perception module, and perform weighted fusion and vector synthesis of the real contact force vector, virtual predicted guidance force vector and tactile enhancement force vector according to the control intention signal; and perform safety limiting on the synthesized force vector, and finally output the hybrid force feedback command.

8. A method for implementing the remote precision control system for engineering machinery based on tactile sensing and force feedback as described in claim 1, characterized in that, Includes the following steps: S1. The remote operation subsystem collects multi-dimensional force information and three-dimensional environmental information of the interaction between the operating machinery and the environment in real time, and transmits them back to the central processing unit. S2. Collect operator bioelectric signals through the proximal control subsystem, and generate control intention signals after processing by the intention perception module; S3. The virtual environment online prediction module of the central processing unit updates the local virtual environment synchronized with the remote environment in real time based on the returned three-dimensional environment information. S4, the adaptive hybrid force feedback module integrates the control intention signal, local virtual environment information and multi-dimensional force information to calculate and generate hybrid force feedback commands; S5. Apply the hybrid force feedback command to the force feedback actuator of the near-end control subsystem, and at the same time send the control command generated based on the control intention signal to the remote operation subsystem to drive the construction machinery to perform the corresponding action, forming a remote operation closed loop.

Images