Dynamic compensation multi-mode interaction control device and method based on gravity center pressure field

Abstract

The invention discloses a dynamic compensation multi-modal interaction control device and method based on a gravity center pressure field, and relates to the field of multi-degree-of-freedom robot control and VR interaction equipment. The device comprises a pedal assembly, an embedded control unit, a dynamic compensation mechanism and a communication module, can collect plantar pressure distribution data of an operator in real time, calculate the gravity center track of a human body and generate a corresponding control signal to control the action of external equipment, and has a dynamic compensation function to maintain the balance of the operator. According to the method, through the steps of pressure data collection, data processing and gravity center track calculation, control signal generation, dynamic compensation adjustment, mode switching and the like, natural, visual and efficient control over robot chassis movement and virtual character displacement is achieved; the problems that in the prior art, robot remote control operation is complex, learning cost is high, VR interaction is high in dizziness, and the equipment size is large are solved, and the wide application prospect is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-degree-of-freedom robot control and VR interactive equipment, and relates to a dynamic compensation multimodal interactive control device and method based on a center of gravity pressure field, and in particular to an interactive control device and method for multi-degree-of-freedom robot control and VR (Virtual Reality) interactive equipment. Background Art

[0002] Interactive control is a very important technical field in both robotics and VR. Robot interactive control refers to the use of hardware or software interfaces to achieve real-time or semi-real-time control of robot behavior by humans, while the robot feeds back environmental information through sensors to form a two-way interaction. Its core is to enable robots to perform tasks efficiently and safely and adapt to dynamic environments or human needs. In terms of robot remote control, high-degree-of-freedom systems (such as more than 16 degrees of freedom) rely on inverse kinematics algorithms and discrete button combinations to achieve complex operations. For example, humanoid robots need to use more than 30 button combinations to complete walking and upper limb coordinated movements. In order to improve control accuracy, some solutions introduce exoskeleton mapping technology (such as Tesla Optimus uses optical finger tracking), but its equipment is bulky and expensive, and requires complex sensor calibration and ambient lighting conditions, making it difficult to popularize. In addition, existing force-position coordination control algorithms are mostly based on terminal impedance models, which require precise dynamic modeling. When the robot contacts the environment, it is impossible to achieve whole-arm compliant control, which limits the safety of human-machine collaboration.

[0003] CN119458311A discloses a remote control system and method based on an easy-to-wear motion capture force interaction device. It includes: a motion capture force interaction device, which is used to obtain the motion information of the operator's upper limbs and obtain the environmental force information at the remote end, and then perform force interaction; a data communication and control module, which is used to issue control instructions in both directions and record and visualize data; a remote robot perception and execution module, which is used to execute position control instructions and feed back the remote environmental force information to the motion capture force interaction device, and at the same time adjust the position, perceive and report the force information during the execution process.

[0004] VR interactive control aims to capture the user's behavior in the virtual environment through input devices and provide multi-sensory feedback to give the user an immersive experience. Its core is to reduce interaction delay, improve naturalness, and make the user feel that the interaction with the virtual world is real and credible. In the field of VR interaction, the mainstream solutions are divided into two categories: based on handles (such as the Meta Quest series) and mechanical motion platforms (such as the Disney Holotile treadmill). The former causes dizziness in users due to the separation of visual-vestibular perception, and the latter occupies an area of ​​more than 2m 2, the cost exceeds 50,000 yuan and is difficult to promote. Although some companies have tried to achieve remote control through VR headsets combined with motion capture devices, such solutions still require wearing complex equipment, and the movement trajectory is limited by physical space and cannot support unlimited scene exploration.

[0005] CN119536526A discloses an LBE enhanced multimodal perception interactive VR display control method and system, which collects the user's multimodal perception data through VR head display equipment and hand controller, uses Kalman filter algorithm to fuse position data and action data, calculates the user's continuous motion trajectory, and dynamically adjusts the activity range based on the continuous motion trajectory to generate a directional dynamic range. The multimodal data is synchronized, feature extracted and weighted fused to generate an interactive state vector.

[0006] In short, the existing robot interactive control device requires the handle to handle chassis movement and other degrees of freedom (such as robotic arms) at the same time. The way to operate the robot chassis movement through the handle is not intuitive, resulting in complex operation logic and high learning costs, and it is difficult to adapt to the complex control requirements of multi-degree-of-freedom robots. The existing VR interactive control device uses the handle to control the movement of the character in the virtual scene, which is not intuitive and is very easy to cause dizziness. There are problems such as unnatural movement, strong dizziness, and large equipment size. Summary of the invention

[0007] The purpose of the present invention is to provide a dynamic compensation multimodal interactive control device and method based on the center of gravity pressure field, so as to solve the problems of complex robot remote control operation, high learning cost, strong dizziness in VR interaction, and large equipment size in the prior art. The purpose of the present invention is achieved through the following specific technical solutions.

[0008] The primary aspect of the present invention is to provide a dynamic compensation multimodal interactive control device based on the center of gravity pressure field, including a pedal assembly, an embedded control unit and a communication module. The upper surface of the pedal assembly contacts the sole of the operator (the operator is standing or sitting), and the pedal assembly is provided with a pressure sensor array for real-time collection of the operator's sole pressure distribution data; the embedded control unit is electrically connected to the pressure sensor array, and is used to process the collected pressure distribution data, calculate the trajectory of the center of gravity of the human body, and generate corresponding control signals according to the trajectory of the center of gravity; the communication module is electrically connected to the embedded control unit, and is used to send the control signal to an external device, and the external device includes a multi-degree-of-freedom robot or a VR interactive device.

[0009] Furthermore, it also includes a dynamic compensation mechanism, which is electrically connected to the embedded control unit and is used to perform dynamic compensation adjustment on the pedal assembly according to the control signal to maintain the balance of the operator.

[0010] Furthermore, the pressure sensor array includes a plurality of piezoresistive sensors or resistive thin film sensors evenly distributed at four corners of the pedal assembly.

[0011] Furthermore, the dynamic compensation mechanism includes a dual-axis movable pedal, a harmonic reducer, a DC brushless motor and an angle sensor. The dual-axis movable pedal is connected to the DC brushless motor through the harmonic reducer. The angle sensor is embedded in the pedal axis for real-time detection of the pedal inclination angle.

[0012] Furthermore, the embedded control unit has a built-in mode selection algorithm for realizing fast switching between the robot remote control mode and the VR interaction mode.

[0013] Another aspect of the present invention is to provide a multi-degree-of-freedom robot, comprising the above-mentioned dynamic compensation multimodal interactive control device based on the center of gravity pressure field, wherein the multi-degree-of-freedom robot receives the control signal sent by the control device through the communication module to realize the coordinated action control of chassis movement and other degrees of freedom. The multi-degree-of-freedom robot has at least 16 degrees of freedom and can realize the precise control of complex actions, including walking, upper limb coordinated actions, etc. through the control device.

[0014] Another aspect of the present invention is to provide a VR device, comprising the above-mentioned dynamic compensation multimodal interactive control device based on the center of gravity pressure field, wherein the VR device receives the control signal sent by the control device through the communication module to realize the displacement control of the virtual character. The VR device supports sitting / standing dual-mode operation, and the user can move freely in the virtual world without wearing complex equipment, realizing borderless virtual scene exploration.

[0015] Another aspect of the present invention is to provide a dynamic compensation multimodal interactive control method based on the center of gravity pressure field, using the dynamic compensation multimodal interactive control device based on the center of gravity pressure field, comprising the following steps: Pressure data collection: The pressure sensor array provided on the pedal assembly collects the operator's foot pressure distribution data in real time; Data processing and center of gravity trajectory calculation: The collected pressure distribution data is transmitted to the embedded control unit for processing, and the center of gravity trajectory of the human body is calculated, including the center of gravity position coordinates and its change trend; Control signal generation: Based on the center of gravity trajectory, a corresponding control signal is generated according to a preset mapping relationship, and the control signal is used to control the action of an external device.

[0016] Furthermore, it also includes dynamic compensation adjustment: the operator's balance state is monitored in real time according to the center of gravity trajectory, and when it is detected that the operator's center of gravity deviation rate or amplitude exceeds a safety threshold, the pedal assembly is dynamically compensated and adjusted through a dynamic compensation mechanism to maintain the operator's balance.

[0017] Furthermore, it also includes mode switching: through the built-in mode selection algorithm of the embedded control unit, rapid switching between the robot remote control mode and the VR interaction mode is achieved.

[0018] Furthermore, in the data processing and center of gravity trajectory calculation steps, one or more combinations of median filtering, Kalman filtering, Gaussian filtering and outlier filtering are used to pre-process the pressure distribution data to improve the accuracy and stability of the data.

[0019] Furthermore, it also includes nonlinear weight compensation: a neural network model based on a multi-layer perceptron is used to compensate for the differences in pressure distribution of users of different weights through training with labeled data sets.

[0020] The technical solution provided by the present invention has the following beneficial technical effects.

[0021] In terms of robot remote control: functional decoupling improves efficiency, and the pedals independently control the movement of the chassis, allowing the handle to focus on fine operations such as the robotic arm. The chassis direction is driven by the pressure distribution on the sole of the foot, and natural interaction reduces the learning cost. The pedal-type design is small in size and low in cost, supports multi-degree-of-freedom expansion, and combines pressure field analysis to achieve borderless control.

[0022] VR control: Control the displacement of virtual characters through the natural center of gravity movement of the human body to reduce dizziness. The volume is less than 0.1m³, the cost is low, and it supports sitting / standing dual modes, and the deployment is flexible. Multi-modal interaction expansion, distinguishing user action intentions through pressure field analysis, combined with exoskeleton technology to achieve full-body interaction, breaking through the space limitations of traditional motion capture equipment.

[0023] Common technology breakthroughs: Dynamic safety boundary control detects the velocity and range of the center of gravity trajectory in real time, actively intervenes in balance by lifting the pedal, and has a short response time. The nonlinear compensation model is based on neural network training, adapts to the pressure distribution of users of different weights, and improves control accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the device structure of the dynamic compensation multi-modal interactive control device based on the center of gravity pressure field of the present invention.

[0025] Figure 2 It is the overall block diagram of the hardware system of the dynamic compensation multi-modal interactive control device based on the center of gravity pressure field of the present invention.

[0026] Figure 3 It is a core algorithm flow chart of the dynamic compensation multi-modal interactive control method based on the center of gravity pressure field of the present invention.

[0027] Figure 4It is a schematic diagram of dynamic compensation of the multi-modal interactive control device based on the center of gravity pressure field of the present invention.

[0028] Figure numerals: 1 - pedal assembly, 11 - pressure sensor array, 2 - bottom box. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings of the specification. Obviously, the described implementation is only a part of the implementation of the present invention, not all of the implementations. Based on the implementation of the present invention, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0030] A dynamic compensation multi-modal interactive control device based on center of gravity pressure field, such as Figure 1 As shown, it includes a pedal assembly 1 and a bottom box 2, and an embedded control unit and a communication module are arranged inside the bottom box 2. The upper surface of the pedal assembly 1 contacts the sole of the operator (the operator is standing or sitting), and is provided with a pressure sensor array 11 for real-time collection of the operator's sole pressure distribution data; the embedded control unit is electrically connected to the pressure sensor array, and is used to process the collected pressure distribution data, calculate the trajectory of the center of gravity of the human body, and generate corresponding control signals according to the trajectory of the center of gravity; the communication module is electrically connected to the embedded control unit, and is used to send the control signal to an external device, and the external device includes a multi-degree-of-freedom robot or a VR interactive device. The pressure sensor array 11 is a 2×2 pressure sensor array, with a total of 4 piezoresistive sensors (with integrated digital-to-analog conversion modules), and a single sensor has a range of 40kg. The embedded control unit includes an ESP32 microcontroller, which supports WIFI and Bluetooth communications.

[0031] The control method of the above device comprises the following steps: 1. Pressure data acquisition and preprocessing The ESP32 microcontroller reads the sensor data in real time and processes it as follows: Data verification and filtering (median filtering and Kalman filtering combined to eliminate random interference).

[0032] Normalization processing, determine the coordinates of the center of gravity based on real-time measurement data:

[0033] in, F i is the pressure value measured by the i-th sensor, ( X i , Y i ) is the position coordinate of the i-th sensor.

[0034] 2. Center of gravity displacement mapping The operator's intention is determined by calculating the offset vector between the current center of gravity position and the initial position (the operator's standing balance position): The center of gravity moves forward beyond the set threshold, and the robot / virtual character moves forward; If the center of gravity moves backward beyond the set threshold, the robot / virtual character moves backward; When the center of gravity deviates to the left or right and exceeds the set threshold, the control device moves to the left or right; When the pressure is distributed diagonally, the control device turns in the corresponding direction; specifically, when the pressure of the left front pressure sensor and the right rear pressure sensor is greater than the threshold, it turns left; when the pressure of the right front pressure sensor and the left rear pressure sensor is greater than the threshold, it turns right.

[0035] Fuzzy logic judgment is performed based on the threshold to avoid misjudgment of slight shaking.

[0036] 3. Real-time control signal generation The speed gradient is set according to the degree of center of gravity deviation, and the control signal is generated in real time. It is sent to the robot chassis or VR interaction system via wireless communication (Bluetooth / Wi-Fi), and the control target responds to the operator's body movements in real time. Example 2

[0037] A dynamic compensation multimodal interactive control device based on a center of gravity pressure field comprises: a pedal assembly, the upper surface of which contacts the sole of an operator and is provided with a pressure sensor array for collecting the operator's sole pressure distribution data in real time; an embedded control unit, electrically connected to the pressure sensor array, for processing the collected pressure distribution data, calculating the trajectory of the center of gravity of the human body, and generating a corresponding control signal according to the trajectory of the center of gravity; a communication module, electrically connected to the embedded control unit, for sending the control signal to an external device, the external device comprising a multi-degree-of-freedom robot or a VR interactive device; a dynamic compensation mechanism, the dynamic compensation mechanism is electrically connected to the embedded control unit, for dynamically compensating and adjusting the pedal assembly according to the control signal to maintain the balance of the operator.

[0038] The pedal assembly is a dual-axis movable pedal, the main body is made of aluminum alloy, and the rotary joint adopts a harmonic reducer. An angle sensor is embedded in the pedal axis to detect the pedal inclination in real time. The pressure sensor array is a 2×2 resistive film sensor array, with a total of 4 resistive film sensors (with integrated digital-to-analog conversion modules), and a single sensor has a range of 40kg. The embedded control unit includes an ESP32 microcontroller, which supports WIFI and Bluetooth communication. The dynamic compensation mechanism includes two high-response brushless DC motors (BLDC) to drive the harmonic reducer; a shock-absorbing structure is installed at the power output end to reduce mechanical vibration and noise and improve user experience. The dynamic compensation mechanism analyzes the center of gravity offset and predicted trajectory in real time. When it is detected that the user's center of gravity offset rate or amplitude exceeds the safety threshold, the motor is controlled to drive the pedal to actively lift in the opposite direction at a corresponding angle; the feedback response time is controlled within 100 milliseconds to effectively prevent the user from falling and instability.

[0039] The control method of the above device comprises the following steps: 1. Pressure distribution data acquisition and preprocessing The ESP32 microcontroller reads the sensor data in real time and processes it as follows: The signal after digital-to-analog conversion is collected in real time by a multi-channel ADC module; Signal preprocessing uses Gaussian filtering and outlier filtering to improve data stability; According to the two-dimensional distribution of pressure values, the center of gravity trajectory position with millimeter-level accuracy is obtained by weighted average method:

[0040] in, F i is the pressure value measured by the i-th sensor, ( X i , Y i ) is the position coordinate of the i-th sensor.

[0041] 2. Dynamic center of gravity trajectory prediction and safety control The recurrent neural network (RNN) model is used to predict the future short-term (within 0.5 seconds) center of gravity trajectory: Input layer: center of gravity trajectory and speed in the past second; Hidden layer: Long short-term memory (LSTM) units; Output layer: predict the center of gravity trajectory within the next 0.5 seconds; When the predicted trajectory speed exceeds the safety threshold or is out of the stable range, the pedal actively triggers the mechanical structure to perform real-time dynamic compensation, lifting the pedal in the opposite direction, thereby actively adjusting the operator's posture and preventing instability.

[0042] 3. Non-linear weight compensation control In view of the weight differences among different operators, a neural network model based on a multi-layer perceptron (MLP) is used: Establish a labeled data set, including the pressure distribution and actual control effects of 40kg-120kg users; Input layer: pressure values ​​of sensor array; Hidden layer: two-layer neural network structure, activation function uses ReLU; Output layer: real-time estimated center of gravity offset vector; After training is completed, individual differences among users can be compensated in real time, improving the accuracy and response sensitivity of equipment control by different operators and solving the problem of poor adaptability of linear models.

[0043] 4. Multimodal interaction architecture design and mode switching Software-defined interactive modes, with fast switching through the built-in mode selection algorithm of the ESP32 microcontroller.

[0044] Robot remote control mode: The pressure field data is mapped into chassis movement control instructions in real time; The degree of pedal tilt maps the chassis speed gradient; The other degrees of freedom are operated independently through external devices (such as handles or exoskeletons).

[0045] VR Interaction Mode: Real-time center of gravity trajectory data drives the movement of virtual characters; Enhance immersion, reduce dizziness, and coordinate real-time acceleration data with visual feedback; Users can move freely in the virtual world without complicated wear.

[0046] 5. Center of gravity displacement mapping The operator's intention is determined by calculating the offset vector between the current center of gravity position and the initial position (the operator's standing balance position): The center of gravity moves forward beyond the set threshold, and the robot / virtual character moves forward; If the center of gravity moves backward beyond the set threshold, the robot / virtual character moves backward; When the center of gravity deviates to the left or right and exceeds the set threshold, the control device moves to the left or right; When the pressure is distributed diagonally, the control device turns in the corresponding direction; specifically, when the pressure of the left front pressure sensor and the right rear pressure sensor is greater than the threshold, it turns left; when the pressure of the right front pressure sensor and the left rear pressure sensor is greater than the threshold, it turns right.

[0047] Fuzzy logic judgment is performed based on the threshold to avoid misjudgment of slight shaking.

[0048] 6. Real-time control signal generation The speed gradient is set according to the degree of center of gravity deviation, and the control signal is generated in real time. It is sent to the robot chassis or VR interaction system via wireless communication (Bluetooth / Wi-Fi), and the control target responds to the operator's body movements in real time.

[0049] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The protection scope of the present invention is defined by the claims and their equivalent technical solutions.

Claims

1. A dynamic compensation multi-modal interactive control device based on the center of gravity pressure field, characterized in that: include: A pedal assembly, the upper surface of which contacts the sole of the operator's foot, and the pedal assembly is provided with a pressure sensor array for collecting the operator's sole pressure distribution data in real time; An embedded control unit, electrically connected to the pressure sensor array, for processing the collected pressure distribution data, calculating the trajectory of the center of gravity of the human body, and generating a corresponding control signal according to the trajectory of the center of gravity; A communication module is electrically connected to the embedded control unit and is used to send the control signal to an external device, wherein the external device includes a multi-degree-of-freedom robot or a VR interactive device.

2. The dynamic compensation multi-modal interactive control device based on the center of gravity pressure field according to claim 1 is characterized in that: It also includes a dynamic compensation mechanism, which is electrically connected to the embedded control unit and is used to perform dynamic compensation adjustment on the pedal assembly according to the control signal to maintain the balance of the operator.

3. The dynamic compensation multi-modal interactive control device based on the center of gravity pressure field according to claim 1 is characterized in that: The pressure sensor array includes a plurality of piezoresistive sensors or resistive film sensors evenly distributed at four corners of the pedal assembly.

4. The dynamic compensation multi-modal interactive control device based on the center of gravity pressure field according to claim 2 is characterized in that: The dynamic compensation mechanism includes a biaxial movable pedal, a harmonic reducer, a DC brushless motor and an angle sensor. The biaxial movable pedal is connected to the DC brushless motor through the harmonic reducer. The angle sensor is embedded in the axis of the pedal and is used to detect the inclination angle of the pedal in real time.

5. The dynamic compensation multi-modal interactive control device based on the center of gravity pressure field according to claim 1 is characterized in that: The embedded control unit has a built-in mode selection algorithm for realizing rapid switching between the robot remote control mode and the VR interaction mode.

6. A multi-degree-of-freedom robot, characterized in that: It comprises the dynamic compensation multimodal interactive control device based on the center of gravity pressure field as described in any one of claims 1 to 5, wherein the multi-degree-of-freedom robot receives the control signal sent by the control device through the communication module to realize the coordinated action control of the chassis movement and other degrees of freedom.

7. A VR device, characterized in that: The invention comprises a multimodal interactive control device based on a center of gravity pressure field and a dynamic compensation device as described in any one of claims 1 to 5, wherein the VR device receives a control signal sent by the control device through the communication module to realize displacement control of a virtual character.

8. A dynamic compensation multi-modal interactive control method based on center of gravity pressure field, characterized in that: The dynamic compensation multimodal interactive control device based on the center of gravity pressure field according to any one of claims 1 to 5 comprises the following steps: Pressure data collection: The pressure sensor array provided on the pedal assembly collects the operator's foot pressure distribution data in real time; Data processing and center of gravity trajectory calculation: The collected pressure distribution data is transmitted to the embedded control unit for processing, and the center of gravity trajectory of the human body is calculated, including the center of gravity position coordinates and its change trend; Control signal generation: Based on the center of gravity trajectory, a corresponding control signal is generated according to a preset mapping relationship, and the control signal is used to control the action of an external device.

9. The dynamic compensation multi-modal interactive control method based on the center of gravity pressure field according to claim 8 is characterized in that: It also includes dynamic compensation adjustment: the operator's balance state is monitored in real time according to the center of gravity trajectory, and when it is detected that the operator's center of gravity deviation rate or amplitude exceeds a safety threshold, the pedal assembly is dynamically compensated and adjusted through a dynamic compensation mechanism to maintain the operator's balance.

10. The dynamic compensation multi-modal interactive control method based on gravity pressure field according to claim 8, characterized in that: It also includes mode switching: through the built-in mode selection algorithm of the embedded control unit, fast switching between robot remote control mode and VR interaction mode can be achieved.

11. The dynamic compensation multi-modal interactive control method based on gravity pressure field according to claim 8, characterized in that: In the data processing and center of gravity trajectory calculation steps, one or more combinations of median filtering, Kalman filtering, Gaussian filtering and outlier filtering are used to pre-process the pressure distribution data to improve the accuracy and stability of the data.

12. The dynamic compensation multi-modal interactive control method based on center of gravity pressure field according to claim 8, characterized in that: It also includes nonlinear weight compensation: a neural network model based on a multi-layer perceptron is used to compensate for differences in pressure distribution among users of different weights through training with labeled data sets.

Images