Intelligent motion capture system and method of multi-mode sensor

Abstract

The invention discloses an intelligent motion capture system and method of a multi-mode sensor, and the method comprises the steps: creating and installing links among an optical module, an inertia module, a touch module, a synchronization core module and a to-be-detected object; sending a synchronous starting pulse signal based on the synchronous core module, setting the synchronous starting pulse signal as a clock reference starting point, and starting the optical module, the inertia module and the touch module based on the clock reference starting point; performing data acquisition on a to-be-measured object based on the optical module, the inertia module and the touch module to obtain an object acquisition information data set; and creating an acquisition original timestamp based on the synchronous start pulse signal, creating a sensor network delay model to dynamically adjust the acquisition original timestamp to obtain a calibration timestamp, and analyzing and correcting the time offset of the object data acquisition information. The method and the device have the effect of improving the accuracy of intelligent motion capture of the multi-mode sensor.

Description

Technical Field

[0001] The present application relates to the field of motion capture technology, and in particular to an intelligent motion capture system and method using a multimodal sensor. Background Art

[0002] Currently, motion capture refers to the technology used to record and process the movements of people or other objects. Motion capture is widely used in a variety of fields, including military, entertainment, sports, medicine, and robotics. It is a key research method in ergonomics and biomechanics. It also holds significant application value in a wide range of promising fields, including robotic control, virtual reality, and medical rehabilitation. Therefore, motion capture based on multimodal sensors is crucial.

[0003] The existing intelligent motion capture methods of multimodal sensors have large errors in sensor synchronization. In application scenarios with high requirements for real-time interaction, such as real-time feedback in virtual reality and instant control of robots, such synchronization errors will lead to untimely system response, poor user experience, and may even cause operational errors or safety accidents. In addition, the data formats and characteristics generated by different types of sensors are different. To effectively fuse these heterogeneous data, there are huge algorithmic challenges. Existing algorithms often find it difficult to ensure real-time while taking into account the accuracy of data fusion, and cannot meet the needs of high-precision, low-latency data processing in actual application scenarios, resulting in low accuracy of multimodal motion capture, which still needs improvement. Summary of the Invention

[0004] In order to improve the accuracy of intelligent motion capture using a multimodal sensor, the present application provides an intelligent motion capture system and method using a multimodal sensor.

[0005] In a first aspect, the present application provides a method for intelligent motion capture using a multimodal sensor, which adopts the following technical solutions: An intelligent motion capture method using a multimodal sensor comprises the following steps: Create links between the optical module, inertial module, tactile module, synchronization core module and the object to be measured, and install them; Sending a synchronous start pulse signal based on the synchronous core module, setting the synchronous start pulse signal as the clock reference starting point, and starting the optical module, the inertial module, and the tactile module based on the clock reference starting point; Based on the optical module, inertial module, and tactile module, data of the object to be measured is collected to obtain an object collection information data set; Create the original acquisition timestamp based on the synchronous start pulse signal, create a sensor network delay model to dynamically adjust the original acquisition timestamp to obtain a calibrated timestamp, analyze the time offset of the object data acquisition information and make corrections; Adjust the data frame rates of different information in the object acquisition information data set to be consistent, and then adjust the data phases to be consistent to obtain the object information adjustment data set, upload the object information adjustment data set to the ring buffer for storage, and output a data stream alignment completion signal; After receiving the data stream alignment completion signal, multimodal data fusion is performed based on the object frequency modulation information data set to create an object skeleton model of the object to be tested, and the tactile pressure of the object to be tested detected by the tactile module is mapped to the object skeleton model of the object to be tested.

[0006] Preferably, an optical module, an inertial module, a tactile module and a synchronous core module are obtained, and signal connection links are established between the optical module, the inertial module, the tactile module, the synchronous core module and the object to be measured respectively; The optical module is composed of a plurality of infrared optical cameras, and the plurality of infrared optical cameras are arranged in an array around the object to be measured and installed; The inertial module is composed of multiple IMU (Inertial Measurement Unit) nodes, which identify the key parts of the object to be measured and install IMU nodes on the key parts of the object to be measured; The tactile module is composed of a plurality of flexible piezoresistive sensors, which are arranged in an array and covered on the surface of the object to be measured; The synchronous core module includes a global clock generator, which is driven by an FPGA (Field Programmable Gate Array) to establish signal connection links between the global clock generator and the optical module, the inertial module, and the tactile module.

[0007] Preferably, based on the device properties and installation positions of each component in the optical module, the inertial module, and the tactile module, the delay compensation parameter information of each component is pre-determined, and the fixed delay of each component in the optical module, the inertial module, and the tactile module is compensated based on the delay compensation parameter information of each component, and a delay compensation completion result is output after compensation; Sending a synchronous start pulse signal to the optical module, the inertial module, and the tactile module simultaneously based on the global clock generator; When the delay compensation completion result is received and when the optical module, inertial module, and tactile module receive the synchronous start pulse signal, the synchronous start pulse signal is marked as the clock reference starting point of the optical module, inertial module, and tactile module, and the start operation of the optical module, inertial module, and tactile module is controlled based on the clock reference starting point.

[0008] Preferably, each infrared optical camera in the optical module continuously captures an image of the object to be measured to obtain object captured image information, and the image capturing time of each infrared optical camera is recorded to obtain image information capturing time; The 9-axis attitude data of the object to be measured is collected in real time according to each IMU node in the inertial module to obtain the object attitude data information, and the data collection time of each IMU node is recorded to obtain the attitude information collection time; According to the pressure value of the contact surface of the object to be measured detected by each flexible piezoresistive sensor in the tactile module in real time, a real-time contact pressure value is obtained, the real-time contact pressure value is converted into object contact pressure information, and the data collection time of each flexible piezoresistive sensor is recorded to obtain the tactile information collection time; The object captured image information, the image information acquisition time, the object posture data information, the posture information acquisition time, the object contact pressure information and the tactile information acquisition time are combined to form an object acquisition information data set of the object to be measured.

[0009] Preferably, the original timestamp of the acquisition is created according to the sending moment of the synchronous start pulse signal; A sensor network delay model is created, and based on the sensor network delay model, the original timestamps collected are dynamically adjusted to obtain calibrated timestamps according to the delay conditions of the optical module, the inertial module, and the tactile module.

[0010] Preferably, the time offsets of the optical module and the inertial module are calculated based on the Kalman filter algorithm to obtain the optical information acquisition time offset and the posture information acquisition time offset; Correcting the image information acquisition time based on the optical information acquisition time offset; The posture information collection time is corrected based on the posture information collection time offset.

[0011] Preferably, the optical data frame rate when the infrared optical camera performs data acquisition to obtain the image information of the object is recorded, and the tactile data frequency when the flexible piezoresistive sensor performs data acquisition to obtain the posture data information of the object is recorded; Based on a downsampling algorithm, the sampling frequency of the tactile data frequency is adjusted to be consistent with the sampling frequency of the consistent optical data frame rate, thereby obtaining tactile downsampled data information; Based on the interpolation compensation phase difference, adjusting the data phase of the tactile downsampled data information to obtain tactile data alignment adjustment information, adjusting the data phase of the object captured image information to obtain optical data alignment adjustment information, and outputting a data phase adjustment completion result, wherein the tactile data alignment adjustment information and the optical data alignment adjustment information are combined to form an object information adjustment data set; Obtain the ring buffer. When the data phase adjustment completion result is received, upload the object information adjustment data set to the ring buffer for storage. After the upload is completed, output the data stream alignment completion signal.

[0012] Preferably, whether the active markers provided in the infrared optical camera are blocked is detected, and if blocked, the optical data alignment adjustment information is corrected in real time based on the object posture data information; Based on the neural network algorithm and combined with the optical data alignment adjustment information, the accurate posture in the object posture data information is determined to obtain the accurate posture information of the object to be measured, and the accumulated error of the IMU node is reversely corrected based on the accurate posture information of the object to be measured; Based on concurrent operation of lock-free queues, an object skeleton model of the object to be tested is created, the deformation coefficient of the object to be tested is pre-set, the pressure distribution of the object to be tested is judged based on the tactile data alignment adjustment information to obtain the pressure distribution matrix, the contact force vector of the object to be tested is determined based on the deformation coefficient of the object to be tested and the pressure distribution matrix, and the tactile pressure detected in the tactile module is mapped to the object skeleton model of the object to be tested based on the contact force vector of the object to be tested.

[0013] In a second aspect, the present application provides an intelligent motion capture system with a multimodal sensor, which adopts the following technical solutions: An intelligent motion capture system of a multimodal sensor, comprising: a hardware installation module configured to establish a link between the optical module, the inertial module, the tactile module, and the synchronization core module and the object to be measured, and to perform installation; a data initialization module configured to send a synchronization start pulse signal based on the synchronization core module, set the synchronization start pulse signal as a clock reference starting point, and start the optical module, the inertial module, and the tactile module based on the clock reference starting point; A data acquisition module is configured to collect data from the object to be measured based on the optical module, the inertial module, and the tactile module to obtain an object collection information data set; A data calibration and correction module is configured to create an original acquisition time stamp based on a synchronous start pulse signal, create a sensor network delay model to dynamically adjust the original acquisition time stamp to obtain a calibration time stamp, analyze the time offset of the object data acquisition information and make corrections; a data stream alignment module configured to align the data frame rates of different information in the object acquisition information dataset, and then align the data phases to obtain an object information adjustment dataset, upload the object information adjustment dataset to a ring buffer for storage, and output a data stream alignment completion signal; The data fusion module is configured to perform multimodal data fusion based on the object frequency modulation information data set after receiving the data stream alignment completion signal, create an object skeleton model of the object to be tested, and map the tactile pressure of the object to be tested detected by the tactile module to the object skeleton model of the object to be tested.

[0014] In summary, this application includes at least one of the following beneficial technical effects: 1. A global clock generator distributes microsecond-level synchronization pulses to each component. Based on the precise clock signal, this ensures that in subsequent data acquisition, the infrared optical cameras in the optical module, the IMU nodes in the inertial module, and the flexible piezoresistive sensors in the tactile module all collect data under the same time reference. This fundamentally solves the problem of asynchronous data acquisition caused by clock differences between different sensors, improves the quality of data acquisition, provides data support for subsequent motion capture data fusion, and thus improves the accuracy of intelligent motion capture using multimodal sensors. 2. By creating a skeleton model of the object to be tested, the deformation coefficient of the object to be tested is pre-set, and the pressure distribution of the object to be tested is determined based on the tactile data alignment adjustment information to obtain a pressure distribution matrix. The contact force vector of the object to be tested is determined based on the deformation coefficient of the object to be tested and the pressure distribution matrix. Based on the contact force vector of the object to be tested, the tactile pressure detected in the tactile module is mapped to the skeleton model of the object to be tested. This realizes the joint analysis of force perception and action, making the operation intention and action force of the object to be tested clearer and more transparent, further improving the accuracy of intelligent motion capture of the multimodal sensor; 3. With the help of the design of ring buffer and lock-free queue, the data processing efficiency of the system is greatly improved. While ensuring data synchronization and fusion accuracy, it meets the real-time requirements, enabling the system to respond to motion changes in various application scenarios in a timely manner, further improving the accuracy of intelligent motion capture of multimodal sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of an intelligent motion capture method using a multimodal sensor, which is mainly embodied in this embodiment; Figure 2 This is a schematic diagram of the modules of the intelligent motion capture system using multimodal sensors, which is mainly embodied in this embodiment.

[0016] Figure numerals: 1. Hardware installation module; 2. Data initialization module; 3. Data acquisition module; 4. Data calibration and correction module; 5. Data stream alignment module; 6. Data fusion module. DETAILED DESCRIPTION

[0017] The present application is further described in detail below with reference to the accompanying drawings.

[0018] The embodiments of the present application disclose an intelligent motion capture method using a multimodal sensor.

[0019] An intelligent motion capture method using a multimodal sensor comprises the following steps: Reference Figure 1 Step S1: Create a link between the optical module, inertial module, tactile module, synchronization core module and the object to be measured, and install them. Step S1 specifically includes: Obtain an optical module, an inertial module, a tactile module and a synchronous core module, and establish signal connection links between the optical module, the inertial module, the tactile module, the synchronous core module and the object to be measured.

[0020] The optical module is composed of a plurality of infrared optical cameras, which are arranged in an array around the object to be measured and installed.

[0021] The infrared optical camera in the embodiments of this application utilizes a high-frame-rate infrared optical camera array with a camera frame rate of ≥200Hz. This high-frame-rate infrared optical camera can rapidly capture moving images of the object being measured, ensuring sufficient image data is acquired in a short period of time. Furthermore, it is equipped with active markers that can actively emit light or reflect light of specific wavelengths in an optical environment, improving their visibility and recognition accuracy in complex backgrounds and providing a reliable foundation for subsequent optical data processing.

[0022] The inertial module consists of multiple IMU nodes, which identify the key parts of the object to be measured and install IMU nodes at the key parts of the object to be measured.

[0023] For example, if the object to be measured is a human body, IMU nodes can be installed at various joints of the human body to detect and identify the human body's posture. The IMU nodes in the embodiments of this application can output 9-axis posture data, including information such as three-axis acceleration, three-axis angular velocity, and three-axis magnetic field strength. This comprehensively describes the changes in the object's posture in space and provides rich data support for posture estimation.

[0024] The tactile module consists of multiple flexible piezoresistive sensors, which are arranged in an array and covered on the surface of the object to be measured.

[0025] The sampling rate of the flexible piezoresistive sensor in the embodiment of the present application is ≥1kHz, which can quickly sense pressure changes on the contact surface, convert the pressure signal into an electrical signal for collection and processing, and provide a high-resolution data source for tactile perception.

[0026] The synchronous core module includes a global clock generator, which is driven by FPGA and establishes signal connection links between the global clock generator and the optical module, the inertial module, and the tactile module.

[0027] Reference Figure 1 In step S2, a synchronous start pulse signal is sent based on the synchronous core module, the synchronous start pulse signal is set as the clock reference starting point, and the optical module, inertial module, and tactile module are started based on the clock reference starting point. Step S2 specifically includes: Based on the device properties and installation positions of each component in the optical module, inertial module, and tactile module, the delay compensation parameter information of each component is determined in advance. Based on the delay compensation parameter information of each component, the fixed delay of each component in the optical module, inertial module, and tactile module is compensated, and the delay compensation completion result is output after compensation.

[0028] The delay compensation parameter information of each component is used to compensate for possible fixed delays in the initial stage of each component to ensure the synchronization of data at the acquisition source.

[0029] Based on the global clock generator, a synchronous start pulse signal is sent to the optical module, the inertial module, and the tactile module at the same time.

[0030] When the delay compensation completion result is received and when the optical module, inertial module, and tactile module receive the synchronous start pulse signal, the synchronous start pulse signal is marked as the clock reference starting point of the optical module, inertial module, and tactile module, and the start operation of the optical module, inertial module, and tactile module is controlled based on the clock reference starting point.

[0031] The global clock generator in the embodiments of this application can distribute microsecond-level synchronization pulses to each component. This precise clock signal ensures that subsequent data acquisition, including the infrared cameras in the optical module, the IMU nodes in the inertial module, and the flexible piezoresistive sensors in the tactile module, is performed under the same time reference, fundamentally resolving the issue of asynchronous data acquisition caused by clock differences between different sensors.

[0032] Reference Figure 1 In step S3, data is collected from the object to be measured based on the optical module, the inertial module, and the tactile module to obtain an object collection information data set. Step S3 specifically includes: The infrared optical cameras in the optical module continuously capture images of the object to be measured to obtain object captured image information, and the image capturing time of each infrared optical camera is recorded to obtain image information capturing time.

[0033] The 9-axis attitude data of the object to be measured is collected in real time by each IMU node in the inertial module to obtain the object attitude data information, and the data collection time of each IMU node is recorded to obtain the attitude information collection time.

[0034] The real-time contact pressure value is obtained by real-time detection of the pressure value of the contact surface of the object to be measured by each flexible piezoresistive sensor in the tactile module, and the real-time contact pressure value is converted into object contact pressure information. The data acquisition time of each flexible piezoresistive sensor is recorded to obtain the tactile information acquisition time.

[0035] The object captured image information, image information acquisition time, object posture data information, posture information acquisition time, object contact pressure information and tactile information acquisition time are combined to form an object acquisition information data set of the object to be measured.

[0036] Reference Figure 1 In step S4, the original acquisition timestamp is created based on the synchronous start pulse signal, a sensor network delay model is created to dynamically adjust the original acquisition timestamp to obtain a calibration timestamp, and the time offset of the object data acquisition information is analyzed and corrected. Step S4 specifically includes: Create the acquisition original timestamp according to the sending moment of the synchronous start pulse signal.

[0037] A sensor network delay model is created, and based on the delay of the optical module, inertial module, and tactile module in the sensor network delay model, the original timestamp is dynamically adjusted to obtain a calibrated timestamp.

[0038] Specifically, the sensor network delay model is based on the software timestamp calibration algorithm based on dynamic network delay prediction, which refers to the sensor network delay calculation formula Where T2 represents the calibration timestamp, T1 represents the original acquisition timestamp, α is the dynamic attenuation factor, t2 is the current time, and t1 is the time corresponding to the synchronization start pulse signal. Through the sensor network delay model, the timestamp can be dynamically adjusted according to the actual delay of the sensor network, making it more accurately reflect the actual acquisition time of the data.

[0039] Step S4 further includes: Based on the Kalman filter algorithm, the time offsets of the optical module and the inertial module are calculated respectively to obtain the optical information acquisition time offset and the posture information acquisition time offset.

[0040] The image information acquisition time is corrected based on the optical information acquisition time offset.

[0041] The embodiment of the present application improves the accuracy of the image information acquisition time by correcting the image information acquisition time, facilitates the matching degree in the time dimension during subsequent data fusion, and improves the accuracy of data fusion.

[0042] The attitude information collection time is corrected based on the attitude information collection time offset.

[0043] The embodiment of the present application improves the accuracy of the posture information collection time by correcting the posture information collection time, facilitates the matching degree in the time dimension during subsequent data fusion, and improves the accuracy of data fusion.

[0044] Reference Figure 1 In step S5, the data frame rates of different information in the object acquisition information data set are adjusted to be consistent, and the data phases are adjusted to be consistent to obtain the object information adjustment data set. The object information adjustment data set is uploaded to the ring buffer for storage and a data stream alignment completion signal is output. Step S5 specifically includes: The optical data frame rate when the infrared optical camera is used to collect data to obtain the image information of the object is recorded, and the tactile data frequency when the flexible piezoresistive sensor is used to collect data to obtain the posture data information of the object is recorded.

[0045] Based on a downsampling algorithm, the sampling frequency of the tactile data frequency is adjusted to be consistent with the sampling frequency of the consistent optical data frame rate, thereby obtaining tactile downsampling data information.

[0046] In the embodiment of the present application, the tactile data frequency is downsampled to the sampling frequency of the optical data frame rate to ensure that the sampling frequencies of data in different modalities are consistent for subsequent data processing.

[0047] Based on interpolation compensation phase difference, the data phase of the tactile downsampling data information is adjusted to obtain tactile data alignment adjustment information, the data phase of the object captured image information is adjusted to obtain optical data alignment adjustment information, and the data phase adjustment completion result is output, wherein the tactile data alignment adjustment information and the optical data alignment adjustment information are combined to form an object information adjustment data set.

[0048] In the embodiment of the present application, the phase difference is compensated by interpolation, thereby further eliminating the problem of data phase inconsistency that may be caused by the adjustment of the sampling frequency.

[0049] Obtain the ring buffer. When the data phase adjustment completion result is received, upload the object information adjustment data set to the ring buffer for storage. After the upload is completed, output the data stream alignment completion signal.

[0050] In actual application, the ring buffer has efficient data storage and reading characteristics, can realize fast data exchange and synchronization in a multi-threaded environment, and avoids the performance overhead and data competition problems brought by the traditional locking mechanism. In the embodiment of the present application, lock-free read and write alignment of multimodal data is achieved by using a ring buffer.

[0051] Reference Figure 1 In step S6, after receiving the data stream alignment completion signal, multimodal data fusion is performed based on the object frequency modulation information data set to create an object skeleton model of the object to be tested, and the tactile pressure of the object to be tested detected by the tactile module is mapped to the object skeleton model of the object to be tested. Step S6 specifically includes: Detect whether the active markers equipped in the infrared optical camera are blocked. If blocked, perform real-time error correction on the optical data alignment adjustment information based on the object posture data information.

[0052] Based on the neural network algorithm and combined with the optical data alignment adjustment information, the accurate posture in the object posture data information is judged to obtain the accurate posture information of the object to be measured. Based on the accurate posture information of the object to be measured, the accumulated error of the IMU node is reversely corrected, thereby improving the accuracy of data fusion.

[0053] Based on concurrent operation of lock-free queues, an object skeleton model of the object to be tested is created, the deformation coefficient of the object to be tested is pre-set, the pressure distribution of the object to be tested is judged based on the tactile data alignment adjustment information to obtain the pressure distribution matrix, the contact force vector of the object to be tested is determined based on the deformation coefficient of the object to be tested and the pressure distribution matrix, and the tactile pressure detected in the tactile module is mapped to the object skeleton model of the object to be tested based on the contact force vector of the object to be tested.

[0054] Specifically, the tactile pressure distribution is mapped to the skeletal model using the formula F = K * P, where F represents the contact force vector, K is the flexibility coefficient (deformation coefficient), and P is the pressure distribution matrix. This approach enables a joint analysis of force perception and action, enabling the system to more accurately understand the user's operational intent and action force.

[0055] In the embodiment of the present application, by mapping tactile pressure to the object skeleton model of the object to be tested, a joint analysis of force perception and action is achieved, making the operational intention and action force of the object to be tested clearer.

[0056] The lock-free queue in the embodiments of this application is a concurrent data structure that allows multiple threads to perform concurrent operations without locking. Traditional queues typically use mutexes to achieve thread-safe operations, but mutexes can cause contention and performance bottlenecks in high-concurrency situations. To avoid the use of locks, lock-free queues use a concurrency algorithm based on atomic operations. The design goal of lock-free queues is to provide high-performance concurrent operations while maintaining thread safety.

[0057] The embodiments of this application utilize a ring buffer and lock-free queue design to support data throughput exceeding 200Hz. This efficient data transmission and processing mechanism can meet the large-scale data processing requirements of real-time motion capture systems, ensuring that data congestion or loss does not occur during high-speed operation.

[0058] The embodiment of the present application also discloses an intelligent motion capture system of a multimodal sensor.

[0059] Reference Figure 2 , an intelligent motion capture system of multimodal sensors, comprising: The hardware installation module is configured to establish a link between the optical module, the inertial module, the tactile module, and the synchronization core module and the object to be measured, and to perform installation.

[0060] The data initialization module is configured to send a synchronization start pulse signal based on the synchronization core module, set the synchronization start pulse signal as the clock reference starting point, and start the optical module, the inertial module, and the tactile module based on the clock reference starting point.

[0061] The data acquisition module is configured to collect data from the object to be measured based on the optical module, the inertial module, and the tactile module to obtain an object acquisition information data set.

[0062] The data calibration and correction module is configured to create an original acquisition time stamp based on a synchronous start pulse signal, create a sensor network delay model to dynamically adjust the original acquisition time stamp to obtain a calibration time stamp, analyze the time offset of the object data acquisition information and make corrections.

[0063] The data stream alignment module is configured to adjust the data frame rates of different information in the object acquisition information data set to be consistent, and then adjust the data phase to be consistent to obtain the object information adjustment data set, upload the object information adjustment data set to the ring buffer for storage and output a data stream alignment completion signal.

[0064] The data fusion module is configured to perform multimodal data fusion based on the object frequency modulation information data set after receiving the data stream alignment completion signal, create an object skeleton model of the object to be tested, and map the tactile pressure of the object to be tested detected by the tactile module to the object skeleton model of the object to be tested.

[0065] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. An intelligent motion capture method using a multimodal sensor, characterized in that: The following steps are involved: Create links between the optical module, inertial module, tactile module, synchronization core module and the object to be measured, and install them; Sending a synchronous start pulse signal based on the synchronous core module, setting the synchronous start pulse signal as the clock reference starting point, and starting the optical module, the inertial module, and the tactile module based on the clock reference starting point; Based on the optical module, inertial module, and tactile module, data of the object to be measured is collected to obtain an object collection information data set; Create the original acquisition timestamp based on the synchronous start pulse signal, create a sensor network delay model to dynamically adjust the original acquisition timestamp to obtain a calibrated timestamp, analyze the time offset of the object data acquisition information and make corrections; Adjust the data frame rates of different information in the object acquisition information data set to be consistent, and then adjust the data phases to be consistent to obtain the object information adjustment data set, upload the object information adjustment data set to the ring buffer for storage, and output a data stream alignment completion signal; After receiving the data stream alignment completion signal, multimodal data fusion is performed based on the object frequency modulation information data set to create an object skeleton model of the object to be tested, and the tactile pressure of the object to be tested detected by the tactile module is mapped to the object skeleton model of the object to be tested.

2. The intelligent motion capture method of a multimodal sensor according to claim 1, characterized in that: The steps for establishing links between the optical module, inertial module, tactile module, synchronization core module and the object to be measured and installing them include: Acquire an optical module, an inertial module, a tactile module, and a synchronous core module, and respectively establish signal connection links between the optical module, the inertial module, the tactile module, the synchronous core module, and the object to be measured; The optical module is composed of a plurality of infrared optical cameras, and the plurality of infrared optical cameras are arranged in an array around the object to be measured and installed; The inertial module is composed of multiple IMU nodes, which identify the key parts of the object to be measured and install IMU nodes at the key parts of the object to be measured; The tactile module is composed of a plurality of flexible piezoresistive sensors, which are arranged in an array and covered on the surface of the object to be measured; The synchronous core module includes a global clock generator, which is driven by an FPGA and establishes signal connection links between the global clock generator and the optical module, the inertial module, and the tactile module.

3. The intelligent motion capture method of a multimodal sensor according to claim 2, characterized in that: The steps of sending a synchronous start pulse signal based on the synchronous core module, setting the synchronous start pulse signal as a clock reference starting point, and starting the optical module, the inertial module, and the tactile module based on the clock reference starting point specifically include: Determining delay compensation parameter information of each component in advance based on the device properties and installation positions of each component in the optical module, the inertial module, and the tactile module, compensating for the fixed delay of each component in the optical module, the inertial module, and the tactile module based on the delay compensation parameter information of each component, and outputting a delay compensation completion result after compensation; Sending a synchronous start pulse signal to the optical module, the inertial module, and the tactile module simultaneously based on the global clock generator; When the delay compensation completion result is received and when the optical module, inertial module, and tactile module receive the synchronous start pulse signal, the synchronous start pulse signal is marked as the clock reference starting point of the optical module, inertial module, and tactile module, and the start operation of the optical module, inertial module, and tactile module is controlled based on the clock reference starting point.

4. The intelligent motion capture method of a multimodal sensor according to claim 3, characterized in that: The steps of collecting data of the object to be measured based on the optical module, the inertial module, and the tactile module to obtain an object collection information data set specifically include: Continuously capturing images of the object to be measured by each infrared optical camera in the optical module to obtain object captured image information, and recording the image capturing time of each infrared optical camera to obtain image information acquisition time; The 9-axis attitude data of the object to be measured is collected in real time according to each IMU node in the inertial module to obtain the object attitude data information, and the data collection time of each IMU node is recorded to obtain the attitude information collection time; According to the pressure value of the contact surface of the object to be measured detected by each flexible piezoresistive sensor in the tactile module in real time, a real-time contact pressure value is obtained, the real-time contact pressure value is converted into object contact pressure information, and the data collection time of each flexible piezoresistive sensor is recorded to obtain the tactile information collection time; The object captured image information, the image information acquisition time, the object posture data information, the posture information acquisition time, the object contact pressure information and the tactile information acquisition time are combined to form an object acquisition information data set of the object to be measured.

5. The intelligent motion capture method of a multimodal sensor according to claim 4, characterized in that: The steps of creating an original acquisition timestamp based on a synchronous start pulse signal, creating a sensor network delay model to dynamically adjust the original acquisition timestamp to obtain a calibrated timestamp, and analyzing the time offset of the object data acquisition information and making corrections include: Create the original timestamp of the acquisition according to the sending time of the synchronous start pulse signal; A sensor network delay model is created, and based on the sensor network delay model, the original timestamps collected are dynamically adjusted to obtain calibrated timestamps according to the delay conditions of the optical module, the inertial module, and the tactile module.

6. The intelligent motion capture method of a multimodal sensor according to claim 5, characterized in that: The steps of creating an original acquisition timestamp based on a synchronous start pulse signal, creating a sensor network delay model to dynamically adjust the original acquisition timestamp to obtain a calibrated timestamp, and analyzing and correcting the time offset of the object data acquisition information also include: Based on the Kalman filter algorithm, the time offsets of the optical module and the inertial module are calculated respectively to obtain the optical information acquisition time offset and the attitude information acquisition time offset; Correcting the image information acquisition time based on the optical information acquisition time offset; The posture information collection time is corrected based on the posture information collection time offset.

7. The intelligent motion capture method of a multimodal sensor according to claim 6, characterized in that: The steps of adjusting the data frame rates of different information in the object acquisition information dataset to be consistent, then adjusting the data phases to be consistent to obtain the object information adjustment dataset, uploading the object information adjustment dataset to a ring buffer for storage, and outputting a data stream alignment completion signal specifically include: Record the optical data frame rate when the infrared optical camera is used to collect data to obtain the image information of the object, and record the tactile data frequency when the flexible piezoresistive sensor is used to collect data to obtain the object's posture data information; Based on a downsampling algorithm, the sampling frequency of the tactile data frequency is adjusted to be consistent with the sampling frequency of the consistent optical data frame rate, thereby obtaining tactile downsampled data information; Based on the interpolation compensation phase difference, adjusting the data phase of the tactile downsampled data information to obtain tactile data alignment adjustment information, adjusting the data phase of the object captured image information to obtain optical data alignment adjustment information, and outputting a data phase adjustment completion result, wherein the tactile data alignment adjustment information and the optical data alignment adjustment information are combined to form an object information adjustment data set; Obtain the ring buffer. When the data phase adjustment completion result is received, upload the object information adjustment data set to the ring buffer for storage. After the upload is completed, output the data stream alignment completion signal.

8. The intelligent motion capture method of a multimodal sensor according to claim 7, characterized in that: After receiving the data stream alignment completion signal, performing multimodal data fusion based on the object frequency modulation information data set, creating an object skeleton model of the object to be tested, and mapping the tactile pressure of the object to be tested detected by the tactile module to the object skeleton model of the object to be tested, specifically includes: Detect whether the active markers in the infrared optical camera are blocked. If blocked, perform real-time error correction on the optical data alignment adjustment information based on the object posture data. Based on the neural network algorithm and combined with the optical data alignment adjustment information, the accurate posture in the object posture data information is determined to obtain the accurate posture information of the object to be measured, and the accumulated error of the IMU node is reversely corrected based on the accurate posture information of the object to be measured; Based on concurrent operation of lock-free queues, an object skeleton model of the object to be tested is created, the deformation coefficient of the object to be tested is pre-set, the pressure distribution of the object to be tested is judged based on the tactile data alignment adjustment information to obtain the pressure distribution matrix, the contact force vector of the object to be tested is determined based on the deformation coefficient of the object to be tested and the pressure distribution matrix, and the tactile pressure detected in the tactile module is mapped to the object skeleton model of the object to be tested based on the contact force vector of the object to be tested.

9. An intelligent motion capture system with multimodal sensors, characterized in that: The intelligent motion capture system of the multimodal sensor is used to implement the intelligent motion capture method of the multimodal sensor according to any one of claims 1 to 8, comprising: a hardware installation module configured to establish a link between the optical module, the inertial module, the tactile module, and the synchronization core module and the object to be measured, and to perform installation; a data initialization module configured to send a synchronization start pulse signal based on the synchronization core module, set the synchronization start pulse signal as a clock reference starting point, and start the optical module, the inertial module, and the tactile module based on the clock reference starting point; A data acquisition module is configured to collect data from the object to be measured based on the optical module, the inertial module, and the tactile module to obtain an object collection information data set; A data calibration and correction module is configured to create an original acquisition time stamp based on a synchronous start pulse signal, create a sensor network delay model to dynamically adjust the original acquisition time stamp to obtain a calibration time stamp, analyze the time offset of the object data acquisition information and make corrections; a data stream alignment module configured to align the data frame rates of different information in the object acquisition information dataset, and then align the data phases to obtain an object information adjustment dataset, upload the object information adjustment dataset to a ring buffer for storage, and output a data stream alignment completion signal; The data fusion module is configured to perform multimodal data fusion based on the object frequency modulation information data set after receiving the data stream alignment completion signal, create an object skeleton model of the object to be tested, and map the tactile pressure of the object to be tested detected by the tactile module to the object skeleton model of the object to be tested.

Images