Crab gait measurement system and method based on wearable inertia and binocular vision

Abstract

The invention discloses a crab movement gait measuring system and method based on wearable inertia and binocular vision, the measuring system comprises a binding module, a wearable IMU, a visual Marker, a binocular camera, a microcontroller module, a communication module and a power supply module, the wearable IMU is fixed on a limb at the tail end of a crab, the visual Marker is fixed on the back of the crab, and the binocular camera is fixed on the back of the crab. The measurement method comprises the following steps: calculating SINS data through time correction data of the wearable IMU, correcting speed data by adopting ZUPT zero-speed data, and performing coordinate system conversion to obtain inertial navigation information under a camera coordinate system; acquiring data by using a binocular camera, and calculating to obtain space coordinate information of the tail end limb of the crab under a camera coordinate system; the pose information of the tail end limb of the crab is obtained through extended Kalman filtering fusion; the attitude angle of each joint is calculated based on the leg length information of the crab through the pose information of the limb at the tail end, and finally the gait information of the crab is obtained. According to the method, the problem of shielding in a vision measurement method is solved by fusing inertia and binocular vision information, and the accuracy of pose information measurement is improved.

Description

Technical Field

[0001] The present invention relates to the fields of motion capture, sensor technology, vision measurement technology, and inertial measurement technology, and specifically to a crab motion gait measurement system and method based on wearable inertia and binocular vision. Background Art

[0002] Since its development in the 1970s, motion capture technology has become one of the core technologies in the fields of digital twin, robot control, human-computer interaction, etc. The core goal is to real-time capture the motion trajectory of targets (such as humans, organisms, robots, etc.) in three-dimensional space through multi-modal sensors and convert it into high-precision pose data. With the complication of application scenarios (such as biological gait detection), the motion capture system has an increasing demand for robustness and anti-occlusion ability.

[0003] In the past thirty years, optical motion capture has developed rapidly. By capturing two-dimensional images of the target surface marker points or natural feature points through cameras based on multi-view geometry and combining triangulation method to calculate three-dimensional coordinates. However, when the marker points or feature points are occluded, the system needs to rely on interpolation algorithms (such as Kalman filter) to fill the data gaps, resulting in untrustworthy data in the occluded areas. Pure inertial motion capture has attracted the attention of researchers in the past two decades. The principle is to collect acceleration, angular velocity, and magnetic field data through wearable inertial measurement units (IMUs) and combine strapdown inertial navigation solution methods or sensor fusion algorithms to solve the pose. However, the errors of accelerometers and gyroscopes will diverge over time in pose solution, resulting in cumulative errors. Wearable devices need to align the initial attitude through magnetometers or external reference frames, and the performance degrades in magnetic field interference environments.

[0004] Many researchers have realized that hybrid motion capture based on multi-modal methods is a way to suppress the errors of single-system motion capture. Therefore, it is crucial to study and develop methods for fusing inertial and binocular vision pose detection.

[0005] Patent CN115644858B designs a wearable intelligent gait analyzer based on multi-sensor data fusion technology, which fuses inertial measurement units, plantar thin-film pressure sensors, and GPS positioning sensors to perform gait recognition through deep neural network algorithms. However, this method has accuracy defects indoors due to the weakening of GPS positioning ability.

[0006] Patent CN201310431846.6 discloses a motion inertial tracking system. This detection system avoids the influence of magnetic field interference on measurement accuracy by removing the magnetometer module. However, this system lacks the ability to detect the absolute yaw angle, and the yaw angle calculated by the accelerometer and gyroscope has a drift problem, which cannot meet the high-precision measurement requirements.

[0007] Patent CN106885566B designs a wearable motion sensor and its method for anti-magnetic field interference. The attitude angle of a human body part is estimated in real time by an inertial sensor. However, due to the single sensing data, the detection device will continuously accumulate errors over time, reducing the credibility of the motion capture data and failing to meet the high-precision measurement requirements.

[0008] Therefore, to solve the problems of capture occlusion and insufficient system monitoring accuracy caused by single sensing data in existing motion capture systems, the present invention designs a crab motion gait measurement system and method based on wearable inertia and binocular vision. The detection device fuses binocular vision and an inertial sensor to solve the problem of motion capture occlusion, and has a relatively high detection accuracy. It is of great significance for helping to improve the robustness and measurement accuracy of the motion capture system. Summary of the Invention

[0009] Aiming at the problems of capture occlusion, low detection accuracy, and single detection method existing in existing vision- and infrared-based motion capture devices, the present invention proposes a crab motion gait measurement system and method based on wearable inertia and binocular vision. The system has a low complexity, is easy to use, is light to wear, greatly reduces the motion interference to the crab, and solves visual occlusion through multi-sensor data fusion, effectively improving the measurement accuracy. To achieve the above object, the technical solution adopted by the present invention is:

[0010] A crab motion gait measurement system based on wearable inertia and binocular vision, characterized in that: it includes a binding module, a microcontroller module, a wearable IMU, a visual Marker, a power module and a binocular camera. The binding module, the microcontroller module, the wearable IMU, the visual Marker and the binocular camera are respectively connected to the power module and powered by the power module respectively. The microcontroller module is fixed on the binding module, and the binding module is fixedly wound around the central axis of the carapace of the biological crab. The microcontroller module includes a microcontrol processing unit and a communication interface module. The wearable IMU includes a wearable module, and the wearable module is respectively fixed on the terminal limb segments of the biological crab. The wearable module includes an IIC communication module, a flexible IMU measurement unit and a flexible FPC circuit board. The flexible IMU measurement unit is connected to the flexible FPC circuit board. The IIC communication module is data-connected to the communication interface module through a connecting wire. The flexible IMU measurement unit is data-connected to the microcontrol processing unit through the IIC communication module and the communication interface module. The binocular camera is fixed above the biological crab and is used to collect video data during the static and moving processes of the crab. The visual Marker includes visual Marker points, and the visual Marker points are respectively fixed on the binding module and the wearable module. The visual Marker points provide identification marks for the binocular camera, and the spatial coordinate information of the wearing position of the Marker points is obtained through calculation.

[0011] It further includes an inertial wearable device installed on the biological crab, and the binding module, the microcontroller module, the wearable IMU, the visual Marker, the power module and the binocular camera are installed on the biological crab through the inertial wearable device.

[0012] As a preferred technical solution of the present invention: the binding module includes a telescopic binding belt, and the microcontroller module is fixed on the telescopic binding belt and fixed on the central axis of the carapace of the biological crab through the telescopic binding belt.

[0013] As a preferred technical solution of the present invention: the flexible IMU measurement unit includes an inertial measurement sensor, and the inertial measurement sensor includes an accelerometer, a gyroscope and a magnetometer, which are respectively used to collect 9-axis raw data of the terminal leg of the crab.

[0014] As a preferred technical solution of the present invention: the communication interface module includes a WIFI communication interface and a first IIC communication interface, the IIC communication module includes a second IIC communication interface, and the first IIC communication interface and the second IIC communication interface are connected through a flexible FPC connecting wire.

[0015] As a preferred technical solution of the present invention: the wearable module includes wearable module one, wearable module two, wearable module three, wearable module four, wearable module five, wearable module six, wearable module seven, and wearable module eight. Wearable module one, wearable module two, wearable module three, wearable module four, wearable module five, wearable module six, wearable module seven, and wearable module eight are respectively installed on the end joints of the crab to collect 9-axis raw data of the end legs of the crab, and send the data to the microcontroller module through the IIC communication module.

[0016] As a preferred technical solution of the present invention: the visual Marker points include visual Marker point one, visual Marker point two, visual Marker point three, visual Marker point four, visual Marker point five, visual Marker point six, visual Marker point seven, visual Marker point eight, and visual Marker point nine. Visual Marker point one, visual Marker point two, visual Marker point three, visual Marker point four, visual Marker point five, visual Marker point six, visual Marker point seven, and visual Marker point eight are respectively fixed on wearable module one, wearable module two, wearable module three, wearable module four, wearable module five, wearable module six, wearable module seven, and wearable module eight, and visual Marker point nine is fixed on the binding module.

[0017] As a preferred technical solution of the present invention: the power supply module includes a micro lithium battery, a charge and discharge management module, and a power conversion module. The charge and discharge management module is connected to the micro lithium battery, and the micro lithium battery is respectively connected to the power supply interfaces of the micro control processing unit, the flexible IMU measurement unit, and the FPC flexible circuit board through the power conversion module.

[0018] In the above structure: the crab motion gait measurement system based on wearable inertia and binocular vision proposed by the present invention includes a binding module, a microcontroller module, a wearable IMU, a visual Marker, a power supply module, and a binocular camera. The power supply module is used to supply power to the binding module, the microcontroller module, the wearable IMU, the visual Marker, and the binocular camera. The microcontroller module is fixed on the binding module, the binding module is fixedly wound around the central axis of the carapace of the biological crab, the wearable module is respectively fixed on the end joints of the biological crab, and the binocular camera is fixed above the biological crab for collecting video data during the static and moving processes of the crab. The visual Marker includes visual Marker points, and the visual Marker points are respectively fixed on the binding module and the wearable module. The recognition markers of the binocular camera are provided through the visual Marker points, and the spatial coordinate information at the wearing position of the Marker points is obtained through calculation.

[0019] Among them, the binding module is fixed around the central axis of the crab's carapace through a telescopic binding belt. The microcontroller module is responsible for the control and communication of the overall system, and can also be responsible for collecting and analyzing the inertial information during the pressing process. Most of the auxiliary circuits and components of the entire device are distributed in the binding device to reduce the impact on the movement caused by the pressing behavior on the end of the wearable biological foot. A microelectromechanical inertial measurement chip is set in the flexible IMU measurement unit. The wearable IMU is responsible for collecting and sending the inertial data of the crab's eight legs during movement. The visual Marker provides the identification marker for the binocular camera, and the spatial coordinate information of the Marker wearing position is obtained through calculation. The power supply module is responsible for the power conversion and stable power supply of the microcontroller module, the wearable IMU, and the binocular camera.

[0020] The usage process of this system is as follows:

[0021] Installation and startup: First, before the biological experiment, fix the binding module on the central axis of the biological crab's carapace, and fix 8 wearable modules at the end joints of the biological crab. At this time, completely fix the wearable modules on the biological crab. Then, when preparing to record the movement data of the biological crab, turn on the binocular camera to collect image data, and at the same time turn on the power supply of this system. The system will automatically collect data and upload it to the host computer for further analysis.

[0022] Collection and analysis: When the crab wears this device and moves, the microcontroller module collects the inertial information of the end joints of the biological crab during movement through the wearable module and transmits the data back to the microcontrol processing unit, and then uploads it to the host computer through the WIFI communication interface for further analysis and calculation. Through the 9-axis raw inertial data, the strapdown inertial navigation speed, position, and attitude data are calculated. Then, through coordinate transformation, the speed, position, and attitude information in the binocular camera coordinate system can be obtained. The speed data in the camera coordinate system is detected by the ZUPT zero-velocity correction method, and the speed, position, and attitude data are corrected by the Kalman filter method to further correct the cumulative error and improve the calculation accuracy of the depth information.

[0023] Computation process of data fusion: First, the velocity data and position data calculated and solved by the strapdown inertial navigation system (SINS) are subtracted from the velocity data and position data calculated by binocular vision. The difference is used as the observation vector of the extended Kalman filter. At the same time, the attitude error, velocity error, position error of SINS, and sensor error are used as the state estimation quantities to construct the state space. Then, when the measurement is updated, the one-step prediction of the error state quantity is used for state estimation through the update of the Kalman gain matrix K to obtain the error estimation quantity. Subsequently, it is fed back to the original calculation quantity of SINS for correction to solve the pose data of the high-precision crab's end limb segment. When the measurement is not updated, that is, when the binocular vision loses the target point due to factors such as occlusion, the one-step prediction quantity of the error state is used as the output and fed back to the original pose calculation quantity of SINS for correction to obtain the pose data of the lost segment of the crab's end limb in the occlusion state, greatly improving the robustness of the motion capture system and the error accuracy of the pose data. Finally, through the inverse kinematics solution of the geometric method, the attitude angles of each joint of the crab's leg are calculated and output.

[0024] A method for measuring the motion gait of a crab based on a wearable inertial and binocular vision, characterized by comprising the following steps:

[0025] S1: The wearable IMU measures the inertial information of the crab at the stationary moment, including the original data of 6 axes of acceleration and angular velocity in the carrier coordinate system, marked as the data set {AG static};

[0026] S2: Based on the data set {AG static}, the static base resolution coarse alignment of the IMU is calculated by the following double-vector attitude determination method to obtain the initial attitude rotation matrix of the IMU The n system is the northeast celestial navigation coordinate system, and the b system is the front upper right carrier coordinate system.

[0027]

[0028] where

[0029]

[0030] S3: The flexible IMU measurement unit measures the inertial information of the crab during the motion moment. Based on the strapdown inertial navigation calculation method, the original data of 9 axes of acceleration, angular velocity, and magnetic field strength are solved to obtain the attitude rotation matrix, velocity, and position data of the wearable IMU (3);

[0031] S4: By calibrating the internal and external parameters of the binocular camera (6), the internal parameter matrix K and the distortion coefficient M are determined, where

[0032]

[0033] Further, determining the external parameters includes the camera spacing T and the rotation matrix R;

[0034] S5: Calculate the sliding variance s of the acceleration data, compare it with the threshold S to solve the static state part of the crab movement, and use the strapdown inertial navigation velocity data V sins As the observation quantity, further, set the attitude, velocity, and position errors to be the state quantities, and update and correct the attitude A zupt , correct the velocity V zupt , correct the position P zupt , through the camera Euler angle attitude in step S4 Convert the corrected attitude, velocity, and position data in the navigation coordinate system to the camera coordinate system;

[0035] S6: Collect the image sequence {P t} through the binocular camera (6), project the left and right images onto the same plane, find the corresponding pixel points in the left and right images, and calculate the parallax d = u left -u right ;

[0036] S7: According to the parallax d and the calibration parameters, calculate the three-dimensional coordinates (X, Y, Z) of the object in the left camera coordinate system, the horizontal coordinate vertical coordinate depth coordinate where f x is the horizontal focal length of the left camera, b is the baseline length, and u left is the vertical pixel coordinate in the left image;

[0037] S8: According to the inertial navigation error differential model and the sensor error model, augment the main error parameters of the accelerometer and gyroscope sensors in the inertial IMU, including the accelerometer three-axis zero bias factor b a and the scale factor S a , the gyroscope three-axis zero bias factor b g and the scale factor S g to the system state matrix F of the extended Kalman filter, and further establish the state vector and the discrete-time state equation

[0038] δx k+1 = φ k+1 / k ·δx k + w k

[0039] where

[0040]

[0041] S9: Further, use the positioning error solved by binocular vision as the system observation quantity to establish a loose integration observation equation

[0042]

[0043] where \(v\sim N(0, R)\) belongs to the measurement noise;

[0044] S10: Based on the discrete-time state equation and the loose integration observation equation, further update the positioning data sequence \(\{P\) loose \}\) of the inertial and binocular vision fusion under loose integration;

[0045] S11: Based on the attitude sequence \(\{\varphi\}\) calculated in step S3 and the positioning data sequence \(\{P\) loose \}\), perform inverse kinematic solution by the geometric method to obtain the initial joints

[0046]

[0047] where \(l_1\) is the length of the first limb segment, \(l_2\) is the length of the second limb segment, \(y, x\) are the coordinates on the vertical plane of the space,

[0048]

[0049] S12: Further, through the triangular relationship formed by the first limb segment and the second limb segment, calculate by the cosine theorem

[0050]

[0051] Then

[0052] \(\theta_3=\varphi-\theta_1-\theta_2\);

[0053] S13: Repeat steps S11 and S12, traverse the attitude sequence \(\{\varphi\}\) and the positioning data sequence \(\{P\) loose \}\), and obtain the set \(\{\theta\) IK \}\) of the joint angle data of the crab limb segments.

[0054] As a preferred technical solution of the present invention: The specific conversion method between the navigation coordinate system and the camera coordinate system in step S4 is as follows: The rotation matrix \(R\) in step S3 is calculated from the Euler attitude angles calibrated by the external parameters

[0055] \(R = R\) x (\varphi)\cdot R y (\theta)\cdot R Z (\psi)

[0056] where \(\psi\) is the yaw angle of rotation about the celestial axis, \(\theta\) is the pitch angle of rotation about the east axis, \(\varphi\) is the roll angle of rotation about the north axis, and the rotation order is ZYX. Specifically expanded as:

[0057]

[0058] The transformation matrix between the vehicle coordinate system and the camera coordinate system can be calculated as follows

[0059]

[0060] As a preferred technical solution of the present invention: the specific method for updating the positioning data sequence {P loose} in the loose combined inertial and binocular vision fusion in step S10 is as follows:

[0061] S101. Calculate the update time δt G = t G - t G(-) , the observation vector system state matrix noise matrix covariance matrix measurement matrix

[0062] S102. Calculate the updated state transition matrix φ = I + Fδt G , and calculate Q k-1 = GQG T δt G(-) , solve to obtain the one-step prediction value of the velocity and position errors

[0063] δx k = φ k / k-1 ·δx k-1 ;

[0064] S103. Calculate the Kalman gain matrix K k =(P (-) H T )(R + HP (-) H T + R k ) -1 , where H is the measurement matrix, P (-) is the covariance matrix of the previous moment,

[0065] Update the measurement prediction

[0066]

[0067] At this time, the loose combined system outputs the state estimation of the velocity and position errors to correct the velocity and position data solved by SINS;

[0068] S104. Calculate the covariance matrix at time t for update:

[0069] P t =(I - KH)P (-)

[0070] P = φP t φ T +Q k

[0071]

[0072] When the binocular vision positioning monitoring quantity stops updating, the loose integration system outputs the one-step prediction value δx of the speed and position error k Correct the speed and position data solved by SINS

[0073] Adopting the technical solution of the present invention will have the following beneficial effects:

[0074] (1) The present invention adopts a flexible structure design, greatly reducing the wearing weight, and flexibly fitting the biological outer surface, solving the problems that most existing wearable devices affect biological movement, etc., and reducing the precision error caused by wearing mismatch

[0075] (2) The present invention adopts the fusion of inertial and binocular vision information, and solves the classical problem of visual occlusion in the field of motion capture through a loose integration method, improving the detection accuracy

[0076] (3) The present invention is based on the kinematic model of crab legs and feet, and calculates the attitude angle data of each joint of the legs and feet through inverse kinematics

[0077] (4) The hardware resources of the wearable node are simple, small, low-power and low-cost, suitable for large-scale adaptation in different biological wearable experiments to analyze biological gait data

[0078] (5) The present invention can analyze the post-event behavior data, store the data through the host computer, establish a crab gait database, expand the gait measurement method, and analyze the target Description of the Drawings

[0079] Figure 1 It is a schematic diagram of the structural composition of the wearable measurement device in the embodiment of the present invention

[0080] Figure 2 It is a schematic block diagram of the wearable measurement device system in the embodiment of the present invention

[0081] Figure 3 It is a schematic diagram of the fusion detection of wearable and binocular vision in the embodiment of the present invention

[0082] Figure 4 It is a schematic diagram of the microcontroller module in the embodiment of the present invention

[0083] Figure 5 It is a schematic diagram of the wearable IMU in the embodiment of the present invention

[0084] Figure 6It is a schematic diagram of the biological wearable device for crabs in the embodiments of the present invention;

[0085] Figure 7 It is a flowchart of the inertial and binocular vision fusion measurement system in the embodiments of the present invention;

[0086] Figure 8 It is a flowchart of the extended Kalman filter fusion update in the embodiments of the present invention;

[0087] Figure 9 It is a flowchart of the experiment of the wearable measurement device in the embodiments of the present invention;

[0088] Figure 10 It is a schematic diagram of the software structure of the wearable measurement device in the embodiments of the present invention;

[0089] Figure 11 It is a schematic diagram of the attitude solution of the end joint of the crab in the embodiments of the present invention;

[0090] Figure 12 It is a schematic diagram of the trajectory of the end limb segment of the crab in the embodiments of the present invention.

[0091] List of reference numerals:

[0092] 1. Binding module; 1-1. Telescopic binding strap; 2. Microcontroller module; 2-1. Microcontroller processing unit; 2-2. Communication interface module; 3. Wearable IMU; 3-1. Wearable module one; 3-2. Wearable module two; 3-3. Wearable module three; 3-4. Wearable module four; 3-5. Wearable module five; 3-6. Wearable module six; 3-7. Wearable module seven; 3-8. Wearable module eight; 3-1-1. Microelectromechanical inertial measurement chip; 3-1-2. IIC communication module; 4. Visual Marker; 4-1. Visual Marker point one; 4-2. Visual Marker point two; 4-3. Visual Marker point three; 4-4. Visual Marker point four; 4-5. Visual Marker point five; 4-6. Visual Marker point six; 4-7. Visual Marker point seven; 4-8. Visual Marker point eight; 4-9. Visual Marker point nine; 5. Power supply module; 6. Binocular camera; 7. Inertial wearable device. Detailed implementation manners

[0093] The present invention will be further described in detail below in conjunction with the accompanying drawings and the detailed implementation manners:

[0094] Refer to Figure 1 、 Figure 2 、 Figure 3 and Figure 7, the crab motion gait measurement system based on wearable inertia and binocular vision proposed by the present invention includes a binding module 1, a microcontroller module 2, a wearable IMU 3, a visual Marker 4, a power supply module 5, and a binocular camera 6. The binding module 1, the microcontroller module 2, the wearable IMU 3, the visual Marker 4, and the binocular camera 6 are respectively connected to the power supply module 5 and powered by the power supply module 5. The microcontroller module 2 is fixed on the binding module 1, and the binding module 1 is fixedly wound around the central axis of the carapace of the biological crab. The microcontroller module 2 includes a microcontrol processing unit 2-1 and a communication interface module 2-2. The wearable IMU 3 includes a wearable module, and the wearable module is respectively fixed on the terminal limb segments of the biological crab. The wearable module includes an IIC communication module 3-1-2, a flexible IMU measurement unit, and a flexible FPC circuit board. The flexible IMU measurement unit is connected to the flexible FPC circuit board. The IIC communication module 3-1-2 is data-connected to the communication interface module 2-2 through a connecting wire. The flexible IMU measurement unit is data-connected to the microcontrol processing unit 2-1 through the IIC communication module 3-1-2 and the communication interface module 2-2. The binocular camera 6 is fixed above the biological crab and is used to collect video data during the static and moving processes of the crab. The visual Marker 4 includes visual Marker points, and the visual Marker points are respectively fixed on the binding module 1 and the wearable module. The recognition marks of the binocular camera 6 are provided through the visual Marker points, and the spatial coordinate information of the wearing position of the Marker points is obtained through calculation.

[0095] It also includes an inertial wearable device 7 installed on the biological crab, and the binding module 1, the microcontroller module 2, the wearable IMU 3, the visual Marker 4, the power supply module 5, and the binocular camera 6 are installed on the biological crab through the inertial wearable device 7. The binding module 1 includes a telescopic binding strap 1-1, and the microcontroller module 2 is fixed on the telescopic binding strap 1-1 and fixed on the central axis of the carapace of the biological crab through the telescopic binding strap 1-1. The flexible IMU measurement unit includes an inertial measurement sensor, and the inertial measurement sensor includes an accelerometer, a gyroscope, and a magnetometer, which are respectively used to collect 9-axis raw data of the end leg of the crab. The communication interface module 2-2 includes a WIFI communication interface and a first IIC communication interface, and the IIC communication module 3-1-2 includes a second IIC communication interface. The first IIC communication interface and the second IIC communication interface are connected through a flexible FPC connection line. The wearable module includes wearable module one 3-1, wearable module two 3-2, wearable module three 3-3, wearable module four 3-4, wearable module five 3-5, wearable module six 3-6, wearable module seven 3-7, and wearable module eight 3-8. The wearable module one 3-1, wearable module two 3-2, wearable module three 3-3, wearable module four 3-4, wearable module five 3-5, wearable module six 3-6, wearable module seven 3-7, and wearable module eight 3-8 are respectively installed on the end joints of the crab to collect 9-axis raw data of the end leg of the crab, and send the data to the microcontroller module 2 through the IIC communication module 3-1-2. The visual Marker points include visual Marker point one 4-1, visual Marker point two 4-2, visual Marker point three 4-3, visual Marker point four 4-4, visual Marker point five 4-5, visual Marker point six 4-6, visual Marker point seven 4-7, visual Marker point eight 4-8, and visual Marker point nine 4-9. The visual Marker point one 4-1, visual Marker point two 4-2, visual Marker point three 4-3, visual Marker point four 4-4, visual Marker point five 4-5, visual Marker point six 4-6, visual Marker point seven 4-7, and visual Marker point eight 4-8 are respectively fixed on the wearable module one 3-1, wearable module two 3-2, wearable module three 3-3, wearable module four 3-4, wearable module five 3-5, wearable module six 3-6, wearable module seven 3-7, and wearable module eight 3-8, and the visual Marker point nine 4-9 is fixed on the binding module 1.The power supply module 5 includes a micro lithium battery, a charge and discharge management module, and a power conversion module. The charge and discharge management module is connected to the micro lithium battery, and the micro lithium battery is respectively connected to the micro control processing unit 2-1, the flexible IMU measurement unit, and the power interface of the FPC flexible circuit board through the power conversion module.

[0096] The crab motion gait measurement system based on wearable inertia and binocular vision proposed by the present invention includes a binding module 1, a microcontroller module 2, a wearable IMU 3, a vision Marker 4, a power supply module 5, and a binocular camera 6. The power supply module 5 is used to supply power to the binding module 1, the microcontroller module 2, the wearable IMU 3, the vision Marker 4, and the binocular camera 6. The microcontroller module 2 is fixed on the binding module 1. The binding module 1 is fixedly wound around the central axis of the carapace of the biological crab. The wearable module is respectively fixed on the end joints of the biological crab. The binocular camera 6 is fixed above the biological crab and is used to collect video data during the static and moving processes of the crab. The vision Marker 4 includes vision Marker points, and the vision Marker points are respectively fixed on the binding module 1 and the wearable module. The recognition marker of the binocular camera 6 is provided through the vision Marker points, and the spatial coordinate information of the wearing position of the Marker points is obtained through calculation.

[0097] Among them, the binding module 2 is fixedly wound around the central axis of the crab's carapace through the telescopic binding belt 1-1. The microcontroller module 2 is responsible for the control and communication of the overall system, and can also be responsible for collecting and analyzing the inertial information during the pressing process. Most of the auxiliary circuits and components of the entire device are distributed in the binding device to reduce the impact on the movement caused by the pressing behavior on the end of the wearing biological foot. A microelectromechanical inertial measurement chip 3-1-1 is provided in the flexible IMU measurement unit. The wearable IMU 3 is responsible for collecting and sending the inertial data of the eight legs of the crab during the movement. The vision Marker 4 provides the recognition marker of the binocular camera 6, and the spatial coordinate information of the wearing position of the Marker is obtained through calculation. The power supply module 5 is responsible for the power conversion and stable power supply of the microcontroller module 2, the wearable IMU, and the binocular camera 6.

[0098] Refer to Figure 1 、 Figure 2 、 Figure 3 and Figure 7 The microcontroller module includes a micro control processing unit 2-1 and a communication interface module 2-2. The micro control processing unit 2-1 is responsible for receiving the inertial data of the eight wearable modules, marking the UNIX time stamp for each inertial data through the SNTP server, and finally sending the calibrated inertial data of the eight wearable modules to the upper computer at high speed through the WIFI communication interface.

[0099] Refer to Figure 1 and6 The wearable IMU 3 includes a wearable module 1 (3-1), a wearable module 2 (3-2), a wearable module 3 (3-3), a wearable module 4 (3-4), a wearable module 5 (3-5), a wearable module 6 (3-6), a wearable module 7 (3-7), and a wearable module 8 (3-8), which respectively collect 9-axis raw data of the end legs of the crab and send the data to the microcontroller module 2 through the first IIC communication interface.

[0100] As Figure 1 , Figure 3 , Figure 6 , Figure 7 As shown in the figure, the installation and startup are as follows: First, before the biological experiment, fix the binding module 1 on the central axis of the crab's carapace, and fix 8 wearable modules at the end joints of the crab. At this time, the wearable modules are completely fixed on the crab. Then, when preparing to record the movement data of the crab, turn on the binocular camera 6 to collect image data. At the same time, turn on the power of the system, and the system will automatically collect data and upload it to the host computer for further analysis.

[0101] As Figure 1 and Figure 7 As shown in the figure, the data collection and analysis are as follows: When the crab wears the device and moves, the microcontroller module 2 collects the inertial information of the end joints of the crab during the movement through the wearable module and sends the data back to the microcontrol processing unit 2-1, and then uploads it to the host computer through the WIFI communication interface for further analysis and calculation. Through the 9-axis raw inertial data, the strapdown inertial navigation speed, position, and attitude data are calculated. Then, through coordinate transformation, the speed, position, and attitude information in the coordinate system of the binocular camera 6 can be obtained. The speed data in the camera coordinate system is detected by the ZUPT zero-velocity correction method, and the speed, position, and attitude data are corrected by the Kalman filter method to further correct the cumulative error and improve the calculation accuracy of the depth information.

[0102] As Figure 7 , 8, as shown in Figure 9, the calculation process of data fusion: First, the velocity data and position data calculated by the strapdown inertial navigation system (SINS) are subtracted from the velocity data and position data calculated by binocular vision. The difference is used as the observation vector of the extended Kalman filter. At the same time, the attitude error, velocity error, position error of the SINS, and sensor error are used as the state estimation quantities to construct the state space. Then, when the measurement is updated, the one-step prediction of the error state quantity is used for state estimation through the update of the Kalman gain matrix K to obtain the error estimation quantity. Then, it is fed back to the original calculation quantity of the SINS for correction to solve the pose data of the high-precision crab's end limb segment. When the measurement is not updated, that is, when the binocular vision loses the target point due to factors such as occlusion, the one-step prediction quantity of the error state is used as the output and fed back to the original pose quantity of the SINS for correction to obtain the pose data of the lost segment of the crab's end limb in the occlusion state, greatly improving the robustness of the motion capture system and the error accuracy of the pose data. Finally, through the inverse kinematics solution of the geometric method, the attitude angles of each joint of the crab's leg are calculated and output.

[0103] The method for measuring the motion gait of a crab based on wearable inertia and binocular vision proposed by the present invention includes the following steps:

[0104] S1: The wearable IMU measures the inertial information at the moment when the crab is stationary, including the original data of 6 axes of acceleration and angular velocity in the carrier coordinate system, marked as the data set {AG static};

[0105] S2: Based on the data set {AG static}, the static base resolution coarse alignment of the IMU is calculated by the following double-vector attitude determination method to obtain the initial attitude rotation matrix of the IMU The n system is the northeast celestial navigation coordinate system, and the b system is the front upper right carrier coordinate system.

[0106]

[0107] Among them

[0108]

[0109] S3: The flexible IMU measurement unit measures the inertial information at the moment when the crab is moving. Based on the strapdown inertial navigation calculation method, the original data of 9 axes of acceleration, angular velocity, and magnetic field strength are solved to obtain the attitude rotation matrix, velocity, and position data of the wearable IMU (3);

[0110] S4: By calibrating the internal and external parameters of the binocular camera (6), the internal parameter matrix K and the distortion coefficient M are determined, where

[0111]

[0112] Further, determining the external parameters includes the camera spacing T and the rotation matrix R;

[0113] S5: Calculate the sliding variance s of the acceleration data, compare it with the threshold S to solve the stationary state part of the crab movement, and use the strapdown inertial navigation velocity data V sins as the observable. Further, set the attitude, velocity, and position errors to the state variables, and update and correct the attitude A zupt , the corrected velocity V zupt , and the corrected position P zupt through the Kalman filter. Through the camera Euler angle attitude {Φ, θ, φ} in step S4, convert the corrected attitude, velocity, and position data in the navigation coordinate system to the camera coordinate system;

[0114] S6: Collect the image sequence {P t} through the binocular camera (6), project the left and right images onto the same plane, find the corresponding pixel points in the left and right images, and calculate the parallax d = u left -u right ;

[0115] S7: According to the parallax d and the calibration parameters, calculate the three-dimensional coordinates (X, Y, Z) of the object in the left camera coordinate system, the horizontal coordinate the vertical coordinate the depth coordinate where f x is the horizontal focal length of the left camera, b is the baseline length, and u left is the vertical pixel coordinate in the left image;

[0116] S8: According to the inertial navigation error differential model and the sensor error model, augment the main error parameters of the accelerometer and gyroscope sensors in the inertial IMU, including the accelerometer three-axis zero bias factor b a and the scale factor S a , the gyroscope three-axis zero bias factor b g and the scale factor S g to the system state matrix F of the extended Kalman filter, and further establish the state vector and the discrete-time state equation

[0117] δx k+1 = φ k+1 / k ·δx k + w k

[0118] where

[0119]

[0120] S9: Further, use the positioning error solved by binocular vision as the system observable to establish a loose integration observation equation

[0121]

[0122] where \(v\sim N(0, R)\) belongs to the measurement noise;

[0123] S10: Based on the discrete-time state equation and the loose integration observation equation, further update the positioning data sequence \(\{P\}\) of the inertial and binocular vision fusion under loose integration loose};

[0124] S11: Based on the attitude sequence \(\{\varphi\}\) calculated in step S3 and the positioning data sequence \(\{P\}\) calculated in S8, perform inverse kinematic solution by the geometric method to obtain the initial joints loose};

[0125]

[0126] where \(l_1\) is the length of the first limb segment, \(l_2\) is the length of the second limb segment, and \(y, x\) are the coordinates on the vertical plane of the space.

[0127]

[0128] S12: Further, through the triangular relationship formed by the first limb segment and the second limb segment, calculate by the cosine theorem

[0129]

[0130] Then

[0131] \(\theta_3=\varphi - \theta_1 - \theta_2\);

[0132] S13: Repeat steps S11 and S12, traverse the attitude sequence \(\{\varphi\}\) and the positioning data sequence \(\{P\}\) loose}, and obtain the set of joint angle data \(\{\theta\}\) of the crab limb segments IK}.

[0133] The conversion method between the navigation coordinate system and the camera coordinate system in step S4 is specifically as follows: The rotation matrix \(R\) in step S3 is calculated from the Euler attitude angles calibrated by the external parameters

[0134] \(R = R\) x (\varphi)\cdot R y (\theta)\cdot R Z (\psi)

[0135] where \(\psi\) is the yaw angle of rotation about the celestial axis, \(\theta\) is the pitch angle of rotation about the east axis, and \(\varphi\) is the roll angle of rotation about the north axis. The rotation order is ZYX, and specifically expanded as:

[0136]

[0137] The conversion matrix between the vehicle coordinate system and the camera coordinate system can be calculated as follows

[0138]

[0139] The specific method for updating the positioning data sequence {P loose} in step S10 of the loose integration of inertial and binocular vision is as follows:

[0140] S101. Calculate the update time δt G = t G - t G(-) , the observation vector system state matrix noise matrix covariance matrix measurement matrix

[0141] S102. Update and calculate the state transition matrix φ = I + Fδt G , and calculate Q k-1 = GQG T δt G(-) , and solve to obtain the one-step prediction value of the velocity and position errors

[0142] δx k = φ k / k-1 ·δx k-1 ;

[0143] S103. Calculate the Kalman gain matrix K k = (P (-) H T )(R + HP (-) H T + R k ) -1 , where H is the measurement matrix, and P (-) is the covariance matrix at the previous moment,

[0144] Update the measurement prediction

[0145]

[0146] At this time, the loose integration system outputs the state estimation of the velocity and position errors to correct the velocity and position data solved by SINS;

[0147] S104. Calculate the covariance matrix at time t after update:

[0148] P t = (I - KH)P (-)

[0149] P = φP t φ T + Q k

[0150]

[0151] When the binocular vision positioning monitoring quantity stops updating, the loose integration system outputs the one-step predicted values δx of the speed and position errors k Correct the speed and position data of the SINS solution

[0152] This method can improve the robustness of the motion capture system, solve the problem of visual occlusion, and improve the data accuracy through multi-modal fusion. In this example, the joint attitude angle solution and the spatial motion trajectory of the end limb are respectively as Figure 9 and Figure 10 shown, which has high value and significance for further analyzing the motion gait characteristics of crab organisms for landing applications on bionic robots

[0153] The above are only the preferred embodiments of the present invention, and are not any other form of limitation to the present invention. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope of protection required by the present invention

Claims

1. The crab motion gait measurement system based on wearable inertial and binocular vision is characterized by: The invention comprises a binding module (1), a microcontroller module (2), a wearable IMU (3), a visual marker (4), a power module (5) and a binocular camera (6), wherein the binding module (1), the microcontroller module (2), the wearable IMU (3), the visual marker (4) and the binocular camera (6) are respectively connected to the power module (5) and are respectively powered by the power module (5), wherein the microcontroller module (2) is fixed on the binding module (1), and the binding module (1) is fixed around the central axis of the shell of the biological crab, wherein the microcontroller module (2) comprises a microcontroller processing unit (2-1) and a communication interface module (2-2), wherein the wearable IMU (3) comprises a wearable module, wherein the wearable modules are respectively fixed on the terminal limbs of the biological crab, and wherein the wearable module comprises an IIC communication module (3-1-2 ), a flexible IMU measurement unit and a flexible FPC circuit board, the flexible IMU measurement unit is connected to the flexible FPC circuit board, the IIC communication module (3-1-2) is data-connected to the communication interface module (2-2) via a connecting line, the flexible IMU measurement unit is data-connected to the microcontroller processing unit (2-1) via the IIC communication module (3-1-2) and the communication interface module (2-2), the binocular camera (6) is fixed above the biological crab, and is used to collect video data of the crab's static and moving processes, the visual marker (4) includes visual marker points, and the visual marker points are respectively fixed on the binding module (1) and the wearable module, and the binocular camera (6) is provided with an identification mark through the visual marker point, and the spatial coordinate information of the marker point where it is worn is obtained by calculation, It also includes an inertial wearable device (7) installed on the biological crab, wherein the binding module (1), the microcontroller module (2), the wearable IMU (3), the visual marker (4), the power module (5) and the binocular camera (6) are installed on the biological crab via the inertial wearable device (7).

2. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 is characterized in that: The binding module (1) comprises a retractable binding belt (1-1), and the microcontroller module (2) is fixed on the retractable binding belt (1-1) and fixed on the central axis of the carapace of the biological crab through the retractable binding belt (1-1).

3. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 is characterized in that: The flexible IMU measurement unit includes an inertial measurement sensor, which includes an accelerometer, a gyroscope and a magnetometer, which are respectively used to collect 9-axis raw data of the crab's terminal legs and feet.

4. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 is characterized in that: The communication interface module (2-2) comprises a WIFI communication interface and a first IIC communication interface, the IIC communication module (3-1-2) comprises a second IIC communication interface, and the first IIC communication interface and the second IIC communication interface are connected via a flexible FPC connection line.

5. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 is characterized in that: The wearable module includes a wearable module one (3-1), a wearable module two (3-2), a wearable module three (3-3), a wearable module four (3-4), a wearable module five (3-5), a wearable module six (3-6), a wearable module seven (3-7) and a wearable module eight (3-8), wherein the wearable module one (3-1), the wearable module two (3-2), the wearable module three (3-3), the wearable module four (3-4), the wearable module five (3-5), the wearable module six (3-6), the wearable module seven and the wearable module eight (3-8) They are respectively installed on the terminal limbs of the crab, collect the 9-axis raw data of the terminal legs and feet of the crab, and send the data to the microcontroller module (2) through the IIC communication module (3-1-2).

6. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 or 5, characterized in that: The visual marker points include visual marker point 1 (4-1), visual marker point 2 (4-2), visual marker point 3 (4-3), visual marker point 4 (4-4), visual marker point 5 (4-5), visual marker point 6 (4-6), visual marker point 7 (4-7), visual marker point 8 (4-8) and visual marker point 9 (4-9). The visual marker point 1 (4-1), visual marker point 2 (4-2), visual marker point 3 (4-3), visual marker point 4 (4-4), visual marker point 5 (4-5), visual marker point 6 (4-6), visual marker point 7 (4-7), visual marker point 8 (4-8) and visual marker point 9 (4-9). Marker point four (4-4), visual marker point five (4-5), visual marker point six (4-6), visual marker point seven (4-7), and visual marker point eight (4-8) are respectively fixed on wearable module one (3-1), wearable module two (3-2), wearable module three (3-3), wearable module four (3-4), wearable module five (3-5), wearable module six (3-6), wearable module seven (3-7), and wearable module eight (3-8), and the visual marker point nine (4-9) is fixed on the binding module (1).

7. The crab motion gait measurement system based on wearable inertial and binocular vision according to claim 1 is characterized in that: The power module (5) comprises a micro lithium battery, a charge and discharge management module, and a power conversion module. The charge and discharge management module is connected to the micro lithium battery. The micro lithium battery is respectively connected to the power interfaces of the micro control processing unit (2-1), the flexible IMU measurement unit, and the FPC flexible circuit board through the power conversion module.

8. The crab motion gait measurement method based on wearable inertial and binocular vision according to any one of claims 1 to 7, characterized in that: The steps include: S1: Wearable IMU (3) measures the inertial information of the crab at rest, including the acceleration and angular velocity in the carrier coordinate system, a total of 6-axis raw data, marked as data set {AG static }; S2: Based on the dataset {AG static } Calculate the IMU static base analytical coarse alignment through the following dual-vector attitude determination method to obtain the initial attitude rotation matrix of the IMU The n system is the northeast sky navigation coordinate system, and the b system is the front upper right carrier coordinate system. in S3: The flexible IMU measurement unit measures the inertial information of the crab during its movement, and calculates the acceleration, angular velocity, and magnetic field strength of a total of 9 axes of raw data based on the strapdown inertial navigation calculation method to obtain the attitude rotation matrix, velocity, and position data of the wearable IMU (3); S4: By calibrating the internal and external parameters of the binocular camera (6), the internal parameter matrix K and the distortion coefficient M are determined, where Furthermore, the external parameters are determined to include the camera spacing T and the rotation matrix R; S5: Calculate the sliding variance s of the acceleration data, compare it with the threshold S to solve the crab motion static state part, and convert the strapdown inertial navigation velocity data V sins As the observation quantity, further, the attitude, speed, and position error correction are set as state quantities, and the corrected attitude A is updated through Kalman filtering zupt , Correction speed V zupt , corrected position P zupt , through the camera Euler angle posture in step S4 Convert the corrected attitude, speed and position data in the navigation coordinate system to the camera coordinate system; S6: Collect image sequence {P t }, project the left and right images onto the same plane, find the corresponding pixels in the left and right images, and calculate the disparity d = u left -u right ; S7: Calculate the three-dimensional coordinates (XYZ) of the object in the left camera coordinate system according to the parallax d and calibration parameters, and the horizontal coordinate Vertical coordinate Depth coordinate where f x is the horizontal focal length of the left camera, b is the baseline length, u left is the vertical pixel coordinate in the left image; S8: According to the inertial guidance error differential model and sensor error model, the main error parameters of the accelerometer and gyroscope sensors in the inertial IMU include the accelerometer three-axis zero bias factor b a With the scale factor S a , gyroscope three-axis bias factor b g With the scale factor S g Augmented to the system state matrix F of the extended Kalman filter, further establish the state vector and discrete time state equation δx k+1 =φ k+1 / k ·δx k +w k in S9: Further, based on the positioning error solved by binocular vision as the system observation quantity, a loose combined observation equation is established Among them, v~N(0,R) belongs to the measurement noise; S10: Based on the discrete time state equation and the loose combination observation equation, the positioning data sequence {P loose }; S11: Based on the posture sequence {φ} calculated in step S3 and the positioning data sequence {P loose }, the inverse kinematics solution is performed by geometric method, and the initial joint is obtained Where l1 is the length of the first limb segment, l2 is the length of the second limb segment, y, x are the coordinates on the vertical plane of space, S12: Further, through the triangular relationship between the first limb and the second limb, the cosine theorem is used to calculate but θ3=φ-θ1-θ2; S13: Repeat steps S11 and S12 to traverse the posture sequence {φ} and the positioning data sequence {P loose }, and obtain the data set of angles of each joint of the crab limb {θ IK }.

9. The crab motion gait measurement method based on wearable inertial and binocular vision according to claim 8 is characterized in that: The specific conversion method between the navigation coordinate system and the camera coordinate system in step S4 is as follows: The rotation matrix R in step S3 is calculated by the Euler attitude angle calibrated by the external parameters R=R x (φ)·R y (θ)·R Z (ψ) Where ψ is the yaw angle around the celestial axis, θ is the pitch angle around the east axis, φ is the roll angle around the north axis, and the rotation order is ZYX, which can be expanded as follows: The transformation matrix between the carrier coordinate system and the camera coordinate system can be given by the following calculation 10. The crab motion gait measurement method based on wearable inertial and binocular vision according to claim 8 is characterized in that: The updating sequence of the positioning data of the inertial and binocular vision fusion under the loose combination in step S10 is loose The specific method of} is as follows: S101, update time calculation δt G =t G -t G(-) , the observation vector System state matrix Noise Matrix Covariance matrix Measurement Matrix S102, update the calculation state transfer matrix φ=I+Fδt G , and calculate Q k-1 =GQG T δt G(-) , the solution gives the one-step prediction value of the speed and position error δx k =φ k / k-1 ·δx k-1 ; S103, calculate the Kalman gain matrix K k =(P (-) H T )(R+HP (-) H T +R k ) -1 , where H is the measurement matrix, P (-) is the covariance matrix of the previous moment, Update measurement forecast At this time, the state estimation of the output speed and position error of the loose combined system is To correct the speed and position data solved by SINS; S104, calculate and update the covariance matrix at time t: P t =(I-KH)P (-) P=φP t φ T +Q k When the binocular vision positioning monitoring quantity stops updating, the combined system outputs the one-step prediction value of the speed and position error δx k Corrected the speed and position data solved by SINS.

Images