A finger motion tracking method and system based on a compressed sensing model

Abstract

The application discloses a finger motion tracking method and system based on a compressed sensing model, and relates to the technical field of finger motion tracking; a coupling relationship of finger motion information collected by a flexible sensor and an infrared camera is solved by using compressed sensing to obtain a measurement matrix; an electric signal generated when the flexible sensor collects finger bending is converted into a digital signal; the digital signal and the measurement matrix are solved by using compressed sensing; and the bending angles of all finger joints are reconstructed. The number of the flexible sensors used can be far less than the number of hand degrees of freedom, so that the arrangement difficulty of the flexible sensor on a glove is simplified, the system structure is simple, and the position of the flexible sensor does not need to be accurately fixed. The finger motion tracking visual scheme and the flexible sensor glove are fused; the angle calculated by the visual scheme is mapped out by using a compressed sensing algorithm through the flexible sensor; and the site limitation and the shielding problem of the visual scheme are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of finger motion tracking, and particularly relates to a finger motion tracking method and system based on a compressed sensing model. BACKGROUND

[0002] Finger motion is one of the most complex and flexible motion systems of the human body. Hand motion capture technology for tracking the motion of the human hand is widely used in many fields such as entertainment, sports, medical applications, and robot technology. Using finger motion tracking technology to interactively control the human and the robot or the virtual environment can more comprehensively and intuitively express the user's intention, and significantly improves the efficiency of the application. Especially for complex control that cannot be achieved by traditional control methods, for example, controlling a virtual hand model to interact with a virtual environment in a VR virtual environment, controlling a humanoid robot, and relying on a mouse, a keyboard, or a tablet to operate is difficult to complete the above tasks, but using finger motion tracking technology can directly transmit finger motion information to the virtual hand model and the humanoid robot, and can more easily and smoothly control, greatly improving the user experience.

[0003] Existing finger motion tracking solutions are a finger motion tracking method using an inertial sensor (IMU), a finger motion tracking method using a flexible sensor, and a finger motion tracking method using a camera. Currently, these hand motion capture solutions all have their own basic limitations.

[0004] The finger motion tracking method using the inertial sensor (IMU) usually uses a six-axis or nine-axis sensor, and each sensor is placed at a finger joint. The bending angle and direction of all finger joints are obtained through each inertial sensor (IMU) attached to the finger, so as to reconstruct the finger motion information. However, the inertial sensor (IMU) used has a "drift" problem, is easily affected by electromagnetic interference, and is affected by the magnetic field of a sound box, a computer case, and a mobile phone, resulting in a decrease in precision. In the actual use process, it is necessary to be away from these electronic devices or to frequently calibrate.

[0005] The finger motion tracking method using the flexible sensor places a flexible sensor at each joint that needs to estimate the angle. The bending angle of the joint is mapped through the original data of the sensor. Based on the theory of biological anatomy, the three degrees of freedom (DoF) of the wrist are not considered, and the finger joints have 21 degrees of freedom (DoF). Therefore, at least 21 flexible sensors need to be placed on the hand to map the bending angles of all fingers of the hand through the original data of the sensors. The circuit for connecting at least 21 sensors to transmit signals is very complex, and consumes a large amount of circuit and computing resources.

[0006] The finger motion tracking method using a camera uses an infrared camera, an RGB camera or an RGB-D camera (depth camera) to collect images of finger motion, calculates the finger bending angle through image processing, or uses a deep learning method to train a model, and infers the finger bending angle through the trained model. This method requires a large amount of computing resources. Since the visual-based finger motion tracking method needs to be used with an external camera, its limitations include: (1) the recognition effect is affected by the resolution and frame rate of the camera, (2) it is limited by the shooting site, and (3) the finger is blocked, which leads to poor recognition effect. SUMMARY

[0007] To solve the problems of the prior art, the present application provides a finger motion tracking method and system based on a compressed sensing model. The coupling relationship of the finger motion information collected by the flexible sensor and the infrared camera is solved by compressed sensing, and the measurement matrix is obtained. Then, the electrical signal generated by the flexible sensor when the finger is bent is converted into a digital signal. The digital signal and the measurement matrix are solved by compressed sensing, and the bending angles of all finger joints are reconstructed.

[0008] To achieve the above-mentioned object, the technical scheme adopted by the present application is as follows:

[0009] In a first aspect, the present application provides a finger motion tracking system based on a compressed sensing model, which comprises a flexible sensor, a signal processing module, a signal sending module, a signal receiving module, an infrared camera, an infrared light source, an image processing module and a microcontroller. The flexible sensor, the signal processing module and the signal sending module are arranged at a predetermined position on the glove carrier. The infrared camera and the infrared light source are arranged on one side of the hand motion area. The signal receiving module, the image processing module and the microcontroller are arranged on an armUltra development board.

[0010] The flexible sensor is arranged on the glove at a predetermined position corresponding to the finger joint, and generates an electrical signal when the finger moves.

[0011] The signal processing module amplifies and filters the electrical signal of the flexible sensor, and then converts the amplified and filtered electrical signal into a digital signal through A / D conversion.

[0012] The signal sending module transmits the digital signal output by the signal processing module to the signal receiving module.

[0013] The signal receiving module outputs the digital signal sent by the signal sending module to the microcontroller.

[0014] The signal sending module and the signal receiving module use wired transmission and / or Bluetooth and / or wifi and / or 1.2g / 2.4G / 5.8g wireless transmission.

[0015] An infrared camera is placed on one side of the hand movement area to collect the infrared image of the hand wearing the flexible sensor glove.

[0016] An infrared light source is placed with the infrared camera to illuminate the hand wearing the flexible sensor glove.

[0017] An image processing module calculates the angle data of all joints of the hand through the deep learning algorithm of the infrared image collected by the infrared camera, and then outputs the angle data to the microcontroller for processing.

[0018] The microcontroller compressively senses and solves the measurement matrix obtained from the digital signal received by the signal receiving module and the angle data output by the image processing module, and compressively senses and solves the measurement matrix and the digital signal output by the signal receiving module to reconstruct the angle data.

[0019] The deep learning algorithm includes a palm detection model, a hand key point detection model, and an angle calculation model.

[0020] The palm detection model searches and matches the hand infrared image through a YOLO network to obtain a hand bounding box;

[0021] The hand key point detection model operates in the hand bounding box area to obtain the three-dimensional coordinates of the 3D hand key points;

[0022] The angle calculation model converts the three-dimensional coordinates of the 3D hand key points into the finger joint bending angle x1 through linear algebra or spatial analytic geometry.

[0023] In a second aspect, the present application provides a finger movement tracking method based on a compressive sensing model, which is realized by using the finger movement tracking system based on the compressive sensing model, and includes the following steps:

[0024] S1: a user wears a glove carrier provided with a flexible sensor, a signal processing module, and a signal sending module, and an infrared camera and an infrared light source are arranged on one side of the user's hand movement area;

[0025] S2: when the finger moves, the flexible sensor generates an electrical signal, and the infrared camera synchronously collects the infrared movement image of the finger under the irradiation of the infrared light source;

[0026] S3: the electrical signal collected by the flexible sensor in S2 is converted into a digital signal through A / D conversion, the finger infrared movement image is converted into finger joint bending angle data using a deep learning algorithm, and the digital signal and the finger joint bending angle data are sent to a microcontroller, and the ISTA iterative solution formula of the compressive sensing algorithm is used to solve the measurement matrix for subsequent actual detection.

[0027] S4: The measurement matrix obtained according to S3 collects the electrical signal generated by the flexible sensor, and the infrared camera synchronously collects the infrared motion image of the finger under the irradiation of the infrared light source;

[0028] S5: When the finger moves, the signal processing module converts the electrical signal generated by the flexible sensor into a digital signal y2 through A / D conversion, and transmits the converted digital signal y2 to the microcontroller through the signal sending module and the signal receiving module.

[0029] The microcontroller uses the ISTA iterative solution formula of the compressive sensing algorithm to solve the digital signal y2 converted by the signal processing module and the measurement matrix Φ calculated in S3, and obtains the angle data of the real-time finger joint bending.

[0030] The specific process of S3 is as follows:

[0031] The signal processing module converts the electrical signal generated by the flexible sensor into a digital signal y1 through A / D conversion, and transmits the converted digital signal y1 to the microcontroller through the signal sending module and the signal receiving module.

[0032] The image processing module uses a deep learning algorithm to convert the infrared motion image of the finger shot by the infrared camera into finger joint bending angle data x, and then outputs the angle data x to the microcontroller.

[0033] The microcontroller uses compressive sensing to solve the digital signal y1 converted by the signal processing module and the angle data x1 converted by the image processing module, and obtains the measurement matrix Φ.

[0034] The compressive sensing is shown in formula (1) and formula (2):

[0035] y=Φx (1)

[0036] x=Ψs (2)

[0037] Wherein, x is the angle data estimated by the image processing module through the infrared camera, y is the flexible sensor signal, Ψ is the sparse base, Φ is the measurement matrix, and s is the sparse coefficient.

[0038] The ISTA iterative solution formula of the compressive sensing algorithm is shown in formula (3) and formula (4):

[0039] r (k) =x2 (k-1) -ρΦ T (Φx2 (k-1) -y2) (3)

[0040]

[0041] Wherein, k represents the number of iterations, and ρ represents the iteration step.

[0042] Beneficial technical effects

[0043] The number of flexible sensors used in the application can be much smaller than the number of hand degrees of freedom, simplifying the difficulty of arranging the flexible sensors on the glove, and the system structure is simple and does not need to accurately fix the position of the flexible sensor.

[0044] The number of flexible sensors used in the application is small, the circuit complexity is low, and the required bandwidth for communication with the microchannel is low.

[0045] The application combines finger motion tracking visual scheme and flexible sensor gloves, and maps the angle calculated by the visual scheme through the compression sensing algorithm through the flexible sensor, solving the field restriction and occlusion problem of the visual scheme. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A finger motion tracking system module based on a compression sensing model is provided for the embodiment of the application;

[0047] Figure 2 A deep learning algorithm architecture diagram is provided for the embodiment of the application;

[0048] Figure 3 A finger motion tracking method flowchart based on a compression sensing model is provided for the embodiment of the application. DETAILED DESCRIPTION

[0049] The specific embodiments of the application will be further described in detail below in combination with the drawings and embodiments.

[0050] The embodiment provides a finger motion tracking method and system based on a compression sensing model. The coupling relationship of the finger motion information collected by the flexible sensor and the infrared camera is solved by using compression sensing, a measurement matrix is obtained, and the electrical signal generated by the flexible sensor when the finger is bent is converted into a digital signal. The digital signal and the measurement matrix are solved by compression sensing, and the bending angles of all finger joints are reconstructed.

[0051] In a first aspect, the embodiment provides a finger motion tracking system based on a compression sensing model, as shown in Figure 1 The system includes a flexible sensor 1, a signal processing module 2, a signal sending module 3, a signal receiving module 4, an infrared camera 5, an infrared light source 6, an image processing module 7, and a microcontroller 8.

[0052] The flexible sensor 1, in the embodiment, a gallium-indium alloy liquid metal sensor is used, and the resistance will change during stretching.

[0053] The signal processing module 2, in this embodiment, is model ADS7953. It uses built-in circuitry to amplify and filter the electrical signal generated by the flexible sensor, and performs A / D conversion to convert the electrical signal into a digital signal y1.

[0054] In this embodiment, the signal transmitting module 3 uses a CH9140 transmitter to send the digital signal y1 converted by the signal processing module 3 to the signal receiving module 4.

[0055] In this embodiment, the signal receiving module 4 uses a CH9140 receiver to transmit the received digital signal y1 to the microcontroller 8.

[0056] In this embodiment, the infrared camera 5 is an SG2060-K CMOS infrared camera, which acquires infrared motion images of the finger and transmits the images to the image processing module 7.

[0057] In this embodiment, the infrared light source 6 is an MHS-B6W lamp, which is placed at the same location as the infrared camera to illuminate the hand wearing the flexible sensor glove.

[0058] The image processing module 7, in this embodiment, uses a Cyclone IEP4CE10ALTERA FPGA and employs a deep learning algorithm to convert the infrared motion image of the finger captured by the infrared camera into angle data x1 of the finger joint bending.

[0059] The deep learning algorithm, such as Figure 2 As shown, it includes a palm detection model, a hand key point detection model, and an angle calculation model.

[0060] The hand detection model retrieves and matches infrared images of the hand, and obtains the bounding box of the hand through a YOLO network.

[0061] The hand key point detection model operates within the hand bounding box area to obtain the three-dimensional coordinates of the 3D hand key points.

[0062] An angle calculation model converts the three-dimensional coordinates of key points of the 3D hand into the finger joint bending angle x1 through linear algebra or spatial analytic geometry.

[0063] The microcontroller 8, an EFM32 Cortex-M, uses compressed sensing to solve for the digital signal y1 converted by the signal processing module 2 and the angle data x converted by the image processing module 7 to obtain the measurement matrix Φ. It also uses compressed sensing to solve for the digital signal y2 converted by the signal processing module 2 and the measurement matrix Φ to obtain the real-time finger joint bending angle data x2.

[0064] In a second aspect, the present application provides a finger motion tracking method based on a compressed sensing model, which is implemented by using a finger motion tracking system based on the compressed sensing model, as shown in the following steps: Figure 3

[0065] S1: a user wears a glove carrier provided with a flexible sensor, a signal processing module and a signal sending module, and an infrared camera and an infrared light source are arranged on one side of the user's hand motion area;

[0066] S2: when the finger moves, the flexible sensor generates an electrical signal, and the infrared camera synchronously collects an infrared motion image of the finger under the irradiation of the infrared light source;

[0067] S3: the electrical signal collected by the flexible sensor in S2 is converted into a digital signal through A / D conversion, a deep learning algorithm is used to convert the infrared motion image of the finger into finger joint bending angle data, and the digital signal and the finger joint bending angle data are sent to a microcontroller through a signal sending module, and an ISTA iterative solution formula of a compressed sensing algorithm is used to solve to obtain a measurement matrix, which is used for subsequent actual detection;

[0068] In this embodiment, the signal processing module converts the electrical signal generated by the flexible sensor into a digital signal y1, and sends the converted digital signal y1 to the microcontroller through the signal sending module and the signal receiving module;

[0069] After the image processing module converts the infrared motion image of the finger captured by the infrared camera into finger joint bending angle data x using a deep learning algorithm, the image processing module outputs the finger joint bending angle data x to the microcontroller;

[0070] The microcontroller uses compressed sensing to solve the digital signal y1 converted by the signal processing module and the angle data x1 converted by the image processing module, and obtains a measurement matrix Φ;

[0071] The compressed sensing is shown in formula (1) and formula (2):

[0072] y = Φx (1)

[0073] x = Ψs (2)

[0074] Wherein, x is the angle data estimated by the image processing module through the infrared camera, y is the flexible sensor signal, Ψ is a sparse basis, Φ is a measurement matrix, and s is a sparse coefficient;

[0075] The ISTA iterative solution formula of the compressed sensing algorithm is shown in formula (3) and formula (4):

[0076] r (k) = x2 (k-1) - ρΦ T (Φx2 (k-1) -y2) (3)​

[0077]

[0078] wherein k represents the number of iterations, and p represents the iteration step;

[0079] S4: The second time is obtained according to S3 The measurement matrix is used to generate an electrical signal by the flexible sensor, and the infrared camera is used to synchronously collect the infrared motion image of the finger under the irradiation of the infrared light source

[0080] S5: When the finger moves, the signal processing module converts the electrical signal generated by the flexible sensor into a digital signal y2 through A / D conversion, and transmits the converted digital signal y2 to the microcontroller through the signal sending module and the signal receiving module;

[0081] The microcontroller uses the ISTA iterative solving formula of the compressed sensing algorithm to solve the digital signal y2 converted by the signal processing module and the measurement matrix Φ calculated in S3, and obtains the angle data of the real-time finger joint bending.

Claims

1. A finger motion tracking system based on a compressed sensing model, characterized by: The system comprises a flexible sensor, a signal processing module, a signal sending module, a signal receiving module, an infrared camera, an infrared light source, an image processing module and a microcontroller, wherein the flexible sensor, the signal processing module and the signal sending module are arranged at preset positions on a glove carrier, the infrared camera and the infrared light source are arranged on one side of a hand movement area, and the signal receiving module, the image processing module and the microcontroller are arranged on an arm Ultra development board; The flexible sensor is arranged at a preset position on the glove corresponding to a finger joint, and generates an electric signal when the finger moves; The signal processing module amplifies and filters the electric signal of the flexible sensor, and then converts the amplified and filtered electric signal into a digital signal through A / D conversion; The signal sending module transmits the digital signal output by the signal processing module to the signal receiving module; The signal receiving module outputs the digital signal sent by the signal sending module to the microcontroller; The signal sending module and the signal receiving module adopt wired transmission and / or Bluetooth and / or wifi and / or 1.2g / 2.4G / 5.8g wireless transmission; The infrared camera is placed on one side of the hand movement area and is used to collect the infrared image of the hand wearing the flexible sensor glove; The infrared light source is placed with the infrared camera and is used to illuminate the hand wearing the flexible sensor glove; The image processing module calculates the angle data of all joints of the hand through a deep learning algorithm based on the infrared image of the hand collected by the infrared camera, and then outputs the angle data to the microcontroller for processing; The microcontroller compressively senses and solves the digital signal received by the signal receiving module and the angle data output by the image processing module to obtain a measurement matrix, and compressively senses and solves the measurement matrix and the digital signal output by the signal receiving module to reconstruct the angle data.

2. The finger movement tracking system based on the compressive sensing model according to claim 1, wherein the deep learning algorithm comprises a palm detection model, a hand key point detection model and an angle calculation model.

3. The finger movement tracking system based on the compressive sensing model according to claim 2, wherein the palm detection model searches and matches the infrared image of the hand, and obtains a hand bounding box through a yolo network; The hand key point detection model operates in the hand bounding box area to obtain the three-dimensional coordinates of the 3D hand key points; The method comprises the following steps: S1: a user wears a glove carrier provided with a flexible sensor, a signal processing module and a signal sending module, and arranges an infrared camera and an infrared light source on one side of a hand movement area of the user; An angle calculation model converts the three-dimensional coordinates of the 3D hand key points into finger joint bending angles through linear algebra or spatial analytic geometry .

4. A method for finger motion tracking based on a compressed sensing model, implemented by the finger motion tracking system based on a compressed sensing model according to claim 1, characterized in that: S2: when the finger moves, the flexible sensor generates an electric signal, and the infrared camera synchronously collects an infrared movement image of the finger under the irradiation of the infrared light source; S3: the electric signal collected by the flexible sensor in S2 is converted into a digital signal through A / D conversion, a deep learning algorithm is used to convert the infrared movement image of the finger into finger joint bending angle data, the digital signal and the finger joint bending angle data are sent to a microcontroller, and an ISTA iterative solving formula of a compressive sensing algorithm is used to solve a measurement matrix, which is used for subsequent actual detection. ​ ​ S4: Collecting the electrical signal generated by the flexible sensor according to the measurement matrix obtained in S3, and synchronously collecting the infrared motion image of the finger under the irradiation of the infrared light source by the infrared camera; S5: When the finger moves, the signal processing module converts the electric signal generated by the flexible sensor into a digital signal through A / D conversion , and the converted digital signal is transmitted to the microcontroller through the signal sending module and the signal receiving module ; The microcontroller uses the ISTA iterative solution formula of the compressed sensing algorithm to process the digital signal converted by the signal processing module and the measurement matrix calculated in S3 By solving, the angle data of the real-time finger joint bending is obtained .

5. The finger motion tracking method based on the compressed sensing model according to claim 4, characterized in that: The specific process of S3 is: The signal processing module converts the electric signal generated by the flexible sensor into digital signal through A / D conversion , and sends the converted digital signal to the microcontroller through the signal sending module and the signal receiving module . The image processing module uses a deep learning algorithm to convert the infrared motion images of the finger taken by the infrared camera into finger joint bending angle data which is then output to the microcontroller; Microcontroller uses compressed sensing on digital signals converted by signal processing module and angle data converted by image processing module Solve for the measurement matrix .

6. The method of claim 5, wherein: The compressed sensing is shown in formulas (1) and (2): （1） （2） wherein, is angle data estimated by the image processing module through the infrared camera, is a flexible sensor signal, is a sparse basis, is a measurement matrix, is a sparse coefficient.

7. The method of claim 6, wherein: The ISTA iterative solution formula of the compressed sensing algorithm is shown in formulas (3) and (4): （3） （4） Wherein, k represents the number of iterations, and ρ represents the iteration step length.

Images