Gesture signal processing device and processing method based on aperture and pinhole near-field imaging

Abstract

The application discloses a gesture signal processing device based on an aperture and a pinhole near-field imaging and a method thereof, and is characterized in that a pinhole structure and an aperture are arranged; the pinhole structure is arranged on a pinhole light shield plate; the pinhole structure is located in the center of the horizontal direction of the gesture signal processing device; the aperture is arranged on an aperture seat, and the apertures are distributed on both sides of the horizontal direction of the gesture signal processing device; an LED array faces a detected object; and a photoelectric sensor acquires the light emitted by the LED array reflected by the detected object through the pinhole structure and the apertures. The above technical scheme can improve the accuracy of gesture recognition, has the function of measuring human physiological parameters, has the functions of motion error compensation and physiological parameter measurement, and reflects more detailed information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent recognition and processing of gesture signals. More specifically, the present application relates to a gesture signal processing device based on an aperture and a pinhole near-field imaging. The present application also relates to a gesture signal processing method used by the signal processing device. BACKGROUND

[0002] Gesture recognition is an important human-computer interaction technology that can convert human gesture actions into computer-recognizable instructions, enabling intelligent interaction with computers. Gesture recognition technology is widely used, for example, in smart homes, virtual reality, games, and medical fields, and can be used to achieve more intelligent and personalized interaction. With the development of technology and the expansion of application scenarios, gesture recognition technology will become more and more popular and mature, bringing more convenient and intelligent experiences to human-computer interaction.

[0003] The research on gesture recognition can be traced back to the 1980s, but due to the limitations of computer technology and sensor technology at that time, the application range of gesture recognition was very limited. With the continuous development of computer and sensor technology, gesture recognition technology has made great progress and has a more extensive application prospect.

[0004] Currently, gesture recognition technology mainly includes two types: image recognition-based and motion sensor-based:

[0005] 1. Image recognition-based gesture recognition technology uses cameras or depth cameras to capture images or depth images of gesture actions, and then extracts gesture features through image processing and machine learning algorithms for recognition;

[0006] 2. Motion sensor-based gesture recognition technology uses accelerometers, gyroscopes, and other sensor devices to obtain motion data of gesture actions, and then uses signal processing and machine learning algorithms to recognize gestures. Different gesture recognition technologies have their own advantages and disadvantages, but environmental light, background interference, and motion interference are common problems.

[0007] In order to overcome these problems, researchers have proposed many new gesture recognition methods in recent years, such as deep learning-based gesture recognition, sensor and image fusion-based gesture recognition, etc.

[0008] The results of the prior art document search related to the present application are as follows:

[0009] 1. Chinese patent (application) No. 201810224699.8, entitled "Electromyographic signal gesture recognition method based on deep learning and attention mechanism", describes a technical solution as follows:

[0010] The method for recognizing gestures based on deep learning and attention mechanism is as follows: denoising and filtering gesture electromyographic signals; using a sliding window to extract a classic feature set for each window data and constructing a new feature-based electromyographic image; designing a deep learning framework based on convolutional neural networks, recurrent neural networks, and attention mechanisms, and optimizing the network structure parameters; training a classifier model using the designed deep learning framework and training data; inputting test data into the trained deep learning network model, and according to the likelihood output by the last layer, the class corresponding to the maximum likelihood is the recognized class.

[0011] The technical effects recorded therein are:

[0012] The method for recognizing gestures based on deep learning and attention mechanism is as follows: denoising and filtering gesture electromyographic signals; using a sliding window to extract a classic feature set for each window data and constructing a new feature-based electromyographic image; designing a deep learning framework based on convolutional neural networks, recurrent neural networks, and attention mechanisms, and optimizing the network structure parameters; training a classifier model using the designed deep learning framework and training data; inputting test data into the trained deep learning network model, and according to the likelihood output by the last layer, the class corresponding to the maximum likelihood is the recognized class.

[0013] 2. The patent literature with Chinese patent (application) number 202011526884.6: "A safe gesture recognition sensor and a gesture recognition method", the technical solution recorded therein is:

[0014] The safe gesture recognition sensor includes a plurality of groups of sensing units distributed on the boundary lines of the gesture recognition area, at least one confirmation module located in the gesture recognition area, a first signal comparison module, and a second signal comparison module. The first sensing module and the second sensing module in each group of sensing units have overlapping gesture sensing areas in the spatial range. The first signal comparison module is used to receive the sensing signals of the first sensing module and the second sensing module in each group of sensing units and compare them. The second signal comparison module is used to receive the first comparison signal output by the first signal comparison module and the sensing signal output by the confirmation module and compare them.

[0015] The technical effects recorded therein are:

[0016] The method for recognizing gestures based on deep learning and attention mechanism is as follows: denoising and filtering gesture electromyographic signals; using a sliding window to extract a classic feature set for each window data and constructing a new feature-based electromyographic image; designing a deep learning framework based on convolutional neural networks, recurrent neural networks, and attention mechanisms, and optimizing the network structure parameters; training a classifier model using the designed deep learning framework and training data; inputting test data into the trained deep learning network model, and according to the likelihood output by the last layer, the class corresponding to the maximum likelihood is the recognized class.

[0017] 3. The patent literature with Chinese patent (application) number 202210151112.1: "Multi-feature image fusion gesture recognition method based on frequency-modulated continuous wave radar", the technical solution recorded therein is:

[0018] The gesture recognition method based on the frequency modulation continuous wave radar multi-feature image fusion comprises the following steps: step 1, collecting gesture data in real time at the radar end, and pre-processing the gesture data to obtain intermediate frequency signals of the gesture data; step 2, performing feature parameter calculation on the intermediate frequency signals to obtain continuous motion parameters of the gesture; step 3, filtering out incoherent objects and extracting a data segment of interest containing gesture information from the continuous motion parameter data segment; and step 4, constructing a gesture feature image according to the data segment of interest, and inputting the gesture feature image into a shallow convolutional neural network for real-time recognition and classification.

[0019] The technical effect recorded is:

[0020] The present application can generate three feature images of gestures in real time, and perform real-time recognition and classification of gestures according to the feature images.

[0021] However, the above-mentioned disclosed technical solution has not solved the problems and defects of the prior art, such as sensitivity to ambient light, background interference and motion interference. SUMMARY

[0022] The present application provides a gesture signal processing device based on light barrier and pinhole near-field imaging, which aims to improve the accuracy of gesture signal recognition.

[0023] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is:

[0024] The gesture signal processing device based on light barrier and pinhole near-field imaging of the present application comprises an LED array and a photoelectric sensor; the gesture signal processing device is provided with a pinhole structure and a light barrier hole; the pinhole structure is arranged on a pinhole light shield plate; the pinhole structure is located in the center of the horizontal direction of the gesture signal processing device; the light barrier hole is arranged on a light barrier hole seat, and the light barrier hole is distributed on both sides of the horizontal direction of the gesture signal processing device; the LED array faces the detected object; and the photoelectric sensor acquires the light emitted by the LED array reflected by the detected object through the pinhole structure and the light barrier hole.

[0025] The lower end of the light barrier hole, i.e. the entrance of the reflected light, has a larger aperture than the upper end.

[0026] The light barrier hole is arranged obliquely, and the oblique direction is that the lower end of the light barrier hole opens outwardly from the gesture signal processing device.

[0027] The aperture of the pinhole structure is 0.2mm, and the thickness of the pinhole light shield plate is 0.4mm.

[0028] The upper end surface, the lower end surface and the inner hole surface of the small hole structure are provided with a thin layer of light-absorbing material; the light-absorbing material comprises black paint.

[0029] The LED array comprises a plurality of LED light emitting devices, which are arranged on the LED array mounting frame and are distributed at equal intervals.

[0030] The LED light emitting devices emit near-infrared light and green light.

[0031] In order to achieve the same invention purpose as the above technical scheme, the application also provides a gesture signal processing method of the gesture signal processing device based on the diaphragm and the small hole near-field imaging, and the technical scheme is as follows:

[0032] The motion displacement compensation method of the detected object in the signal processing method is: first, the displacement deviation caused by motion is calculated by using the image displacement detection method, and then the signal obtained by the gesture signal processing device is corrected according to the offset.

[0033] The image displacement detection method adopts an optical flow method or an SIFI method.

[0034] The specific steps of the motion displacement compensation method include:

[0035] 1) Feature extraction: extracting key points or feature points or corner points from pictures;

[0036] 2) Feature description: describing the feature points by a group of mathematical vectors;

[0037] 3) Feature matching: distance calculation between feature vectors;

[0038] 4) According to the correction amount, the small hole imaging signal and the diaphragm signal are corrected to eliminate the error caused by the relative motion of the device and the wrist;

[0039] 5) At the same time, the small hole imaging signal and the diaphragm signal are fused to perform gesture recognition, so that the input gesture signal has more abundant information, and then the real-time detection algorithm is used to realize real-time gesture recognition, which can improve the gesture recognition accuracy to a certain extent.

[0040] The present application adopts the technical scheme, adopts a small hole structure, can be imaged based on the small hole near field imaging principle, has the characteristics of motion displacement compensation function in real-time gesture recognition; at the same time, the small hole near field imaging signal and the multi-aperture diaphragm signal are fused, which is used for real-time gesture recognition, can improve the accuracy of gesture recognition; and has the function of human physiological parameter measurement; compared with the wrist finger action recognition device of the prior art, it has the functions of motion error compensation and physiological parameter measurement; for example, compared with the structure using only multi-aperture diaphragm, the results of small hole near field imaging can reflect more detailed information, and the information source is more abundant, and it is more advantageous in motion error elimination, real-time gesture recognition and physiological parameter measurement; the device is worn on the back of the wrist, and the thickness can be as small as 8mm. BRIEF DESCRIPTION OF DRAWINGS

[0041] The contents shown in the drawings and the marks in the drawings are briefly described as follows:

[0042] Figure 1 It is the small hole imaging principle of the present application;

[0043] Figure 2 It is the overall structure of the device of the present application;

[0044] Figure 3 It is the front view of the device of the present application;

[0045] Figure 4 It is the top view of the device of the present application;

[0046] Figure 5(a) is a small hole near field imaging diagram of the back of the wrist in the present application;

[0047] Figure 5(b) is an imaging diagram using optical flow method for displacement compensation.

[0048] The marks in the drawings are:

[0049] 1, small hole structure, 2, photoelectric sensor, 3, diaphragm hole, 4, LED array, 5, small hole light shield, 6, diaphragm hole seat, 7, LED array mounting frame. DETAILED DESCRIPTION

[0050] The specific embodiments of the present application will be further described in detail below with reference to the drawings, and the specific embodiments of the present application will be further described in detail below with reference to the drawings, to help the skilled in the art to have more complete, accurate and in-depth understanding of the inventive concept and technical scheme of the present application.

[0051] As Figure 1 The structure of the present application shown in Figure 5 is a gesture signal processing device based on diaphragm and small hole near field imaging, which includes LED array 4 and photoelectric sensor 2. LED array 4 is used to emit double (multi) wavelength light signal to the back of the wrist. The photoelectric sensor 2 is used to capture the exit light of the light transmission structure, that is, to receive the reflected light signal of the back of the wrist.

[0052] The application aims at the motion interference problem and proposes a gesture signal processing device based on an aperture and a pinhole near-field imaging.

[0053] In order to solve the problems existing in the prior art and overcome its defects, and achieve the purpose of improving the accuracy of gesture signal recognition, the technical scheme adopted by the application is:

[0054] As Figure 2 As shown in Figure 5, the gesture signal processing device based on an aperture and a pinhole near-field imaging of the application is provided with a pinhole structure 1 and an aperture hole 3; the pinhole structure 1 is arranged on a pinhole light shield 5; the pinhole structure 1 is located at the center of the horizontal direction of the gesture signal processing device; the aperture hole 3 is arranged on an aperture hole seat 6, and the aperture hole 3 is distributed on both sides of the horizontal direction of the gesture signal processing device; the LED array 4 faces the detected object; the photoelectric sensor 2 obtains the light emitted by the LED array 4 reflected by the detected object through the pinhole structure 1 and the aperture hole 3.

[0055] The light transmission structure with the pinhole structure 1 and the plurality of aperture holes 3 described above is composed of one pinhole structure 1 and six aperture holes 3, and is used for transmitting gesture signals, physiological signals and the like;

[0056] The light path is as follows: the light emitted vertically by the multi (dual) wavelength LED array 4 on the back side of the wrist passes through the muscle, blood vessel and the like in the back side of the wrist and is reflected, and then a part of the light is received by the photoelectric sensor 2 through the pinhole structure 1 to obtain a pinhole imaging signal; another part of the light is received by the photoelectric sensor 2 through the six inclined aperture holes 3 to obtain a reflected light signal, so that the signal source is more abundant and more information is obtained.

[0057] The position of the pinhole structure 1 is located at the center of the entire light transmission structure, and is aimed at obtaining a larger field of view and a more complete signal. It is known from actual tests that the imaging field of view of the pinhole structure 1 of the gesture signal processing device described in the application is about 8 mm in the case of a smaller thickness (about 8 mm).

[0058] The application can obtain image information of the back side of the wrist based on the imaging of the pinhole near field, and the signal is further processed and used for various functions, such as motion displacement error compensation, real-time gesture recognition, human physiological parameter measurement and the like.

[0059] The lower port of the aperture hole 3, i.e. the entrance of the reflected light, has a larger aperture than the upper port.

[0060] The aperture hole 3 is arranged to be inclined, and the inclined direction is that the lower end of the aperture hole 3 opens outwardly of the gesture signal processing device.

[0061] The central pinhole structure 1 is used for pinhole imaging; the aperture of the pinhole structure 1 is 0.2mm; the thickness of the pinhole light shield 5 is 0.4mm.

[0062] The six oblique light aperture holes 3 on the edge are used for conducting the reflected light signals on the back of the wrist.

[0063] Since light has wave-particle duality, the following explains from two angles of geometric optics and wave optics:

[0064] 1. Analyze the principle of pinhole imaging from the angle of geometric optics, as shown in Figure 1

[0065] Light propagates along a straight line, and each point of the object can be regarded as a light source point. After passing through the pinhole, a light spot is obtained, and an image is formed by superimposing infinite light spots. Therefore, there is distortion, and the size of the light spot should be reduced as much as possible to obtain a clearer image.

[0066] As shown in Figure 1 , wherein the actual magnification β is

[0067]

[0068] It is easy to conclude that the smaller the aperture d of the pinhole structure 1, the clearer the imaging is;

[0069] 2. From the angle of wave optics, the smaller the aperture, the more obvious the diffraction, and the more blurred the imaging is. Under the condition of near field and small aperture, Fresnel diffraction will be very obvious, and the boundary of the light spot formed by one point is no longer bright and dark, resulting in more serious distortion.

[0070] Through a plurality of experiments, the imaging clarity of the pinhole structure 1 with different aperture sizes is analyzed, and it is proved that when the diameter of the pinhole structure 1 takes an appropriate value, the imaging quality is best.

[0071] The upper end surface, the lower end surface and the inner hole surface of the pinhole structure 1 are provided with a thin layer of light-absorbing material; the light-absorbing material includes black paint.

[0072] The upper surface and the lower surface of the pinhole structure 1 and its vicinity are provided with a thin coating layer coated with light-absorbing material, such as black paint, to prevent the influence of light transmission on the imaging effect.

[0073] ​A thin layer of light-absorbing material, such as black paint, is applied to the upper and lower surfaces of the pinhole structure 1 and its vicinity to prevent light transmission from affecting the imaging effect. Based on the pinhole imaging principle, to ensure that light can only enter through the opening for imaging, the area around the pinhole must be opaque. If other light-transmitting areas exist near the pinhole, light will also scatter from these areas; this scattered light will interfere with the imaging process, affecting the image quality and sharpness. Therefore, in pinhole imaging, it is essential to ensure that the area near the pinhole is opaque. Applying black paint is a simple and effective way to suppress scattered light, ensuring that the imaging accuracy is not compromised.

[0074] The LED array 4 includes multiple LED light-emitting devices, which are arranged on the LED array mounting frame 7 and distributed at equal intervals.

[0075] like Figure 1 As shown:

[0076] The aforementioned pinhole structure 1 is based on the pinhole near-field imaging principle and can obtain imaging information of the back of the wrist. After further processing, the signal is used for various functions, such as motion displacement error compensation, real-time gesture recognition, and measurement of human physiological parameters.

[0077] The LED light-emitting device emits near-infrared light and green light.

[0078] Based on pinhole near-field imaging, this invention realizes the principle of measuring finger gesture movements and their displacement compensation on the dorsal side of the wrist:

[0079] LED light source array ( Figure 2 The LED array 4) emits near-infrared and green light vertically toward the back of the wrist. The light reaches the skin, and the near-infrared light can reach the muscle layer with very little loss (including absorption and scattering). It is reflected back in large quantities in the muscle layer, and the amount of light reflected changes with the contraction and relaxation of the muscle tissue. Hemoglobin has a good absorption capacity for green light.

[0080] To achieve the same inventive objective as the aforementioned technical solutions, this invention also provides a gesture signal processing method for the gesture signal processing device based on aperture and pinhole near-field imaging, the technical solution of which is as follows:

[0081] The motion displacement compensation method for the detected object in the signal processing method is as follows: first, the displacement deviation caused by motion is calculated using an image displacement detection method, and then the signal obtained by the gesture signal processing device is corrected according to the offset.

[0082] The image displacement detection method described above employs either optical flow or SIFI.

[0083] The specific steps of the motion displacement compensation method include:

[0084] 1. Feature extraction: extracting key points or feature points or corner points from the picture;

[0085] 2. Feature description: describing the feature points with a set of mathematical vectors;

[0086] 3. Feature matching: distance calculation between feature vectors.

[0087] 4. Then, the pinhole imaging signal and the diaphragm signal are corrected according to the correction amount to eliminate the error caused by the relative motion of the device and the wrist;

[0088] 5. Meanwhile, the pinhole imaging signal and the diaphragm signal are fused to perform gesture recognition, so that the input gesture signal has more abundant information, and real-time gesture recognition is realized by using a real-time detection algorithm, which can improve the gesture recognition accuracy to a certain extent.

[0089] The specific image of displacement detection is shown in FIGS. 5(a) and 5(b).

[0090] Therefore, the light returned from the detected object contains rich finger motion information. Part of the reflected light enters the pinhole structure 1, and part of the reflected light passes through the 6 diaphragm inclined holes 3 around the pinhole, and is received by the optical sensor 2 at the same time, Figure 2 ) to obtain two parts of signals.

[0091] Firstly, the displacement compensation algorithm is used for pinhole near-field imaging to detect the motion offset of the entire signal (for example, typical algorithms: optical flow method, SIFI, etc.) and perform motion compensation;

[0092] Secondly, the two parts of signals after correction are fused to be used as input signals for gesture recognition.

[0093] Finally, the fused signals are subjected to a real-time gesture recognition algorithm for gesture recognition and classification.

[0094] The gesture signal processing device of the application can also measure human physiological parameters such as heart rate and blood oxygen parameters.

[0095] Common measurement methods such as frequency model based methods including band-pass filtering, wavelet decomposition and empirical mode decomposition, decompose the observed color trace into components with different frequency bands, select the components falling within the human pulse frequency band (0.4-4.0 Hz) as physiological signals, extract the pulse wave, and thus measure the physiological parameters.

[0096] Image processing of pinhole near-field imaging signals: pre-processing of the image, such as smoothing, filtering, edge detection, etc., to eliminate noise and enhance target features.

[0097] Then, feature extraction: according to the required physiological parameters, the pulse wave is extracted for the specific region in the image.

[0098] Final data analysis: the required physiological parameters are obtained through data analysis technology.

[0099] The application is described above with reference to the drawings, and it is obvious that the specific implementation of the application is not limited by the above method, as long as various non-essential improvements are made by using the method concept and technical solution of the application, or the concept and technical solution of the application is directly applied to other occasions without improvement, which is within the protection scope of the application.

Claims

1. A gesture signal processing device based on aperture and pinhole near-field imaging, comprising an LED array (4) and a photoelectric sensor (2); characterized in that: the gesture signal processing device is provided with a pinhole structure (1) and an aperture hole (3); the pinhole structure (1) is arranged on a pinhole light shield (5); the pinhole structure (1) is located in the center of the gesture signal processing device in the horizontal direction; the aperture hole (3) is arranged on an aperture hole seat (6), and the aperture hole (3) is distributed on both sides of the gesture signal processing device in the horizontal direction; the LED array (4) faces a detected object; and the photoelectric sensor (2) acquires light emitted by the LED array (4) reflected by the detected object through the pinhole structure (1) and the aperture hole (3); the gesture signal processing device has a light transmission structure with the pinhole structure (1) and the aperture hole (3), which is composed of one pinhole structure (1) and six aperture holes (3) and is used for transmitting gesture signals and physiological signals; the light path is as follows: light emitted vertically by the multi-wavelength LED array (4) is reflected by muscle and blood vessel tissues in the back of a wrist, part of the light is received by the photoelectric sensor (2) through the pinhole structure (1) to obtain pinhole imaging signals, and the other part of the light is received by the photoelectric sensor (2) through the six inclined aperture holes (3) to obtain reflected light signals, so that the signal source is more abundant and more information is obtained; the lower end of the aperture hole (3), i.e. the entrance of the reflected light, has a larger aperture than the upper end; and the aperture hole (3) is arranged to be inclined, and the inclined direction is that the lower end of the aperture hole (3) opens outward of the gesture signal processing device.

2. The gesture signal processing apparatus based on aperture and pinhole near field imaging according to claim 1, characterized in that: The aperture of the pinhole structure (1) is 0.2 mm, and the thickness of the pinhole light shield (5) is 0.4 mm.

3. The gesture signal processing apparatus based on aperture and pinhole near field imaging according to claim 1, characterized in that: The upper end surface, the lower end surface and the inner hole surface of the pinhole structure (1) are all provided with a thin layer of light-absorbing material; and the light-absorbing material includes black paint.

4. The gesture signal processing apparatus based on aperture and pinhole near field imaging according to claim 1, characterized in that: The LED array (4) includes a plurality of LED light emitting devices, which are arranged on an LED array mounting frame (7) and are distributed at equal intervals.

5. The gesture signal processing apparatus based on aperture and pinhole near field imaging according to claim 4, characterized in that: The LED light emitting devices emit near-infrared light and green light.

6. The processing method of gesture signal processing apparatus based on aperture and pinhole near-field imaging according to any one of claims 1 to 5, characterized in that: In the signal processing method, a motion displacement compensation method for a detected object is as follows: first, a displacement deviation caused by motion is calculated by using an image displacement detection method, and then the signal acquired by the gesture signal processing device is corrected according to the offset.

7. The processing method of gesture signal processing apparatus based on aperture and pinhole near-field imaging according to claim 6, characterized in that: The image displacement detection method adopts an optical flow method or an SIFI method.

8. The processing method of gesture signal processing apparatus based on aperture and pinhole near-field imaging according to claim 7, characterized in that: The specific steps of the motion displacement compensation method include:

1. feature extraction: extracting key points or feature points or corner points from a picture; 2. feature description: describing the feature points by using a set of mathematical vectors; 3. feature matching: calculating the distance between the feature vectors; 4. correcting the pinhole imaging signals and the aperture signals according to the correction amount to eliminate errors caused by relative motion between the device and the wrist. 5), At the same time, the pinhole imaging signal and the diaphragm signal are fused to perform gesture recognition, so that the input gesture signal has more abundant information, and real-time gesture recognition is realized by using a real-time detection algorithm, and the gesture recognition accuracy can be improved to a certain extent.

Images