A gesture interaction system and method based on multi-channel audio acquisition device

Abstract

The present invention discloses a gesture interaction system and method based on a multi-channel audio acquisition device, including a sensor module integrating at least one ultra-small wireless audio acquisition unit, an information processing module, and an interaction, rendering and display module, obtaining feature data based on audio signals, and using a domain adaptive neural network model for recognition to identify target gestures. Users can quickly, accurately and comfortably control virtual objects, perform gesture recognition and text input by wearing the ultra-small audio acquisition module on their wrists or placing it on an interactive plane, directly converting their hands into interactive planes, enhancing the performance and efficiency of the interactive system, and providing a new method that is efficient, natural and comfortable. The present invention is not only applicable to old electronic devices such as mobile phones and desktop computers, but also can meet the interaction needs of emerging electronic systems such as VR and AR, and can quickly convert multiple planes into interactive areas to improve user interaction efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction technology, and in particular to a gesture interaction system and method based on a multi-channel audio acquisition device. Background Art

[0002] At present, some studies have proposed methods and devices for gesture recognition with the help of sound signals. Most existing methods use ultrasonic sensors to detect the shape and movement trajectory of the hand. However, due to the influence of device size, manufacturing process, etc., ultrasonic sensors will cause problems such as difficult equipment layout and cumbersome signal collection steps, which further leads to problems such as low gesture recognition accuracy and poor interactive performance, greatly affecting the user's interaction efficiency.

[0003] A study has proposed a method for identifying gestures based on receiving audio signals after they have been transmitted through a target gesture made by a user. This method provides an audio-based gesture recognition method, which performs channel estimation based on the original audio signal and the target audio signal to obtain feature data of the channel estimation, and uses a pre-built domain adaptive neural network model to identify the feature data of the channel estimation to obtain the recognition result of the target gesture. However, in this method, the transmission and reception of audio signals and modulated signals require multiple devices to complete, and the different areas of gesture operation will lead to increased errors in audio signal channel estimation. Therefore, the use of this technical solution will lead to excessive complexity of the system, the application conditions will be greatly restricted, and there will be many constraints on the gesture operation area, resulting in large recognition errors.

[0004] At present, there are studies that use ultrasonic transmitters to transmit ultrasonic signals and receive reflected ultrasonic signals from the hands, configure the ultrasonic transmitter and receiver to form a first reference line, process the reflected ultrasonic signals, and determine the one-dimensional gesture corresponding to the projection trajectory of the object moving on the first reference line through the time-frequency distribution of the reflected ultrasonic signals, thereby realizing gesture recognition in a non-contact manner. The device uses the speaker on the mobile phone to provide ultrasonic signals, and introduces multiple ultrasonic transceiver devices to improve the resolution of gesture recognition. However, the gesture recognition resolution of a single ultrasonic transmitter is low, and the recognition accuracy is not high. However, most mobile phones on the market do not have a multi-speaker configuration. Improving the resolution of gesture recognition requires additional configuration of ultrasonic transmitters. Therefore, the scalability of this method is general, and it cannot guarantee that the complexity of the device will be reduced while improving the recognition accuracy.

[0005] Therefore, there is an urgent need for a gesture interaction system that can reduce gesture recognition errors and reduce the complexity of the device while taking into account the comfort and convenience of user use. Summary of the invention

[0006] In view of this, the present invention provides a gesture interaction system and method based on a multi-channel audio acquisition device, which can take into account the comfort and convenience of user use while reducing the error of gesture recognition and the complexity of the device.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is:

[0008] An ultra-small wireless audio acquisition unit includes a high-definition microphone, a lower-computer processor and a wireless communication unit; the high-definition microphone collects the sound generated by the touch gesture of the finger on the interactive plane, converts it into an audio analog signal, and transmits it to the lower-computer processor; the lower-computer processor converts the audio analog signal into an audio digital signal, and wirelessly transmits the audio digital signal through the wireless communication unit.

[0009] A gesture interaction system based on a multi-channel audio acquisition device is aimed at an ultra-small wireless audio acquisition unit and includes a sensor module, an information processing module, and an interaction, rendering and display module.

[0010] At least one ultra-small wireless audio acquisition unit is integrated in the sensor module. The high-definition microphone collects the sound generated by the finger touching the interactive plane at different positions, and converts it into an audio analog signal in the lower-computer processor of each ultra-small wireless audio acquisition unit; the lower-computer processor converts the audio analog signal into an audio digital signal and transmits it to the wireless communication unit; the wireless communication unit wirelessly transmits the audio digital signal to the information processing module.

[0011] In the information processing module, the host computer processor receives the audio digital signals from all ultra-small wireless audio acquisition units, converts them into spectrum graphs, and compresses the spectrum graphs. After that, they are spliced ​​together to obtain a spliced ​​spectrum graph which is input into the trained neural network for classification. The neural network outputs the gesture classification result. Where n is the number of spectrum graphs.

[0012] In the interaction, rendering and display module, the image renderer renders the virtual scene, and the gesture classification is transmitted to the image renderer. The image renderer controls the movement of virtual objects in the virtual scene according to the gesture classification results and displays them on the image display.

[0013] Furthermore, the lower processor includes a digital audio power amplifier chip and a peripheral analog-to-digital conversion circuit, and the upper processor is a PC or a mobile phone.

[0014] Furthermore, the interaction, rendering and display module includes an image display and an image renderer. The image display is a PC or a VR / AR helmet, and the graphics renderer is a Unity 3D platform.

[0015] A gesture interaction method based on a multi-channel audio acquisition device, for a system, specifically comprising the following steps:

[0016] When the gesture interaction system is powered on, the high-definition microphone in the sensor module converts the sound generated by the touch gesture of the finger on the interactive plane into an audio analog signal; the lower-level processor converts the audio analog signal into an audio digital signal, adds a packet header, and puts it into the buffer of the lower-level processor; the audio digital signal is transmitted to the upper-level processor through the wireless communication unit; the upper-level processor filters the audio digital signal and cuts it into 1.5s segments, extracts the frequency characteristics to obtain a spectrum diagram; the upper-level processor receives the audio digital signals of all ultra-small wireless audio acquisition units, converts the spectrum diagram, and compresses the spectrum diagram The gestures are then spliced ​​together to obtain a spliced ​​spectrum map, which is then input into the trained neural network, where n is the number of spectrum maps. The neural network classifies and recognizes the spliced ​​spectrum map to obtain a gesture classification result. The image renderer renders a virtual scene, and the gesture classification result is transmitted to the image renderer. The image renderer controls the movement of virtual objects in the virtual scene according to the gesture classification result and displays it on the image display.

[0017] In the lower computer processing module, when the audio analog signal is greater than the noise threshold, the lower computer processor processes the audio analog signal as no audio data and performs filtering processing, and does not transmit it to the upper computer processor.

[0018] Furthermore, the neural network is trained using supervised learning.

[0019] Furthermore, there are 14 gesture categories, namely: swipe up, swipe down, swipe left, swipe right, single-click, double-click, triple-click, draw an upper semicircle from the right, draw an upper semicircle from the left, draw a left semicircle from the top, draw a left semicircle from the bottom, draw a right semicircle from the top, draw a right semicircle from the bottom, and draw a V-shaped line.

[0020] Beneficial effects:

[0021] 1. The present invention provides an ultra-small wireless audio acquisition unit, which integrates a high-definition microphone, a lower-level processor and a wireless communication unit in a very small device to form an ultra-small audio acquisition sensor, which collects sound signals generated by users performing touch gesture operations such as sliding, tapping, and clicking on various interactive planes, thereby miniaturizing the acquisition equipment and broadening its practicality. Users can quickly, accurately and comfortably control virtual objects, perform gesture recognition, and input text by wearing the ultra-small audio acquisition module on their wrists or placing it on the interactive plane, thereby greatly enhancing the performance and efficiency of the interactive system.

[0022] 2. Based on an ultra-small wireless audio acquisition unit, the present invention provides a gesture interaction system based on a multi-channel audio acquisition device, including a sensor module that integrates at least one ultra-small wireless audio acquisition unit, an information processing module, and an interaction, rendering and display module. The present invention uses a wireless transmission communication method to transmit audio signals to form a separate system, that is, the upper computer and the lower computer are connected in the form of wireless communication, forming a lightweight and easily expandable gesture recognition system, which is more convenient for users to use. The present invention obtains a spectrum graph based on the audio signal, and uses a pre-built domain adaptive neural network to accurately identify the spectrum graph to obtain a gesture classification. In addition, the system can not only be applied to old electronic devices such as mobile phones and desktop computers, but also meet the interaction needs of emerging electronic systems such as VR and AR. It can quickly convert various planes in life into effective interaction areas, thereby improving the user's interaction efficiency.

[0023] 3. For the above system, the present invention provides a gesture interaction method based on a multi-channel audio acquisition device, which detects the sound signals generated when the user uses his fingers to slide, tap, click, etc. on the interactive plane, filters, amplifies and splices the audio signals collected by the ultra-small wireless audio acquisition unit, and inputs the spectrum diagram into a trained deep neural network structure for classification and recognition, thereby determining the user's gesture classification at this time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a system composition diagram of the present invention.

[0025] Figure 2 This is a software flow chart of the ultra-small audio acquisition device proposed by the present invention.

[0026] Figure 3 Schematic diagram of an ultra-small audio acquisition device worn on the wrist.

[0027] Figure 4 This is the floor plan of the ultra-small audio acquisition device.

[0028] Figure 5 The figure is a flow chart of the gesture recognition method.

[0029] Figure 6 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0031] like Figure 3 As shown, the present invention provides an ultra-small wireless audio acquisition unit, comprising a high-definition microphone, a lower-level processor and a wireless communication unit. In the embodiment of the present invention, the parameters of the high-definition microphone are shown in Table 1.

[0032] Table 1 High Definition Microphone Parameters

[0033] Sampling frequency 44.1kHz Sampling Depth 16-bit Bitrate 0.7Mbps

[0034] The ultra-small wireless audio acquisition unit is fixed to the user's wrist in the form of a wristband, or is arranged on a desktop, wall, head-mounted display device and other surfaces by pasting, to detect the sound generated when the user uses his fingers to slide, tap, click and other touch gestures on the back of the hand / palm, and various planes. After filtering, amplifying and splicing the sound collected by the module, its spectrum map is input into the trained deep neural network structure for classification and recognition, and then the user's gesture operation at this time is determined. At the same time, since gesture operations performed at different positions will generate different sound signals relative to multiple audio acquisition modules, the area where the gesture operation is located can be obtained by jointly classifying the sound signals collected by multiple modules, so as to facilitate the division of the gesture operation space. In an embodiment of the present invention, there are one or two ultra-small wireless audio acquisition units.

[0035] like Figure 1 As shown, based on an ultra-small wireless audio acquisition unit, the present invention provides a gesture interaction system based on a multi-channel audio acquisition device, including a sensor module, an information processing module, and an interaction, rendering and display module.

[0036] At least one ultra-small wireless audio acquisition unit is integrated in the sensor module. The high-definition microphone collects the sound generated by the touch gesture of the finger on the interactive plane at different positions, and converts it into an audio analog signal in the lower-computer processor of each ultra-small wireless audio acquisition unit; the lower-computer processor converts the audio analog signal into an audio digital signal and transmits it to the wireless communication unit; the wireless communication unit wirelessly transmits the audio digital signal to the information processing module. The software flow chart is as follows: Figure 2 In the embodiment of the present invention, the lower computer processor includes a digital audio power amplifier chip and a peripheral analog-to-digital conversion circuit.

[0037] In the information processing module, the host computer processor receives the audio digital signals from all ultra-small wireless audio acquisition units, converts the spectrum, and compresses the spectrum. After splicing, a spliced ​​spectrum graph is obtained and input into the trained neural network for classification, and the neural network outputs the gesture classification; where n is the number of spectrum graphs. The information processing module is mainly composed of the host computer processor. The hardware of the host computer processor can use processing units such as PC, mobile phone and head-mounted display integrated machine; the software part of the host computer processor is mainly composed of neural networks. The host computer processor receives the audio signals of each microphone sent by the lower computer processor in real time, and makes a set of audio signals received at the same time into a spectrum graph. The host computer processor inputs the spectrum graph into the trained neural network for classification, and finally outputs the sliding or tapping gesture made by the current finger. Among them, the training of the neural network adopts a supervised learning method.

[0038] In the interaction, rendering and display module, the image renderer renders the virtual scene, and the gesture classification is transmitted to the image renderer. The image renderer controls the movement of virtual objects in the virtual scene according to the gesture classification and displays them on the image display. Figure 4 and Figure 5 As shown, the image renderer can control the moving direction of the virtual object according to the user's directional gesture operation, realize text input according to the user's number and letter writing gestures, and interact with the system interface (audio and video APP, entertainment games, text browsing) according to the user's tapping, sliding and other gesture operations, and display them on the image display.

[0039] This system utilizes touch gestures between human fingers and various planes, and can achieve efficient, accurate and fast interaction by processing and classifying the sound signals generated by gesture operations. It can be used for augmented reality, virtual reality, tablets, smart watches and interactions on various planes, including but not limited to controlling the movement of virtual objects, and inputting text with a virtual keyboard.

[0040] The hardware part of the interaction, rendering and display module can use a virtual reality head-mounted display such as HTC VIVE, or an augmented reality head-mounted display such as Hololens, or a mobile phone, personal computer display and other devices. The software part of the interaction, rendering and display module can use a rendering engine such as Unity 3D or UE 4. The interaction, rendering and display module inputs the direction information obtained in the information classification processing module into the rendering engine, and controls the virtual objects in the virtual environment and performs functions such as sliding gesture recognition. In addition to realizing the control of virtual objects and sliding gesture recognition, users can also use virtual keyboards and virtual cursors to realize functions such as text input.

[0041] like Figure 2 and Figure 6As shown, the present invention proposes a gesture interaction method based on a multi-channel audio acquisition device. For the above system, the specific steps include:

[0042] When the gesture interaction system is powered on, the high-definition microphone in the sensor module converts the sound generated by the touch gesture of the finger on the interactive plane into an audio analog signal; the lower-level processor converts the audio analog signal into an audio digital signal, adds a packet header, and puts it into the buffer of the lower-level processor; the audio digital signal is transmitted to the upper-level processor through the wireless communication unit. In the lower-level processing module, when the audio analog signal is greater than the noise threshold, the lower-level processor processes the audio analog signal as audio-free data and filters it, and does not transmit it to the upper-level processor.

[0043] The host computer processor filters the audio digital signal and cuts it into 1.5s segments, extracts the frequency characteristics to obtain the spectrum graph; the host computer processor receives the audio digital signal of all ultra-small wireless audio acquisition units, converts the spectrum graph, and compresses the spectrum graph After that, they are spliced ​​together to obtain a spliced ​​spectrum graph which is input into the trained neural network, where n is the number of spectrum graphs. The neural network classifies and recognizes the spliced ​​spectrum graph to obtain the gesture classification.

[0044] The image renderer renders a virtual scene, and the gesture classification is transmitted to the image renderer. The image renderer controls the movement of virtual objects in the virtual scene according to the gesture classification and displays them on the image display.

[0045] There are 14 types of gestures: swipe up, swipe down, swipe left, swipe right, single-click, double-click, triple-click, draw an upper semicircle from the right, draw an upper semicircle from the left, draw a left semicircle from the top, draw a left semicircle from the bottom, draw a right semicircle from the top, draw a right semicircle from the bottom, and draw a V-shaped line.

[0046] The above steps are in no particular order.

[0047] In the embodiment of the present invention, the software part of the host computer processor is mainly composed of a neural network. The host computer processor receives the audio digital signals of all ultra-small wireless audio acquisition units, converts the spectrum graph, and compresses the spectrum graph. After splicing, a spliced ​​spectrum graph is obtained and input into the trained neural network, where the training method of the neural network is:

[0048] Step 1: Collect training data. First, divide the data set according to the gesture classification results, generate folders named 0-13, put the spectrograms corresponding to the same category in the same folder, and divide them into training set (accounting for 80% of the data set), test set (accounting for 10% of the data set) and validation set (accounting for 10% of the data set). Then put the training set in the file address data / splitData / train, the test set in the file address data / splitData / test, and the validation set in the file address data / splitData / valid.

[0049] Step 2: Design the structure of the neural network. The present invention adopts the network structure of MobileNetV3, as shown in Table 2.

[0050] Table 2 Neural network structure

[0051]

[0052] Among them, the first column Input represents the shape change of each feature layer of the neural network, and the spectrum graph is split according to the number of channels specified in the input; the second column Operator represents the block structure that the feature layer in the spectrum graph is about to experience, and the present invention adopts the bneck structure for feature extraction; the third and fourth columns respectively represent the number of channels after the inverse residual structure in the bneck structure is raised and the number of channels of the feature layer when input to the bneck structure; the fifth column SE represents whether the attention mechanism is introduced in the feature layer, and the attention mechanism compensates for missing, repeated or discontinuous features; the sixth column NL represents the type of activation function, HS represents h-swish, RE represents RELU, and the activation function maps the feature to the output end; the seventh column s represents the step size used for each block structure.

[0053] Step 3: Weight initialization: Batch-size is set to 50; Max_epoch is set to 64; LR is set to 0.0001; Log_interval is set to 3; Val_interval is set to 1.

[0054] Step 4: Training of the neural network. Input the training set, test set, and validation set into the neural network in order, and obtain the output after the neural network calculation. Under the label of each gesture classification result, calculate the loss function, and then use the back propagation algorithm to update the neuron weights of the neural network, and then repeat the previous process until the loss value reaches the expected effect. The neuron weights will be saved in the weights directory, best.pkl represents the best result, and last.pkl represents the weight obtained in the last epoch. When a complete data set passes through the neural network once and returns once, this process is called an epoch.

[0055] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A gesture interaction system based on a multi-channel audio acquisition device, characterized in that: It includes an ultra-small wireless audio acquisition unit, a sensor module, an information processing module, and an interaction and rendering and display module; The ultra-small wireless audio acquisition unit is fixed to the user's wrist in the form of a wristband, or is placed on various surfaces by pasting, to detect the sound generated when the user uses his fingers to perform touch gestures on various interactive planes; the interactive planes include desktops, walls, and head-mounted display device surfaces; it includes a high-definition microphone, a lower-level processor, and a wireless communication unit; the high-definition microphone collects the sound generated by the finger's touch gestures on the interactive plane, converts it into an audio analog signal, and transmits it to the lower-level processor; The lower computer processor converts the audio analog signal into an audio digital signal, and transmits the audio digital signal wirelessly through the wireless communication unit; In the sensor module, at least one ultra-small wireless audio acquisition unit is integrated. The high-definition microphone collects the sound generated by the gesture of the finger touching the interactive plane at different positions, and converts it into an audio analog signal in the lower-computer processor of each ultra-small wireless audio acquisition unit; the lower-computer processor converts the audio analog signal into an audio digital signal and transmits it to the wireless communication unit; the wireless communication unit wirelessly transmits the audio digital signal to the information processing module; In the information processing module, the host computer processor receives the audio digital signals of all ultra-small wireless audio acquisition units, converts them into a spectrum, and compresses the spectrum After splicing, a spliced ​​spectrum graph is obtained and input into the trained neural network for classification, and the neural network outputs the gesture classification result; among them, n is the number of spectrum graphs; In the interaction, rendering and display module, the image renderer renders a virtual scene, the gesture classification is transmitted to the image renderer, and the image renderer controls the movement of the virtual object in the virtual scene according to the gesture classification result and displays it on the image display.

2. The system according to claim 1, characterized in that The lower processor includes a digital audio power amplifier chip and a peripheral analog-to-digital conversion circuit, and the upper processor is a PC or a mobile phone.

3. The system according to claim 1, characterized in that The interaction, rendering and display module includes an image display and an image renderer. The image display is a PC or a VR / AR helmet, and the graphics renderer is a Unity 3D platform.

4. A gesture interaction method based on a multi-channel audio acquisition device, characterized in that: For the system according to any one of claims 1 to 3, the specific steps include: When the gesture interaction system is powered on, the high-definition microphone in the sensor module converts the sound generated by the touch gesture of the finger on the interactive plane into an audio analog signal; the lower-level processor converts the audio analog signal into an audio digital signal, adds a packet header, and puts it into the buffer of the lower-level processor; the audio digital signal is transmitted to the upper-level processor through the wireless communication unit; the upper-level processor filters the audio digital signal and cuts it into 1.5s segments, extracts the frequency characteristics to obtain a spectrum diagram; the upper-level processor receives the audio digital signals of all ultra-small wireless audio acquisition units, converts the spectrum diagram, and compresses the spectrum diagram After splicing, a spliced ​​spectrum graph is obtained and input into the trained neural network. n is the number of spectrum graphs; the neural network classifies and recognizes the spliced ​​spectrum graphs to obtain the gesture classification result; the image renderer renders the virtual scene, the gesture classification result is transmitted to the image renderer, and the image renderer controls the virtual object to move in the virtual scene according to the gesture classification result and displays it on the image display; In the lower computer processing module, when the audio analog signal is greater than the noise threshold, the lower computer processor processes the audio analog signal as no audio data and performs filtering processing, and does not transmit it to the upper computer processor; The network structure of MobileNetV3 is adopted, and the attention mechanism is introduced to compensate for missing, repeated or discontinuous features.

5. The method according to claim 4, characterized in that Neural networks are trained using supervised learning.

6. The method according to claim 5, characterized in that There are 14 types of gesture categories, namely: swipe up, swipe down, swipe left, swipe right, single-click, double-click, triple-click, draw an upper semicircle from the right, draw an upper semicircle from the left, draw a left semicircle from the top, draw a left semicircle from the bottom, draw a right semicircle from the top, draw a right semicircle from the bottom, and draw a V-shaped line.

Images