Hanging hand touch interaction method for augmented reality interaction

Abstract

The invention relates to the field of augmented reality including virtual reality (VR), augmented reality (AR) and mixed reality (MR), and discloses a hand-hanging touch interaction method for augmented reality interaction and an interaction system thereof.The hand-hanging touch interaction method comprises the following steps that S1, a mobile phone terminal collects audio signals generated when a user knocks or slides a mobile phone in a pocket across clothes; s2, the collected audio signals are processed and classified, the interaction intention of a user is recognized, and signals corresponding to the recognized interaction intention are sent to augmented reality equipment in a wireless communication mode; by knocking and sliding the mobile phone in the pocket across the clothes, the user does not need to lift the hand or directly operate the mobile phone, and limb fatigue and social embarrassment caused by hand lifting interaction are avoided. The interaction mode conforms to natural behavior postures and is particularly suitable for crowded spaces or public places, the user can complete interaction operation without attracting attention of other people, and potential safety hazards possibly caused by direct operation of a mobile phone can be avoided in some specific scenes (such as riding and driving).

Description

Technical Field

[0001] The present invention relates to the field of extended reality, and particularly to a hanging - hand touch interaction method for extended reality interaction. Background Art

[0002] In the current technological application scenarios, the applications of virtual reality (VR) and augmented reality (AR) technologies are becoming increasingly widespread. Especially in the fields of entertainment and life assistance, AR glasses have gradually gained recognition in the consumer market. When users use AR glasses, they often need to cooperate with external devices to achieve various preset functions. As a common computing terminal device, when a mobile phone is used in combination with AR glasses, there are many interaction problems.

[0003] Existing extended reality interaction technologies have various drawbacks. In terms of interaction methods, raising - hand interaction is likely to cause physical fatigue to users and may also lead to social embarrassment in public places, affecting the user experience. From the aspects of interaction accuracy and efficiency, when existing pinching interactions are used for fine operations in an extended reality environment, unexpected displacements and jitters are likely to occur, and they cannot meet the requirements of precise interaction tasks such as fine drawing, twisting and placing of small objects. In the research on the suitability of wearable devices and extended reality device interactions, although some scholars have discussed aspects such as device comfort and wearing position, there is less research on mobile phones as the interaction terminal of extended reality devices, and the above - mentioned problems have not been effectively solved. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the present invention provides a hanging - hand touch interaction method for extended reality interaction, which solves the above problems.

[0005] To achieve the above objectives, the present invention is realized through the following technical solutions: A hanging - hand touch interaction method for extended reality interaction and its interaction system, including the following steps: S1. The mobile phone end collects the audio signals generated by the user knocking or swiping on the mobile phone in the pocket through clothes; S2. Process and classify the collected audio signals, identify the user's interaction intention, and send the signal corresponding to the identified interaction intention to the extended reality device through wireless communication; S3. The extended reality device receives the signal and realizes the corresponding interaction function according to the signal content in combination with the cursor position controlled by the head.

[0006] Preferably, in the step of collecting audio signals, the pyaudio library is used to monitor the mobile phone microphone, the audio sampling format is set to 16 - bit PCM audio format, mono recording, the sampling rate is 11025Hz, and the audio data buffer size is 20ms.

[0007] Preferably, the steps of processing and classifying the audio signal include: Step 1: Use a sliding window method to divide the audio data into audio chunks of a specific size, and perform preprocessing on the audio chunk data using Mel-frequency cepstral coefficient transformation; Step 2: Input the preprocessed audio chunks into a binary classifier for recognition to determine whether there is an "artificial signal"; Step 3: When it is detected that there is an "artificial signal", obtain a 1.2-second original audio segment centered on this signal, and perform preprocessing again to convert the audio signal into a Mel spectrogram; Step 4: Input the Mel spectrogram into a deep learning classification model to obtain at least one interaction intention result of "single beat", "double beat", or "slide".

[0008] Preferably, the binary classifier is constructed using a three-layer fully connected neural network, and the classification results are optimized through a majority voting scheme.

[0009] Preferably, the deep learning classification model performs transfer learning based on the DenseNet architecture and adapts to the classification requirements of the three interaction intentions by modifying the output layer.

[0010] Preferably, in the step of sending the signal corresponding to the recognized interaction intention to the extended reality device, the signal is encoded as a byte object through a Bluetooth connection and then sent to the extended reality device.

[0011] Preferably, in the step where the extended reality device receives the signal, the received byte object is parsed into a string of 0, 1, or 2, corresponding to the "single beat", "double beat", and "slide" interaction intentions respectively.

[0012] Preferably, in the step of implementing the interaction function according to the signal content in combination with the cursor position controlled by the head, the cursor position controlled by the head is calculated by the following formula: where L represents the distance between the head and the screen, represents the angle of deviation of the head in the yaw direction (yaw: heading, the object rotates around the Y axis), represents the angle of deviation of the head in the pitch direction (pitch: pitch, the object rotates around the X axis).

[0013] A hanging hand touch interaction system for extended reality interaction, comprising: an audio processing module, which is used to collect audio data through a mobile phone microphone and preprocess the collected audio data, including dividing the audio data into audio blocks of a specific size in a sliding window manner, and performing Mel cepstral coefficient transformation processing on the audio block data, so that subsequent modules can more accurately identify and classify audio signals; A neural network recognition module, specifically a binary classifier, which is used to identify whether the audio block contains a human signal; An audio selection module, which, after the neural network recognition module determines that there is a human signal in the audio block, obtains an original audio segment of a certain duration centered on the audio block, and further preprocesses the segment to generate input data suitable for the neural network classification module; A neural network classification module, which is used to classify the preprocessed audio segment and identify specific interaction gesture types, such as "single tap", "double tap", "swipe"; A signal sending module, which is used to send the interaction gesture type signal identified by the neural network classification module to the AR / VR glasses unit through wireless communication methods such as Bluetooth.

[0014] Preferably, the system further includes an AR / VR glasses unit, which includes a signal receiving module and an interaction implementation module. The signal receiving module is used to receive the interaction gesture type signal from the signal sending module and parse it into corresponding control instructions. The interaction implementation module realizes specific interaction functions according to the parsed control instructions, combined with the cursor position controlled by the head and the collision detection result between the cursor and the menu icon.

[0015] Beneficial effects The present invention provides a hanging hand touch interaction method for extended reality interaction. Compared with the prior art, it has the following beneficial effects: The present invention performs tapping and swiping operations on the mobile phone in the pocket through clothing, without the user having to raise their hand or directly operate the mobile phone, avoiding the physical fatigue and social embarrassment caused by raising hand interactions. This interaction method conforms to natural behavior postures and is particularly suitable for crowded spaces or public occasions, where users can complete interaction operations without attracting the attention of others. In some specific scenarios (such as cycling, driving), it can avoid the potential safety hazards caused by directly operating the mobile phone. The present invention allows users to complete interactions without taking out the mobile phone, avoiding the risk of both hands leaving the control device, and at the same time protecting the user's privacy, because the interaction actions are blocked by clothing and are not easily noticed by others. The present invention is not only applicable to common extended reality and augmented reality interaction scenarios, but also can provide an efficient and natural interaction experience in more complex environments (such as walking outdoors, public transportation, multi-person occasions, etc.). Through simple tapping and swiping operations, users can complete various low-frequency simple interaction tasks, such as making and answering calls, controlling music, viewing message notifications, etc., greatly enriching the application scenarios of interactions. In summary, the present invention solves many problems existing in the prior art through an innovative hanging-hand touch interaction method, providing users with a more natural, efficient, safe and private interaction method, having significant technical advantages and broad application prospects; BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. is a system framework diagram of a hanging-hand touch interaction system for extended reality interaction proposed by the present invention; Figure 2 FIG. is a real-scene effect diagram of a hanging-hand touch interaction system for extended reality interaction proposed by the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] Please refer to Figure 1 - Figure 2 , the present invention provides two technical solutions, specifically including the following embodiments: Embodiment 1: A hanging-hand touch interaction method and its interaction system for extended reality interaction, including the following steps: S1. The mobile phone terminal collects the audio signals generated by the user's tapping or swiping operations on the mobile phone in the pocket through clothing; S2. Process and classify the collected audio signals, identify the user's interaction intention, and send the signal corresponding to the identified interaction intention to the extended reality device via wireless communication. In the step of collecting audio signals, use the pyaudio library to monitor the mobile phone microphone, set the audio sampling format to 16-bit PCM audio format, record in mono, with a sampling rate of 11025 Hz, and the audio data buffer size is 20 ms; The steps of processing and classifying the audio signals include: Step 1: Use a sliding window method to divide the audio data into audio chunks of a specific size, and preprocess the audio chunk data using the Mel cepstral coefficient transformation; Step 2: Input the preprocessed audio chunks into a binary classifier for identification to determine whether there is an "artificial signal". The binary classifier is constructed using a three-layer fully connected neural network, and the classification results are optimized through a majority voting scheme; Step 3: When it is detected that there is an "artificial signal", obtain a 1.2-second original audio segment centered on this signal, and perform preprocessing again to convert the audio signal into a Mel spectrogram; Step 4: Input the Mel spectrogram into a deep learning classification model to obtain at least one interaction intention result among "single tap", "double tap", and "swipe". The deep learning classification model performs transfer learning based on the DenseNet architecture and adapts to the classification requirements of the three interaction intentions by modifying the output layer.

[0019] S3. The extended reality device receives the signal and realizes the corresponding interaction function according to the signal content combined with the cursor position controlled by the head. In the step of sending the signal corresponding to the identified interaction intention to the extended reality device, encode the signal as a byte object via Bluetooth connection and send it to the extended reality device. In the step of the extended reality device receiving the signal, parse the received byte object into a string of 0, 1, or 2, which respectively correspond to the "single tap", "double tap", and "swipe" interaction intentions. In the step of realizing the interaction function according to the signal content combined with the cursor position controlled by the head, the cursor position controlled by the head is calculated by the following formula: where L represents the distance between the head and the screen, represents the angle of deviation of the head in the yaw direction (yaw: heading, the object rotates around the Y axis), It represents the angle of the representative head offset in the pinch direction (pitch: the object rotates around the X-axis). Different interaction branches are implemented for specific scenarios to achieve specific interaction purposes. For example, when the cursor stays on the QQ icon and a "single tap" signal is received, QQ is controlled to open. If a "double tap" signal is received, the secondary menu related to QQ is opened; if a web page has been opened, when the cursor stays on the upper half of the web page and a "swipe" signal is received, the web page flips downwards. When the cursor stays on the lower half of the web page and a "swipe" signal is received, the web page flips upwards.

[0020] Embodiment 2: Based on Embodiment 1, a hanging hand touch interaction system for extended reality interaction includes: an audio processing module. The audio processing module is used to collect audio data through the mobile phone microphone and preprocess the collected audio data, including dividing the audio data into audio blocks of a specific size in a sliding window manner and performing Mel-frequency cepstral coefficient transformation processing on the audio block data, so that subsequent modules can more accurately identify and classify audio signals; A neural network recognition module. The neural network recognition module is specifically a binary classifier, which is used to identify whether the audio block contains a human signal; An audio selection module. After the neural network recognition module determines that there is a human signal in the audio block, the audio selection module obtains an original audio segment of a certain duration centered on the audio block and further preprocesses the segment to generate input data suitable for the neural network classification module; A neural network classification module. The neural network classification module is used to classify the preprocessed audio segment to identify specific interaction gesture types, such as "single tap", "double tap", "swipe"; A signal sending module. The signal sending module is used to send the interaction gesture type signal recognized by the neural network classification module to the AR / VR glasses unit through wireless communication means such as Bluetooth; The system further includes an AR / VR glasses unit, which includes a signal receiving module and an interaction implementation module. The signal receiving module is used to receive the interaction gesture type signal from the signal sending module and parse it into corresponding control instructions. The interaction implementation module realizes specific interaction functions according to the parsed control instructions, in combination with the cursor position controlled by the head and the collision detection result between the cursor and the menu icon.

[0021] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the application shall be included in the protection scope of the present application.

Claims

1. A touch interaction method and an interaction system for extended reality interaction, characterized in that: The following steps are involved: S1. The mobile phone collects audio signals generated by the user tapping or swiping the mobile phone in the pocket through the clothes; S2, processing and classifying the collected audio signals, identifying the user's interaction intention, and sending the signal corresponding to the identified interaction intention to the extended reality device via wireless communication; S3. The extended reality device receives the signal and implements corresponding interactive functions according to the signal content and the cursor position controlled by the head.

2. A method for extended reality interaction by touching with hands according to claim 1, characterized in that: In the step of collecting audio signals, the pyaudio library is used to monitor the mobile phone microphone, and the audio sampling format is set to 16-bit PCM audio format, mono recording, sampling rate of 11025 Hz, and audio data buffer size of 20 ms.

3. The method for extended reality interaction by touching with hands according to claim 1, characterized in that: The step of processing and classifying the audio signal comprises: Step 1: Use a sliding window method to divide the audio data into audio blocks of a specific size, and use Mel-cepstral coefficient transform to pre-process the audio block data; Step 2: Input the preprocessed audio block into the binary classifier for identification to determine whether there is "artificial signal"; Step 3: When an "artificial signal" is detected, a 1.2-second original audio clip centered on the signal is obtained and preprocessed again to convert the audio signal into a Mel-spectrogram; Step 4: Input the Mel-frequency spectrum into the deep learning classification model to obtain at least one interaction intention result among "single tap", "double tap" and "slide".

4. The method for extended reality interaction by touching with hands according to claim 3, characterized in that: The binary classifier is constructed using a three-layer fully connected neural network, and the classification results are optimized through a majority voting scheme.

5. The method for extended reality interaction by touching with hands according to claim 3, characterized in that: The deep learning classification model performs transfer learning based on the DenseNet architecture, and adapts the classification requirements of the three interaction intentions by modifying the output layer.

6. The method for extended reality interaction by touching with hands according to claim 1, characterized in that: In the step of sending the signal corresponding to the identified interaction intention to the extended reality device, the signal is encoded into a byte object via a Bluetooth connection and then sent to the extended reality device.

7. The method for extended reality interaction by touching with hands according to claim 1, characterized in that: In the step of receiving a signal by the extended reality device, the received byte object is parsed into a string of 0, 1 or 2, corresponding to the "single tap", "double tap" and "slide" interaction intentions respectively.

8. The method for extended reality interaction by touching with hands according to claim 1, characterized in that: In the step of realizing the interactive function according to the signal content combined with the cursor position controlled by the head, the cursor position controlled by the head is calculated by the following formula: Where L represents the distance between the head and the screen. Represents the angle of the head in the yaw direction (yaw: heading, the object rotates around the Y axis), Represents the angle of the head in the pinch direction (pitch: pitch, the object rotates around the X axis).

9. A hand-free touch interaction system for extended reality interaction, based on any one of the hand-free touch interaction methods for extended reality interaction according to claims 1-8, characterized in that: include: An audio processing module, which is used to collect audio data through the mobile phone microphone and pre-process the collected audio data, including dividing the audio data into audio blocks of a specific size by using a sliding window, and performing Mel cepstral coefficient transformation on the audio block data, so that subsequent modules can more accurately identify and classify audio signals; A neural network recognition module, wherein the neural network recognition module is specifically a binary classifier, used to identify whether an audio block contains an artificial signal; An audio selection module, after the neural network recognition module determines that there is an artificial signal in the audio block, the audio selection module obtains an original audio segment of a certain length centered on the audio block, and further preprocesses the segment to generate input data suitable for the neural network classification module; A neural network classification module, which is used to classify the pre-processed audio clips and identify specific interactive gesture types, such as "single tap", "double tap" and "slide"; A signal sending module is used to send the interactive gesture type signal recognized by the neural network classification module to the AR / VR glasses unit via wireless communication methods such as Bluetooth.

10. The hand-free touch interaction system for extended reality interaction according to claim 9, characterized in that: The system also includes an AR / VR glasses unit, which includes a signal receiving module and an interaction implementation module; the signal receiving module is used to receive the interaction gesture type signal from the signal sending module and parse it into a corresponding control instruction; the interaction implementation module implements a specific interaction function according to the parsed control instruction, combined with the cursor position controlled by the head and the collision detection result between the cursor and the menu icon.

Images