Shared space multi-user silent voice interaction method and device and medium

Abstract

The invention relates to a shared space multi-user silent voice interaction method and device and a medium, and the method comprises the following steps: taking a frequency modulation continuous wave signal as an audio carrier and a sensing medium at the same time, directionally broadcasting an audio in a shared space, and capturing a silent voice signal in a multi-user scene through lip motion reflection; performing feature extraction and separation on the silent voice signals to obtain silent voice features of each user; on the basis of silent voice features, a lip movement analysis model driven by an integrated sliding window is adopted, silent voice content of a user is recognized in real time, and real-time interaction is achieved. Compared with the prior art, multi-user silent voices can be accurately recognized in a complex environment, and silent interaction in a privacy sensitive scene is achieved.

Description

Technical Field

[0001] The present invention relates to the field of voice interaction technology, and in particular to a method, device and medium for silent voice interaction among multiple users in a shared space. Background Art

[0002] As voice control systems rapidly become integrated into daily life, privacy protection during interactions is becoming increasingly important. While voice assistants offer increased convenience, they still face challenges in scenarios such as in-vehicle environments, museums, and offices, including privacy breaches, discomfort with public speaking, and interaction in noisy environments. Silent Speech Interfaces (SSIs) enable communication through inaudible speech, meeting the needs of people with speech impairments and also being applicable to privacy-sensitive silent communication scenarios. Existing wireless signal detection technologies, including electromyography (EMG), ultrasound imaging, and video lip reading recognition, have shown potential in improving privacy protection and enabling covert interactions.

[0003] Currently, the spherical wave propagation characteristics of traditional acoustic systems easily cause noise pollution in public spaces. Although directional speaker arrays can improve directivity, they are bulky and still have leakage problems.

[0004] Currently, camera- or millimeter-wave-based SSI requires additional equipment to support two-way interaction. For example, in real-world scenarios like museum tours or in-car interactions, not only silent speech recognition is required, but also audio feedback (such as exhibit explanations or navigation instructions) must be provided. Furthermore, existing acoustic SSI is mostly limited to single-user scenarios (e.g., integrated solutions for smartphones, smartwatches, and smart glasses), requiring the user to be in close proximity to the device, and failing to meet the needs of group interaction for non-device holders such as museum visitors. For example, CN111091831A discloses a silent lip reading recognition method and system. This method continuously emits a millimeter-wave signal as a carrier and focuses it on the user's oral area. The millimeter-wave signal is phase-modulated and partially reflected by the user's speaking behavior. The reflected signal is converted to baseband and corrected, and then a trigonometric transform-based linear reconstruction of speech phase ambiguity is used to obtain user speaking behavior information. This method has the advantages of being contactless, highly penetrating, and highly accurate. It can accurately track fine lip movements and accurately detect accurate voice commands. However, it can only recognize lip readings from a single user and is not applicable to multi-user scenarios.

[0005] Therefore, there is currently a lack of a high-precision, low-latency multi-user silent speech recognition method in privacy scenarios. Summary of the Invention

[0006] The purpose of the present invention is to provide a method, device and medium for silent voice interaction among multiple users in a shared space. By modulating low-frequency sound waves onto a high-frequency carrier and using air nonlinear demodulation to restore audible sound, while maintaining the high directivity of the high-frequency sound waves, the method accurately projects audio to multiple users and simultaneously captures their silent voice features to achieve interaction.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A multi-user silent voice interaction method in a shared space comprises the following steps:

[0009] S1 uses a frequency-modulated continuous wave signal as both the audio carrier and the perception medium, while simultaneously broadcasting audio directionally in a shared space and capturing silent speech signals in multi-user scenarios through lip movement reflections;

[0010] S2, extract and separate the silent speech signal to obtain the silent speech features of each user;

[0011] S3, based on silent speech features, adopts an integrated sliding window-driven lip movement analysis model to recognize the user's silent speech content in real time and realize real-time interaction.

[0012] Said S1 comprises the following steps:

[0013] S11: A frequency modulated continuous wave signal with a preset waveform, bandwidth, and chirp length is used as a carrier signal. The audio signal is loaded onto the carrier signal to obtain a pre-transmitted signal. A periodic time shift is added to the pre-transmitted signal as a transmitted signal. The time shift corresponding to the i-th user is (i-1)t shift , t shift For a time slice;

[0014] S12, using a parametric array multi-directional speaker to send a transmission signal toward the specific location of the user, with different specific locations corresponding to different time shifts, to achieve directional broadcast audio in multiple directions;

[0015] S13, obtaining a plurality of reflected signals received by a single microphone after the transmitted signal is reflected by the lip movements of multiple users, to obtain a silent speech signal, wherein the silent speech signal is a mixed signal.

[0016] The frequency modulated continuous wave signal uses a linear triangle wave as a modulation waveform, a bandwidth of 2 kHz, and a chirp length of 0.25 seconds.

[0017] In S11, a pre-modulated audio optimization algorithm is used to apply an amplitude coefficient A(f) and a phase coefficient φ(f) to the frequency domain of the audio signal to adjust the frequency domain, thereby obtaining an optimized audio signal, which is then loaded onto the carrier signal.

[0018] The S2 comprises the following steps:

[0019] Performing a cross-correlation operation on the silent speech signal and the pre-transmitted signal without adding a periodic time shift to obtain a cross-correlation signal;

[0020] The blind source separation method is used to segment the cross-correlated signals, and the original signal is restored from the observed mixed signal to obtain the silent speech features of each user.

[0021] The lip movement analysis model performs the following steps:

[0022] A time window is defined based on the natural speaking rate, and a sliding window with a preset step size is used to intercept the silent speech feature fragments. The fragments are then mapped to the 64-dimensional frequency domain space through Fbank to obtain a time series spectrogram.

[0023] The time-series spectrogram is input into a word recognition model based on a lightweight convolutional network. The word recognition model processes the input spectrogram through a one-dimensional convolution layer, a batch normalization layer, a ReLU activation function, and a depth-wise separable convolution in a one-dimensional time-channel separable convolution layer, and outputs a streaming preliminary speech recognition result.

[0024] The window size and step size are preset, and the streaming preliminary speech recognition results are processed through a sliding window. Error detection and correction are performed based on the N-gram language model and conditional random fields to output the final speech recognition results.

[0025] The training set construction method of the word recognition model is:

[0026] Design a vocabulary that includes action words, numbers, and conjunctions, and combine them into diverse instruction sequences as training labels;

[0027] Silent speech signals are collected from users of different genders and age groups in different environments, including quiet indoors, noisy indoors, and outdoors;

[0028] The collected signals are enhanced through time domain distortion, time-frequency shift and noise addition and used as training input.

[0029] The error detection and correction based on the N-gram language model and conditional random field is specifically as follows:

[0030] For short words with similar pronunciations, language features are extracted based on the N-gram language model. The language features include monograms, digrams, and trigrams and their posterior probabilities.

[0031] The language features are input into the conditional random field and corrected by selecting high-probability candidates or reconstructing the grammatical error correction mechanism of the sentence.

[0032] A multi-user silent voice interaction device in a shared space comprises a memory, a processor, and a program stored in the memory, wherein the method described is implemented when the processor executes the program.

[0033] A storage medium stores a program, which implements the method when the program is executed.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) The present invention achieves precise projection of audio in a shared space through a multi-directional speaker array and low-frequency sound wave modulation on a high-frequency carrier. It can provide multiple users with a way to interact without having to be close to the device, breaking the limitations of traditional single-user SSI and meeting the needs of multi-user concurrent interaction in groups such as in-vehicle environments and museum visitors, highlighting its versatility and practical value.

[0036] (2) This invention uses frequency-modulated continuous wave (FMCW) millimeter wave signals as a sensing medium to accurately capture tiny lip movement reflections and obtain high-precision silent speech features. Combined with a sliding window-driven lip movement analysis model, it enables real-time interaction, and its low latency ensures a smooth user experience.

[0037] (3) The present invention utilizes the nonlinear effect of air to demodulate and restore audible sound while maintaining the high directivity of high-frequency sound waves, so that audio feedback is clearly audible only in a specific direction and range, effectively protecting user privacy and minimizing sound leakage.

[0038] (4) The present invention adopts a compact word recognition model to support rapid training and transfer learning, thereby reducing complexity and enhancing portability. It can be easily integrated into existing intelligent systems, such as smart homes, smart offices, smart education, etc., to achieve a wider range of silent interaction functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 Schematic diagram of the signal processing process of the present invention;

[0041] Figure 3 Schematic diagram of multi-directional time-shifted signals of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0043] This embodiment provides a method for silent speech interaction among multiple users in a shared space. The core challenges of implementing this method are: 1) The parametric array loudspeaker only operates in a narrowband ultrasonic frequency band, and the perception signal needs to be embedded without interfering with the directional broadcasting function of the multi-directional loudspeaker; 2) In a multi-user scenario, the signal aliasing problem of the same microphone receiving the lip movement features of multiple users needs to be solved; 3) The accuracy of silent speech recognition and the real-time requirements of the system need to be taken into account. Based on this, this embodiment innovatively uses the FMCW signal as the audio carrier to achieve synchronous modulation of the audio transmission and perception signal of the directional loudspeaker, and proposes a multi-user feature separation method based on time-shifted FMCW sound beam, combined with the blind source separation algorithm and sliding window processing, to support sentence-level real-time recognition. Actual measured data show that the word error rate (WER) of the proposed method in a multi-user scenario is as low as 6.5%, and the sentence error rate (SER) is 12.8%, while maintaining audio quality (PESQ = 2.81).

[0044] Specifically, such as Figure 1 As shown, the method includes the following steps:

[0045] S1 uses FM continuous wave signals as both audio carrier and perception medium, while broadcasting audio directionally in a shared space and capturing silent speech signals in multi-user scenarios through lip movement reflection.

[0046] like Figure 2 As shown, S1 includes the following steps:

[0047] S11, using a frequency modulated continuous wave signal with a preset waveform, bandwidth and chirp length as a carrier signal, loading the audio signal onto the carrier signal to obtain a pre-transmitted signal, adding a periodic time shift to the pre-transmitted signal as a transmitted signal.

[0048] In this embodiment, the frequency modulated continuous wave signal uses a linear triangle wave as the modulation waveform, the bandwidth is set to 2 kHz, and the chirp length is set to 0.25 seconds.

[0049] The premodulation audio optimization algorithm is used to adjust the amplitude coefficient A(f) and phase coefficient φ(f) in the frequency domain of the audio signal to obtain the optimized audio signal, which is then loaded onto the carrier signal.

[0050] This embodiment explores the possibility of feature separation under single microphone conditions by changing the FMCW signal transmission mode in different directions. Figure 3 As shown, a periodic time shift is added to the FMCW signal, and the time shift of the i-th user is (i-1)t shift , t shiftThe unvoiced speech characteristics are carried by the FMCW signal after the (i-1)tshift time shift. The FMCW carriers transmitted in different directions are given a specific time offset, so that the characteristics of different users can be directly decoupled by separating the cross-correlation peaks.

[0051] S12, using a parametric array multi-directional speaker to send a transmission signal toward the specific location of the user, with different specific locations corresponding to different time shifts, to achieve directional broadcast audio in multiple directions;

[0052] S13, obtaining a plurality of reflected signals received by a single microphone after the transmitted signal is reflected by the lip movements of multiple users, to obtain a silent speech signal, wherein the silent speech signal is a mixed signal.

[0053] S2, extract and separate the features of the silent speech signal to obtain the silent speech features of each user.

[0054] like Figure 2 As shown, S2 includes the following steps:

[0055] S21, performing a cross-correlation operation on the silent speech signal and the pre-transmitted signal without adding a periodic time shift to obtain a cross-correlation signal;

[0056] S22, using a blind source separation method to segment the cross-correlated signals, recovering the original signals from the observed mixed signals, and obtaining the silent speech features of each user.

[0057] S3, based on silent speech features, adopts an integrated sliding window-driven lip movement analysis model to recognize the user's silent speech content in real time and realize real-time interaction.

[0058] In one embodiment, Figure 2 As shown, the lip movement analysis model performs the following steps:

[0059] S31, based on the natural speaking rate, defines a time window, intercepts a 1-second silent speech feature segment with a sliding window of 0.1s step size, and maps it to the 64-dimensional frequency domain space through Fbank to obtain a time series spectrogram;

[0060] S32: Input the time-series spectrogram into a word recognition model based on a lightweight convolutional network. The word recognition model processes the input spectrogram through a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function, and a depth-wise separable convolution in a one-dimensional time-channel separable convolution layer. This effectively learns the temporal dependencies in silent speech while maintaining robustness to noise and variation, and outputs a streaming preliminary speech recognition result.

[0061] S33, with a preset window size of 1s (covering the full word length) and a step size of 0.15s, processes the streaming preliminary speech recognition results through a sliding window, performs error detection and correction based on the N-gram language model and conditional random field, and outputs the final speech recognition results.

[0062] Designed for real-time silent speech processing, it improves accuracy and enables fast response.

[0063] In one embodiment, the training set construction method of the word recognition model is:

[0064] A1: Design a word set that includes action words, numbers, and conjunctions, and combine them into diverse instruction sequences as training labels;

[0065] A2, collected silent speech signals from users of different genders and age groups in different environments: quiet indoors (6 people), noisy indoors (8 people), and outdoors (6 people);

[0066] A3: Data enhancement is performed on the collected signal through time domain distortion, time-frequency shift and noise addition, and used as training input.

[0067] Table 1 shows the design of some vocabulary sets.

[0068] Table 1

[0069]

[0070] Based on the designed word set, the command sequence examples are as follows:

[0071] Simple commands: "Volume up." "Turn Left." "Shut down."

[0072] Query statements: "I can help you." "I need you to shut it off." "Can you helpme turn the volume louder?" "No, try it again." "Yes, I will unmute it and thenturn the volume up."

[0073] The error detection and correction based on the N-gram language model and conditional random field are as follows:

[0074] For short words with similar pronunciations, language features are extracted based on the N-gram language model. The language features include monograms, digrams, and trigrams and their posterior probabilities.

[0075] The language features are input into the conditional random field and corrected by selecting high-probability candidates or reconstructing the grammatical error correction mechanism of the sentence.

[0076] This example evaluates the performance of the proposed method for different numbers of users (1-4) using word error rate (WER), sequence error rate (SER), and perceptual speech quality (PESQ). PESQ decreases with increasing number of users: 3.02 for a single user and 2.43 for four users, demonstrating that the proposed method can maintain controllable speech quality for multiple users.

[0077] Silent speech recognition performance was evaluated using WER and SER for both users in the training set and those not in the training set. For users in the training set, the WER ranged from 4.56% to 8.96% and the SER ranged from 9.32% to 17.85%. For users not in the training set, the WER ranged from 14.22% to 19.05% and the SER ranged from 21.87% to 30.96%. The overall SER averaged only one error per five conversations, meeting most silent interaction requirements.

[0078] This embodiment also analyzes the word accuracy of all users in the training set and the test set of users not in the training set. The confusion matrix shows an average accuracy of 92.13% and a standard deviation of 6.49%, indicating the high-precision recognition effect of the present invention.

[0079] The above is an introduction to a method embodiment. The following further illustrates the solution of the present invention through an apparatus embodiment.

[0080] A multi-user silent voice interaction device in a shared space includes a memory, a processor, and a program stored in the memory, wherein the processor implements the aforementioned method when executing the program. The device includes:

[0081] Signal transmission and reception module: Uses frequency-modulated continuous wave (FMCW) signals as both audio carriers and sensing media, enabling directionally broadcasting audio in a shared space while capturing silent speech signals in multi-user scenarios through lip motion reflections.

[0082] Feature extraction module: used to extract and separate features of silent speech signals to obtain the silent speech features of each user;

[0083] Silent speech recognition module: Based on silent speech features, it uses an integrated sliding window-driven lip movement analysis model to recognize the user's silent speech content in real time and achieve real-time interaction.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0085] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0086] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A multi-user silent voice interaction method in a shared space, characterized in that: The following steps are involved: S1 uses a frequency-modulated continuous wave signal as both the audio carrier and the perception medium, while simultaneously broadcasting audio directionally in a shared space and capturing silent speech signals in multi-user scenarios through lip movement reflections; S2, extract and separate the silent speech signal to obtain the silent speech features of each user; S3, based on silent speech features, adopts an integrated sliding window-driven lip movement analysis model to recognize the user's silent speech content in real time and realize real-time interaction.

2. A shared space multi-user silent voice interaction method according to claim 1, characterized in that: Said S1 comprises the following steps: S11: A frequency modulated continuous wave signal with a preset waveform, bandwidth, and chirp length is used as a carrier signal. The audio signal is loaded onto the carrier signal to obtain a pre-transmitted signal. A periodic time shift is added to the pre-transmitted signal as a transmitted signal. The time shift corresponding to the i-th user is (i-1)t shift , t shift For a time slice; S12, using a parametric array multi-directional speaker to send a transmission signal toward the specific location of the user, with different specific locations corresponding to different time shifts, to achieve directional broadcast audio in multiple directions; S13, obtaining a plurality of reflected signals received by a single microphone after the transmitted signal is reflected by the lip movements of multiple users, to obtain a silent speech signal, wherein the silent speech signal is a mixed signal.

3. A shared space multi-user silent voice interaction method according to claim 2, characterized in that: The frequency modulated continuous wave signal uses a linear triangle wave as a modulation waveform, a bandwidth of 2 kHz, and a chirp length of 0.25 seconds.

4. A shared space multi-user silent voice interaction method according to claim 2, characterized in that: In S11, a pre-modulated audio optimization algorithm is used to apply an amplitude coefficient A(f) and a phase coefficient φ(f) to the frequency domain of the audio signal to adjust the frequency domain, thereby obtaining an optimized audio signal, which is then loaded onto the carrier signal.

5. The method for multi-user silent voice interaction in a shared space according to claim 1, characterized in that: The S2 comprises the following steps: Performing a cross-correlation operation on the silent speech signal and the pre-transmitted signal without adding a periodic time shift to obtain a cross-correlation signal; The blind source separation method is used to segment the cross-correlated signals, and the original signal is restored from the observed mixed signal to obtain the silent speech features of each user.

6. A shared space multi-user silent voice interaction method according to claim 1, characterized in that: The lip movement analysis model performs the following steps: A time window is defined based on the natural speaking rate, and a sliding window with a preset step size is used to intercept the silent speech feature fragments. The fragments are then mapped to the 64-dimensional frequency domain space through Fbank to obtain a time series spectrogram. The time-series spectrogram is input into a word recognition model based on a lightweight convolutional network. The word recognition model processes the input spectrogram through a one-dimensional convolution layer, a batch normalization layer, a ReLU activation function, and a depth-wise separable convolution in a one-dimensional time-channel separable convolution layer, and outputs a streaming preliminary speech recognition result. The window size and step size are preset, and the streaming preliminary speech recognition results are processed through a sliding window. Error detection and correction are performed based on the N-gram language model and conditional random fields to output the final speech recognition results.

7. A shared space multi-user silent voice interaction method according to claim 6, characterized in that: The training set construction method of the word recognition model is: Design a vocabulary that includes action words, numbers, and conjunctions, and combine them into diverse instruction sequences as training labels; Silent speech signals are collected from users of different genders and age groups in different environments, including quiet indoors, noisy indoors, and outdoors; The collected signals are enhanced through time domain distortion, time-frequency shift and noise addition and used as training input.

8. A shared space multi-user silent voice interaction method according to claim 6, characterized in that: The error detection and correction based on the N-gram language model and conditional random field is specifically as follows: For short words with similar pronunciations, language features are extracted based on the N-gram language model. The language features include monograms, digrams, and trigrams and their posterior probabilities. The language features are input into the conditional random field and corrected by selecting high-probability candidates or reconstructing the grammatical error correction mechanism of the sentence.

9. A multi-user silent voice interaction device in a shared space, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images