Voiceless speech method and voiceless speech device by ultrasonic vocalization

Abstract

To easily and highly accurately predict air vibration caused by vocal cord vibration and to enable voiceless speech by ultrasonic vocalization.SOLUTION: A method includes: an ultrasonic irradiation step S20 of irradiating a pharynx part 78 with ultrasonic waves; a speech production step S26 of producing speech by a speech production organ 56 or a device by combining air vibration by ultrasonic waves and respiratory airflow and air vibration by vocal cord vibration; an analysis step S12 of analyzing sound waves which are speech-produced; and a prediction step S14 of predicting air vibration by vocal cord vibration from air vibration by the newly inputted ultrasonic waves and respiratory airflow based on an analyzed result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method and an apparatus for producing unvoiced speech using ultrasonic speech generation. [Background technology]

[0002] In recent years, the use of voice interfaces has become commonplace with the spread of web conferencing and conversational AI. However, using voice interfaces in public places poses a high risk of leaking secrets and may cause discomfort to others who are present. Therefore, research into voice interaction means using silent speech is progressing. For example, in the following non-patent document 1, an ultrasonic echo probe is attached under the user's chin, and a multi-layer neural network model is constructed that generates acoustic features from a series of ultrasound images, and it has been confirmed that the speaker can be controlled with a signal generated by silent speech. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Naoki Kimura, Michinari Kono, Junichi Rekimoto "Sotto Voce: Silent Speech Interaction Based on Ultrasound Images Using Deep Neural Networks" Proceedings of ACM CHI 2019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the information that can be obtained from ultrasound images is limited due to the nature of the images (2D), and currently, palatal-mounted tongue tracking capacitance arrays and electromyographic sensors are used as complements.

[0005] In view of the above, an object of the present invention is to provide a method and device for unvoiced speech that can predict air vibrations caused by vocal cord vibration simply and with high accuracy, and enable unvoiced speech by ultrasonic speech generation. [Means for solving the problem]

[0006] In order to achieve the above object, the method for unvoiced speech using ultrasonic speech production according to the first aspect includes an irradiation step of irradiating ultrasonic waves toward the pharynx, a speech production step of combining ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration to produce speech using a speech organ or device, an analysis step of analyzing the produced sound waves, and a prediction step of predicting air vibrations due to vocal cord vibration from newly input ultrasonic waves and air vibrations due to respiratory airflow based on the analysis results.

[0007] According to the method for unvoiced speech by ultrasonic speech generation according to the first aspect, ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration are combined to produce speech, so that changes in ultrasonic waves due to articulation, changes in air vibrations due to respiratory airflow, and changes in air vibrations due to vocal cord vibration appear on a single voice waveform. As a result, air vibrations due to vocal cord vibration can be predicted easily and accurately, and unvoiced speech by ultrasonic speech generation can be realized. Effect of the Invention

[0008] As described above, the method for unvoiced speech using ultrasonic utterance according to the first aspect has the excellent effect of being able to predict air vibrations caused by vocal cord vibration simply and with high accuracy, and to realize unvoiced speech using ultrasonic utterance. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a system for implementing a method for unvoiced speech by ultrasonic utterance according to a first embodiment. FIG. [Diagram 2] 2 is a flowchart showing a procedure for creating a voice generation model in the method for unvoiced speech by ultrasonic utterance shown in FIG. 1. [Diagram 3] 3 is a schematic diagram showing a structure of a voice to which the voice generation model shown in FIG. 2 is applied. [Figure 4]FIG. 11 is a flow chart showing a procedure for speech generation in a voiceless speech method using ultrasonic utterance according to a second embodiment. [Diagram 5] FIG. 11 is a schematic diagram showing an example of a usage state of the voiceless speech device by ultrasonic voice generation according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First Embodiment Hereinafter, a first embodiment of a method for unvoiced speech by ultrasonic speech generation according to the present invention will be described with reference to FIGS. 1 to 3 and 5. FIG.

[0011] (Overall composition) First, the overall configuration of a method for unvoiced speech using ultrasonic speech generation according to the present invention will be described with reference to FIG. An example of a system 10 for implementing the method for unvoiced speech by ultrasonic speech generation according to this embodiment is shown in Fig. 1. The system 10 is composed of an analysis unit 12, a prediction unit 14, and a sending unit 16. The analysis unit 12 implements an analysis step S12 in which ultrasonic waves, air vibrations caused by respiratory airflow, and air vibrations caused by vocal cord vibration are combined to analyze sound waves generated by articulation organs 56 or equipment.

[0012] [Analysis section 12] The analysis unit 12 is composed of a data input unit 18, a feature extraction unit 20, and a voice generation model creation unit 22, and creates a voice generation model by extracting features from sound waves generated by articulation organs 56 or equipment by combining ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration, and stores the model in a model storage unit 24. The voice generation model is not particularly limited, and may be a hidden Markov model (hereinafter referred to as "HMM") or a deep neural network (hereinafter referred to as "DNN"), which has an efficient algorithm and a small footprint (occupied area) compared to unit selection type voice synthesis, but statistical voice synthesis based on DNN is described in detail below.

[0013] [Prediction Section 14] The prediction unit 14 is composed of a voice parameter generation unit 26 and a voice waveform generation unit 28. A voice generation model created by learning from a data set of frame-by-frame acoustic features and language features predicts air vibrations due to vocal cord vibration and language features of voiced speech from newly input ultrasonic speech and air vibrations due to respiratory airflow.

[0014] [Transmission section 16] The transmitter 16 is composed of a voiced speech transmitter 30 that performs voiced speech and an ultrasonic transmitter 32 that performs unvoiced speech. The voiced speech transmitter 30 is composed of a voice conversion unit 34 and a speaker 36. Based on the voice waveform 46 generated by the voice waveform generator 28, the voice conversion unit 34 converts unvoiced speech into voiced speech to create a voice signal, and the speaker 36 transmits the voice signal.

[0015] The ultrasonic transmitter 32 is composed of an ultrasonic modulator 40 connected to the sound source 38, and an ultrasonic transducer 44 connected to a directivity controller 42. Based on the audio waveform 46 generated by the audio waveform generator 28, the ultrasonic waves are modulated within the frequency band propagating through the air, and are transmitted with their directivity controlled by the directivity controller 42 as necessary.

[0016] (Speech generation model) Next, the procedure for creating a speech production model will be described with reference to FIGS.

[0017] [Ultrasonic irradiation step S20] First, ultrasound is irradiated toward the pharynx (pharynx) 78.

[0018] [Voiced speech step S22, unvoiced speech step S24] Next, while ultrasound is being irradiated toward the pharynx 78, voiced speech and unvoiced speech are performed, and speech is produced in a state where ultrasound, air vibrations caused by vocal cord vibration, and air vibrations caused by respiratory airflow are superimposed.

[0019] [Articulation Step S26] Here, articulation refers to producing changes in sound by the movement of the articulatory organs 56 or by a device (e.g., a silent speech device 80), and is not limited to producing different vowels. As shown in Fig. 5, the articulatory organs 56 are a collective term for the speech organs located above the vocal cords 60, and include the lips (upper lip 62 and lower lip 64), teeth (upper teeth 66 and lower teeth 68), gums 70, palate 72, uvula 74, tongue 76, pharynx (pharyngeal part) 78, and the like.

[0020] [Dataset acquisition step S28] A data set is then obtained from sound waves produced by the articulatory apparatus 56 or device with superimposed ultrasound, air vibrations due to vocal cord vibration, and air vibrations due to respiratory airflow.

[0021] [Feature extraction step S30] Next, linguistic features and acoustic features are extracted from the dataset on a frame-by-frame basis. As shown in Fig. 3, speech characterized by a speech waveform 46 and a fundamental frequency pattern 48 has a hierarchy of units such as a phoneme sequence 50 (phonemes 50A, 50B, etc.), a word sequence 52 (word 52A, word 52B, etc.), and a sentence, and needs to be retained in an appropriate unit depending on the speech and the features extracted from the speech.

[0022] For example, language features influenced by phonemes 50A and 50B, words 52A and 52B, parts of speech, etc. are stored on a phoneme-by-phoneme basis, and acoustic features are stored on a frame-by-frame basis. Since the time units of both features are different in this way, a correspondence is set between a frame-by-frame acoustic feature sequence and a phoneme-by-phoneme language feature sequence, and pairs of frame-by-frame acoustic features and language features are prepared.

[0023] [Speech generation model creation S32] Next, the linguistic features associated with the acoustic features are input into the DNN, and training is performed based on the squared error minimization criterion.

number

[0024] where ot is the acoustic feature at frame t, and l t is the language feature at frame t, λ is the model parameter (speech parameter) of the DNN, and g λ (·) is the nonlinear transformation function represented by the DNN. The model parameters of the DNN can be efficiently estimated by backpropagation.

[0025] [Speech generation model memory S34] Then, the speech generation model created through learning is stored in the model storage unit 24.

[0026] (Action and Effects) As described above, the method for unvoiced speech by ultrasonic speech production in the first embodiment includes an ultrasonic irradiation step S20 of irradiating ultrasonic waves toward the pharyngeal region 78, a speech production step S26 of combining ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration to produce speech using the articulatory organs 56 or equipment, an analysis step S12 of analyzing the produced sound waves, and a prediction step S14 of predicting air vibrations due to vocal cord vibration from newly input ultrasonic waves and air vibrations due to respiratory airflow based on the analysis results.

[0027] According to the above method, ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration are combined to produce sound, so that changes in ultrasonic waves due to sound production, changes in air vibrations due to respiratory airflow, and changes in air vibrations due to vocal cord vibration appear on a single voice waveform. As a result, it has the excellent effect of being able to easily and accurately predict air vibrations due to vocal cord vibration and realize voiceless speech by ultrasonic speech production.

[0028] In addition, the method of unvoiced speech by ultrasonic utterance according to the first embodiment uses DNN to learn and create a speech generation model, so that language features related to frames can also be used. That is, in DNN speech synthesis, the time correspondence between the current acoustic feature sequence and the language feature sequence is determined in advance, so that language features can be used for frames, unlike HMM speech synthesis. As a result, it has the excellent effect of being able to model more detailed features.

[0029] In this embodiment, acoustic features and linguistic features are used as features, and DNN is used as a speech generation model, but this is not limited to these, and features extracted from other data sets may be applied to DNN or other speech generation models.

[0030] <Second embodiment> Next, a second embodiment of the method for unvoiced speech by ultrasonic speech generation according to the present invention will be described with reference to Fig. 4. Note that the same steps as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0031] A procedure for making speech (voiced speech and unvoiced speech) in the method for unvoiced speech by ultrasonic speech production according to this embodiment is described in Fig. 4. As shown in Fig. 4, the method is characterized in that it can perform both voiced speech and unvoiced speech by converting newly input ultrasonic speech and air vibration due to respiratory airflow into voiced speech based on air vibration due to predicted vocal cord vibration.

[0032] More specifically, air vibrations due to ultrasound and respiratory airflow are input in data acquisition step S36, and a voice generation model is called in voice generation model calling step S38. Then, in voice parameter generation step S40, air vibrations due to vocal cord vibration and their linguistic features are calculated and predicted based on Equation 1 or its modified form, and a voice waveform 46 is generated (S42).

[0033] If silent speech is to be performed by ultrasonic speech in step 44, the ultrasonic waves are modulated into ultrasonic waves that propagate through the air in ultrasonic modulation step S46. If directivity control is to be performed in step 48, directivity control is performed based on the transmission distance of the ultrasonic waves and the surrounding conditions in directivity control step 50, and the ultrasonic waves are output and sent (S52). As a result, silent speech is possible by ultrasonic speech in the inaudible range.

[0034] On the other hand, if the ultrasonic speech and air vibrations due to respiratory airflow acquired in the data acquisition step S36 are converted into voiced speech based on air vibrations due to predicted vocal cord vibrations (S54) and output and transmitted, voiced speech in the audible range becomes possible.

[0035] (Action and Effects) As described above, the method for unvoiced speech using ultrasonic speech in this embodiment includes a voice conversion step S54 for converting newly input ultrasonic speech and air vibrations due to respiratory airflow into voiced speech based on air vibrations due to predicted vocal cord vibration, and a voiced speech transmission step S56 for transmitting the voiced speech.

[0036] Since voiced speech is produced by converting newly input ultrasonic speech and air vibrations caused by respiratory airflow, this has the excellent effect of making voiced speech possible through speech recognition and reconstruction even in cases where vocal cords 60 have been damaged and voiced speech is difficult.

[0037] In addition, it is possible to convert newly input ultrasonic speech and air vibrations due to respiratory airflow into voiced speech based on the predicted air vibrations due to vocal cord vibration, and to perform both unvoiced speech and voiced speech. By performing unvoiced speech while checking voiced speech in real time, it has the excellent effect of allowing the user to learn more natural and free unvoiced speech.

[0038] <Third embodiment> Next, a fourth embodiment of the method for unvoiced speech by ultrasonic speech generation according to the present invention will be described with reference to Fig. 5. Note that the same steps as those in the first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

[0039] A speech device that enables silent speech by ultrasonic speech generation is shown in Figure 5. It is characterized by having both a sound source unit that generates ultrasonic waves and a sound collection unit that collects sounds generated by articulation.

[0040] Specifically, the silent speech device 80 has a sound source unit 82 that generates ultrasonic waves, and a sound collection unit 88 that collects sounds produced by articulation organs 56 or equipment by combining ultrasonic waves irradiated from the sound source unit 82 toward the pharyngeal region 78, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration.

[0041] The sound source unit 82 is composed of an ultrasonic transducer that generates ultrasonic waves of a predetermined frequency, and the sound collection unit 88 is composed of a microphone fixed to the outer wall 84 or the inner wall 86 of the device, for example.

[0042] The silent speech device 80 is tubular and worn between the lips, and is shaped to pick up vibration sounds transmitted through the device's outer wall 84 or inner wall 86, but is not limited to this and may be shaped to be attached to a smartphone, smartwatch, etc.

[0043] Also, the silent speech device 80 itself may be configured to produce sound, such as an ultrasonic whistle.

[0044] (Action and Effects) As described above, the silent speech device using ultrasonic speech production in this embodiment has a sound source unit 82 that generates ultrasonic waves, and a sound collection unit 88 that collects sounds produced by articulation organs 56 or equipment by combining ultrasonic waves, air vibrations caused by respiratory airflow, and air vibrations caused by vocal cord vibrations.

[0045] The ultrasonic waves emitted by the sound source unit 82, the air vibrations caused by respiratory airflow, and the air vibrations caused by vocal cord vibration are combined and the sound produced by articulation by the articulation organs 56 or the device is collected by the sound collection unit 88, so that changes in ultrasonic waves due to articulation, changes in air vibrations caused by respiratory airflow, and changes in air vibrations caused by vocal cord vibration appear on a single voice waveform. As a result, air vibrations caused by vocal cord vibration can be predicted easily and accurately, and unvoiced speech by ultrasonic phonation can be realized.

[0046] Furthermore, if the silent speech device 80 itself is structured to perform speech production, more detailed modeling becomes possible.

[0047] <Supplementary explanation of each embodiment> Although the method and device for silent speech using ultrasonic speech generation have been described in each embodiment above, the present invention is not limited to these, and appropriate design changes are possible without departing from the gist of the invention. [Explanation of symbols]

[0048] S12 Analysis step S14 Prediction step S20 Ultrasonic irradiation step S26 Articulation Steps 12 Analysis Department 14 Prediction Department 60 Vocal Cords 78 Pharynx

Claims

1. An irradiation step of irradiating an ultrasound wave toward a pharyngeal region; A speech production step in which ultrasonic waves, air vibrations due to respiratory airflow, and air vibrations due to vocal cord vibration are combined to produce speech using a speech organ or device; An analysis step of analyzing the generated sound wave; a prediction step of predicting air vibrations due to vocal cord vibration from newly input air vibrations due to ultrasound and respiratory airflow based on the analysis results; A method for silent speech using ultrasonic speech generation comprising:

2. The analyzing step includes: A feature extraction step of extracting acoustic features and linguistic features from the produced sound waves; a speech generation model creation step of inputting the linguistic features associated with the acoustic features on a frame-by-frame basis into a deep neural network to learn and create a speech generation model; The method for unvoiced speech by ultrasonic voicing according to claim 1 , comprising:

3. a voice conversion step of converting newly input ultrasonic speech and air vibrations due to respiratory airflow into voiced speech based on the predicted air vibrations due to vocal cord vibration; a voiced speech output step of outputting the voiced speech; The method for unvoiced speech by ultrasonic voicing of claim 1 further comprising:

4. A sound source unit that generates ultrasonic waves; A sound pickup unit that captures the sound produced by combining ultrasonic waves, air vibrations caused by respiratory airflow, and air vibrations caused by vocal cord vibration; A silent speech device using ultrasonic speech generation.

Images