Voice Recognition Device

Abstract

To provide a voice recognition device that can recognize utterance not using a vocal cord.SOLUTION: The voice recognition device includes: an imaging unit that acquires an image including a lip region of a speaker; a lip movement locus detection unit that detects a lip movement locus of the speaker from the image; a non-audible sound detection unit that detects a non-audible sound propagating in the air from a voice of the speaker; a frequency pattern extraction unit that analyzes frequency characteristics of the non-audible sound, and extracts a frequency pattern; a lip movement locus data storage unit that previously stores a correspondence between the lip movement locus and utterance content; a non-audible sound pattern storage unit that previously stores a correspondence between the frequency pattern of the non-audible sound and utterance content; an utterance candidate extraction unit that refers to the lip movement locus data storage unit to extract candidates for utterance content from the lip movement locus; an utterance determination unit that, in a case in which a plurality of candidates for utterance content are extracted, refers to the non-audible sound pattern storage unit to determine specific utterance content among the plurality of candidates for utterance content; and an output unit that outputs information about the determined utterance content.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice recognition device. [Background technology]

[0002] Many elderly people speak without using their vocal cords. Also, when speaking softly, speech may be produced without using the vocal cords. As a method for knowing the content of speech of people who do not use their vocal cords, a method for recognizing words from mouth movements has been reported (for example, Patent Document 1).

[0003] Patent Document 1 describes a lip reading device that recognizes words based on mouth movements, which includes a first mouth shape detection means that detects a first mouth shape that needs to be formed in advance when uttering a specific sound, which is different from the mouth shape corresponding to the vowel of the sound, and a second mouth shape that is the mouth shape formed when finishing uttering a sound, based on mouth shape information that indicates the speaker's mouth shape, and a recognition means that recognizes the words uttered by the speaker based on the detected first and second mouth shapes.

[0004] Also, a method has been reported for detecting speech content from inaudible sounds emitted when speaking without using the vocal cords (for example, Patent Document 2).

[0005] Patent Document 2 discloses a method of attaching a stethoscope-type microphone to the surface of a human body to collect flesh-conducted vibration sounds of non-audible murmurs that are produced in conjunction with speaking behavior that does not involve regular lip movement of the vocal cords.

[0006] However, in Japanese, there are multiple utterances that involve the same lip movements, and the method of decoding lip movements using a camera, as in Patent Document 1, has the problem that it can only decode a limited number of words.

[0007] Furthermore, the method described in Patent Document 2 requires the user to attach a dedicated microphone in advance, which limits the situations in which the method can be used. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2008-310382 A [Patent Document 2] International Publication No. 2004 / 021738 Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present invention is to provide a voice recognition device capable of recognizing the contents of an utterance even when the speaker speaks without using the vocal cords. [Means for solving the problem]

[0010] A voice recognition device according to an embodiment of the present disclosure is characterized by having an imaging unit that acquires an image including a lip area during a speaker's speaking movement, a lip movement trajectory detection unit that detects a trajectory of the speaker's lip movement from the image, an inaudible sound detection unit that detects inaudible sounds propagating through the air from the sound of the speaker when speaking, a frequency pattern extraction unit that analyzes frequency characteristics of inaudible sounds and extracts frequency patterns, a lip movement trajectory data storage unit that pre-stores a correspondence between the lip movement trajectory and speech contents, an inaudible sound pattern storage unit that pre-stores a correspondence between the frequency patterns of inaudible sounds and speech contents, an utterance candidate extraction unit that refers to the lip movement trajectory data storage unit to extract candidates for utterance contents from the utterance contents, an utterance determination unit that refers to the inaudible sound pattern storage unit to determine a specific utterance content from the multiple candidates for utterance contents when the utterance candidate extraction unit extracts multiple candidates for utterance contents, and an output unit that outputs information regarding the utterance content determined by the utterance determination unit.

[0011] In the above voice recognition device, it is preferable that the inaudible sound detection section starts detecting the inaudible sound of the speaker when triggered by the start of lip movement of the speaker detected by the lip movement trajectory detection section.

[0012] In the above voice recognition device, it is preferable that the inaudible sound detection section detects, as the inaudible sound, a sound wave of 20 kHz or more and 70 kHz or less.

[0013] In the above voice recognition device, it is preferable that the utterance determination unit determines the utterance content based on the presence or absence of a peak in the frequency pattern and the position of a peak occurring in a specific frequency band.

[0014] In the above voice recognition device, the plurality of utterance contents having substantially the same lip movement trajectories may include at least two of "na", "ta", and "da".

[0015] In the above voice recognition device, the plurality of utterance contents having substantially the same lip movement trajectories may include "shi" and "chi."

[0016] In the above voice recognition device, the plurality of utterance contents having substantially the same lip movement trajectories may include "a" and "wa." Effect of the Invention

[0017] According to the voice recognition device of the present invention, even if the speaker speaks without using the vocal cords, the contents of the speech can be recognized. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram of a speech recognition device according to an embodiment of the present disclosure. [Diagram 2] 13A is an example of a face contour recognized by a face image recognition unit, and FIG. 13B is an example of a mouth contour included in the face contour of FIG. [Diagram 3] FIG. 13 shows the trajectory of lip movement when uttering “na,” “ta,” and “da,” where (a) shows the trajectory of lip movement in the y direction, and (b) shows the trajectory of lip movement in the x direction. [Figure 4] (a) to (c) are the frequency spectra of the voice when "na", "ta", and "da" are spoken, respectively. [Diagram 5]4 is a flowchart for explaining an operation procedure of the voice recognition device according to an embodiment of the present disclosure. [Figure 6] FIG. 13 is a diagram showing the trajectory of lip movement when uttering "shi" and "chi", where (a) shows the trajectory of lip movement in the y direction, and (b) shows the trajectory of lip movement in the x direction. [Figure 7] (a) and (b) are the frequency spectra of the sounds produced when the words "shi" and "chi" are spoken, respectively. [Figure 8] 13 is a diagram showing the trajectory of lip movement when uttering "a" and "ha," where (a) shows the trajectory of lip movement in the y direction, and (b) shows the trajectory of lip movement in the x direction. [Figure 9] (a) and (b) are the frequency spectra of the sounds when "a" and "ha" are spoken, respectively. [Figure 10] 1 is a schematic diagram illustrating a configuration of a conversation system using a voice recognition device according to a first embodiment. [Figure 11] FIG. 1 is a block diagram of a voice recognition device according to a first embodiment. [Figure 12] FIG. 11 is a block diagram of a voice recognition device according to a modified example of the first embodiment. [Figure 13] FIG. 11 is a schematic diagram of a configuration of an interpretation device using a voice recognition device according to a second embodiment. [Figure 14] FIG. 11 is a block diagram of a voice recognition device according to a second embodiment. [Figure 15] FIG. 11 is a schematic diagram of a configuration of a voice-operated device operation system using a voice recognition device according to a third embodiment. [Figure 16] FIG. 11 is a block diagram of a voice recognition device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, a speech recognition device according to the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to the embodiments described above, but extends to the inventions described in the claims and their equivalents.

[0020] FIG. 1 shows a block diagram of a voice recognition device 1001 according to an embodiment of the present disclosure. The voice recognition device 1001 includes an imaging unit 1, a lip movement trajectory detection unit 2, an inaudible sound detection unit 3, a frequency pattern extraction unit 4, a lip movement trajectory data storage unit 5, an inaudible sound pattern storage unit 6, an utterance candidate extraction unit 7, an utterance determination unit 8, an output unit 9, and a face image recognition unit 10. The voice recognition device 1001 may be an information terminal such as a smartphone or a tablet terminal. However, the present invention is not limited to such an example, and may be realized as an embedded module using a single board computer. Alternatively, the voice recognition device 1001 may be arranged on a server, and data acquired by the imaging unit 1 and the inaudible sound detection unit 3 may be transmitted to the server. The imaging unit 1 is configured by a camera, and the inaudible sound detection unit 3 is configured by a microphone. The lip movement trajectory data storage unit 5 and the inaudible sound pattern storage unit 6 are configured by a hard disk or a semiconductor memory. The lip movement trajectory detection unit 2, frequency pattern extraction unit 4, utterance candidate extraction unit 7, utterance determination unit 8, output unit 9, and face image recognition unit 10 are realized as software (programs) by a computer provided in a voice recognition device 1001 including a CPU, ROM, RAM, etc.

[0021] The imaging unit 1 is a camera and includes a CMOS (Complementary Metal Oxide Semiconductor) type or CCD (Charge Coupled Device) type image sensor. The imaging unit 1 acquires an image including the lip area during a speaker's speaking action, and supplies the captured image to the face image recognition unit 10 for each frame. The camera may be one that is already installed in an information terminal such as a smartphone or tablet terminal, or an external camera may be used.

[0022] The facial image recognition unit 10 has a function of identifying the outline of the speaker's face and lips and automatically tracking them using a built-in application program for facial recognition. This makes it possible to capture the speaker's facial image even if the speaker moves within the imaging range of the imaging unit 1.

[0023] The lip movement trajectory detection unit 2 detects the trajectory of the speaker's lip movement from the image acquired by the imaging unit 1. FIG. 2(a) is an example of a face contour recognized by the face image recognition unit 10, and FIG. 2(b) is an example of a mouth contour included in the face contour of FIG. 2(a). As shown in FIG. 2(a), the face image recognition unit 10 can determine the positions of the contours of the face 21, eyebrows 22, eyes 23, nose 24, and mouth 25. As shown in FIG. 2(b), the lips open and close in the vertical direction (y direction) and expand and contract in the horizontal direction (x direction) when speaking. Therefore, the feature points for indicating the lip movement are the lower end y1 of the upper lip, the upper end y2 of the lower lip, the left end x1 of the lips, and the right end x2 of the lips. In addition, the feature amount of the lip movement is the time change in the vertical distance (Δy=y1-y2) and the time change in the horizontal distance (Δx=x2-x1).

[0024] The lip movement trajectory data storage unit 5 stores in advance the correspondence between the lip movement trajectory (speech lip movement profile) and the speech content. FIG. 3 shows the lip movement trajectory when "na", "ta", and "da" are spoken, with FIG. 3(a) showing the lip movement trajectory in the y direction and FIG. 3(b) showing the lip movement trajectory in the x direction. The horizontal axis of FIG. 3(a) and (b) is the time (seconds) from the start of lip movement. The vertical axis of FIG. 3(a) is the vertical distance Δy (mm), and the vertical axis of FIG. 3(b) is the horizontal distance Δx (mm). In FIG. 3(a), Lny, Lty, and Ldy respectively show the lip movement trajectory in the y direction when "na", "ta", and "da" are spoken. Also, in FIG. 3(b), Lnx, Ltx, and Ldx respectively show the lip movement trajectory in the x direction when "na", "ta", and "da" are spoken. The lip movement trajectory data storage unit 5 stores in advance the correspondence between the lip movement trajectory and the speech content in various utterances other than the above examples. The lip movement trajectory data storage unit 5 may store a learning model generated by machine learning using artificial intelligence (AI) to determine which speech content the feature amount of the lip movement trajectory is similar to.

[0025] The utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 and extracts candidates for the contents of the utterance from the trajectory of the lip movement. In the example shown in Fig. 3(a) and (b), the three curves (Lny, Lty, Ldy) representing the temporal change of the lip movement in the y direction when "na", "ta", and "da" are uttered draw almost the same trajectory, and the three curves (Lnx, Ltx, Ldx) representing the temporal change of the lip movement in the x direction draw almost the same trajectory. Therefore, when an utterance is detected in which the curve representing the temporal change of the lip movement in the y direction is similar to one of the three curves (Lny, Lty, Ldy) and the curve representing the temporal change of the lip movement in the x direction is similar to one of the three curves (Lnx, Ltx, Ldx), it is known that the contents of the utterance are "na", "ta", or "da", but it is not possible to specify which of these utterances it is. Therefore, in such a case, three candidates for the contents of the utterance are extracted.

[0026] The inaudible sound detection unit 3 detects inaudible sounds propagating through the air from the voice of a speaker when he / she speaks. A MEMS (Micro Electro Mechanical Systems) microphone built into a smartphone or tablet terminal can be used as the inaudible sound detection unit 3. By using a MEMS microphone, it is possible to detect speech including the frequency band of inaudible sounds. In the case where the inaudible sound band is cut in the MEMS microphone built into a terminal such as a smartphone to reduce noise, such band restriction can be released. Instead of using a microphone that is already provided in the smartphone or the like, a microphone capable of detecting inaudible sounds may be attached externally. It is preferable that the inaudible sound detection unit 3 detects sound waves of 20 kHz or more and 70 kHz or less as inaudible sounds.

[0027] It is preferable that the inaudible sound detection unit 3 starts detecting the inaudible sound of the speaker when the start of the speaker's lip movement detected by the lip movement trajectory detection unit 2 is used as a trigger. In addition to being generated when the speaker speaks, inaudible sounds may also be generated when the speaker moves their body, etc., and since this becomes noise, it may be difficult to detect the timing when the speaker starts speaking from the inaudible sound alone. Therefore, it is preferable that the inaudible sound detection unit 3 starts detecting the inaudible sound after the lip movement trajectory detection unit 2 detects that the speaker's lips have started to move. In this way, it is possible to accurately detect the inaudible sound generated by the speaker's speech.

[0028] The frequency pattern extraction unit 4 analyzes the frequency characteristics of the non-audible sound and extracts a frequency pattern. FIG. 4(a) shows the frequency spectrum of the voice when "na" is spoken, FIG. 4(b) shows the frequency spectrum of the voice when "ta" is spoken, and FIG. 4(c) shows the frequency spectrum of the voice when "da" is spoken. In FIG. 4(a) to (c), the horizontal axis shows frequency (kHz) and the vertical axis shows power (dB). Even when speech is performed without using the vocal cords, differences appear in the frequency distribution (frequency pattern) of the non-audible sound region depending on the tongue movement and the way the throat is breathed out. By utilizing the difference in frequency distribution, speech that cannot be identified by the lip movement trajectory can be identified.

[0029] The inaudible sound pattern storage unit 6 stores in advance the correspondence between the frequency pattern of the inaudible sound and the content of the speech. That is, the inaudible sound pattern storage unit 6 stores information on whether or not a peak occurs at a specific frequency, and if a peak occurs, on the position of the peak occurring in a specific frequency band, as a feature point in the frequency pattern for identifying each of a plurality of speeches having substantially the same lip movement trajectory. For example, the inaudible sound pattern storage unit 6 preferably stores the presence or absence of a peak in the frequency pattern of each of the inaudible sounds of "na", "ta", and "da" and the position of the peak occurring in the specific frequency band. Specifically, as shown in FIG. 4(a), when "na" is spoken, the entire tongue is lightly pressed against the upper palate, so no clear peak appears in the frequency pattern in the range of 20 kHz to 30 kHz. Also, as shown in FIG. 4(b), when "ta" is spoken, the tip of the tongue is strongly pressed against the upper palate as if flicking it, so the frequency pattern has a peak Pt in the range of 25 kHz to 30 kHz. Furthermore, as shown in FIG. 4(c), when "da" is uttered, the tongue tip is lightly placed on the upper palate compared to "ta" due to the voiced consonant, so that the frequency pattern has a peak Pd in ​​the range of 20kHz to 25kHz, which is lower than that of "ta". In this way, the inaudible sound pattern storage unit 6 stores that when the utterance is "na", no clear peak appears in the frequency pattern in the range of 20kHz to 30kHz, when the utterance is "ta", a peak appears in the range of 25kHz to 30kHz, and when the utterance is "da", a peak appears in the range of 20kHz to 25kHz. In this way, the inaudible sound pattern storage unit 6 stores in advance, for each of a plurality of utterances having substantially the same lip movement trajectory, information on whether or not a peak appears at a specific frequency in the frequency pattern of the inaudible sound, and, if a peak appears, in which frequency band the peak appears. However, in addition to these, the inaudible sound pattern storage unit 6 also stores other combinations of utterances that have approximately the same lip movement trajectories but different inaudible sound frequency patterns, such as "shi" and "chi," and "a" and "ha."

[0030] When the utterance candidate extraction unit 7 extracts a plurality of utterance content candidates, the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 to determine a specific utterance content from among the plurality of utterance content candidates. For example, when the utterance candidate extraction unit 7 extracts three utterance content candidates "na", "ta", and "da", the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 to determine a specific utterance content from among the three utterance content candidates. As described above, when the lip movement trajectory detected by the lip movement trajectory detection unit 2 is a curve similar to that shown in Figs. 3(a) and (b), the utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 to extract "na", "ta", and "da" as utterance content candidates from the lip movement trajectory. Next, the utterance determination unit 8 refers to the inaudible sound pattern memory unit 6 and compares the frequency pattern of the detected inaudible sound with the frequency patterns of each of the three candidate utterance contents ("na", "ta", and "da") to determine a specific utterance content from among the three candidate utterance contents.

[0031] The utterance determination unit 8 can determine the utterance contents based on the presence or absence of a peak occurring in a specific frequency band in the frequency pattern and the position of the peak. For example, as shown in FIG. 4(a), when no power peak is detected in a frequency range of 20 kHz to 30 kHz, the detected utterance can be determined to be "na". Also, as shown in FIG. 4(b), when a power peak Pt is detected in a frequency range of 25 kHz to 30 kHz, the detected utterance can be determined to be "ta". Alternatively, as shown in FIG. 4(c), when a power peak Pd is detected in a frequency range of 20 kHz to 25 kHz, the detected utterance can be determined to be "da". In this manner, when the utterance candidate extraction unit 7 extracts three utterance content candidates ("na", "ta", and "da"), the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 and determines one of "na", "ta", and "da" as a specific utterance content from among the three utterance content candidates.

[0032] When the utterance candidate extraction unit 7 extracts one candidate for the utterance content, the utterance determination unit 8 can determine the candidate as the utterance content uttered by the speaker without referring to the inaudible frequency pattern. In this case, the process of referring to the inaudible frequency pattern can be omitted, so that the utterance content uttered by the speaker can be quickly determined, and the processing load on the voice recognition device 1001 can be reduced. However, if there is no problem in terms of capability, the utterance determination unit 8 may compare the frequency pattern of the detected voice with the frequency pattern stored in the inaudible sound pattern storage unit 6. This makes it possible to increase the reliability of the pronunciation determination.

[0033] The output unit 9 outputs information related to the utterance content determined by the utterance determination unit 8. When a display device is connected to the output unit 9, the detected utterance content can be displayed as text information on the screen of the display device. When an audio playback device is connected to the output unit 9, the detected utterance content can be output as audio. For example, audio output via earphones or the like may be used in addition to the screen display.

[0034] Next, the operation procedure of the voice recognition device according to the present embodiment will be described. Fig. 5 is a flowchart for explaining the operation procedure of the voice recognition device according to the embodiment of the present disclosure. First, in step S101, the camera, which is the imaging unit 1, is operated. The camera captures an image including the lip area during a speaker's speaking action.

[0035] Next, in step S102, the face image recognition unit 10 identifies the outline of the speaker's face and lips.

[0036] Next, in step S103, the lip movement trajectory detection section 2 detects the trajectory of the speaker's lip movement from the image captured by the camera.

[0037] Next, in step S104, the utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 that prestores the correspondence between the lip movement trajectory and the utterance contents, and extracts candidates for the utterance contents from the lip movement trajectory.

[0038] Meanwhile, after the camera is activated and detects that the speaker's lips have begun to move, this triggers the inaudible sound sensor, which is the inaudible sound detection unit 3, to operate in step S105 and detect inaudible sounds propagating through the air from the voice of the speaker when he or she speaks.

[0039] Next, in step S106, the frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound, and in step S107, extracts a frequency pattern.

[0040] Next, in step S108, the utterance candidate extraction unit 7 judges whether the number of utterance candidates extracted in step S104 is multiple or one. If there is only one utterance candidate, in step S109, the utterance determination unit 8 determines the utterance by the speaker as a single utterance candidate. An example of a case where there is only one utterance candidate is, for example, a vowel. In this case, the utterance content can be determined only by the trajectory of lip movement. Therefore, if there is only one utterance candidate, there is no need to refer to the frequency pattern of the inaudible sound, so that the utterance content can be determined efficiently. However, if there is no problem in terms of capability, the utterance determination unit 8 may compare the frequency pattern of the detected voice with the frequency pattern stored in the inaudible sound pattern storage unit 6. This makes it possible to increase the reliability of the pronunciation determination.

[0041] On the other hand, if the utterance candidate extraction unit 7 extracts multiple utterance content candidates, in step S110, the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6, which pre-stores the correspondence between the frequency patterns of inaudible sounds and the utterance contents, and determines a specific utterance content from among the multiple utterance content candidates based on the frequency pattern.

[0042] Next, in step S111, the output unit 9 outputs the determined speech content.

[0043] In the above description, the combination of "na", "ta" and "da" is exemplified as a plurality of speech candidates extracted from the lip movement trajectory, but the present invention is not limited to such an example. In other words, when a plurality of speech contents having substantially the same lip movement trajectory are a combination of two of "na", "ta" and "da", one of the two combinations of speech contents may be determined. Furthermore, as other examples of the plurality of speech candidates, there are a combination of "shi" and "chi" and a combination of "a" and "ha", and a method of determining a specific utterance from these combinations will be described below.

[0044] First, a case where multiple utterance candidates are a combination of "shi" and "chi" will be described. FIG. 6 shows the trajectory of lip movement when "shi" and "chi" are spoken, with FIG. 6(a) showing the trajectory of lip movement in the y direction and FIG. 6(b) showing the trajectory of lip movement in the x direction. In FIG. 6(a), Lsy and Lcy respectively show the trajectory of lip movement in the y direction (Δy=y1-y2) shown in FIG. 2(b) when "shi" and "chi" are spoken. Also, in FIG. 6(b), Lsx and Lcx respectively show the trajectory of lip movement in the x direction (Δx=x2-x1) when "shi" and "chi" are spoken.

[0045] The utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 and extracts candidates for the utterance contents from the lip movement trajectory. In the example shown in Fig. 6(a) and (b), the curves Lsy and Lcy representing the temporal change in the y direction of the lip movement when "shi" and "chi" are uttered are almost the same curve, and the curves Lsx and Lcx representing the temporal change in the x direction of the lip movement are almost the same curve. When the y direction trajectory of the lip movement detected by the lip movement trajectory detection unit 2 is similar to the curve shown in Fig. 6(a) and the x direction trajectory of the lip movement is similar to the curve shown in Fig. 6(b), the utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 and extracts "shi" and "chi" from the lip movement trajectory as candidates for the utterance contents. Therefore, in this case, it is known that the utterance contents are either "shi" or "chi", but it is not possible to specify which of these is the utterance. Therefore, in such a case, two candidates for the utterance contents are extracted.

[0046] The frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound and extracts the frequency pattern. Figure 7(a) shows the frequency spectrum of the voice when "shi" is spoken, and Figure 7(b) shows the frequency spectrum of the voice when "chi" is spoken.

[0047] The inaudible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of inaudible sounds and the contents of speech. For example, as shown in FIG. 7(a), when "shi" is spoken, no clear peak appears in the frequency pattern near 40 kHz. On the other hand, as shown in FIG. 7(b), when "chi" is spoken, a peak Pc appears in the frequency pattern near 40 kHz by pressing the center of the tongue against the upper palate. In this way, the inaudible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of the inaudible sounds of "shi" and "chi" and the contents of speech.

[0048] Next, the utterance determination unit 8 refers to the inaudible sound pattern memory unit 6 and compares the frequency pattern of the detected inaudible sound with the frequency patterns of the two candidate utterance contents ("shi" and "chi") to determine a specific utterance content from among the two candidate utterance contents.

[0049] The utterance determination unit 8 can determine the utterance content based on the presence or absence of a peak in the frequency pattern and the position of the peak occurring in a specific frequency band. As shown in FIG. 7(a), when no power peak is detected in a frequency near 40 kHz, the detected utterance can be determined to be "shi". Also, as shown in FIG. 7(b), when a power peak Pc is detected in the vicinity of 40 kHz, the detected utterance can be determined to be "chi". In this manner, when the utterance candidate extraction unit 7 extracts two utterance content candidates ("shi" and "chi"), the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 and determines one of "shi" and "chi" as the specific utterance content from among the two utterance content candidates.

[0050] Next, a case where multiple utterance candidates are a combination of "a" and "ha" will be described. FIG. 8 shows the trajectory of lip movement when "a" and "ha" are spoken, with FIG. 8(a) showing the trajectory of lip movement in the y direction and FIG. 8(b) showing the trajectory of lip movement in the x direction. In FIG. 8(a), Lay and Lhy respectively represent the trajectory of lip movement in the y direction (Δy=y1-y2) shown in FIG. 2(b) when "a" and "ha" are spoken. Also, in FIG. 8(b), Lax and Lhx respectively represent the trajectory of lip movement in the x direction (Δx=x2-x1) when "a" and "ha" are spoken.

[0051] The utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 and extracts candidates for the utterance contents from the lip movement trajectory. In the example shown in Fig. 8(a) and (b), the curves Lay and Lhy representing the temporal change in the y direction of the lip movement when "a" and "ha" are uttered are almost the same curve, and the curves Lax and Lhx representing the temporal change in the x direction of the lip movement are almost the same curve. When the y direction trajectory of the lip movement detected by the lip movement trajectory detection unit 2 is similar to the curve shown in Fig. 8(a) and the x direction trajectory of the lip movement is similar to the curve shown in Fig. 8(b), the utterance candidate extraction unit 7 refers to the lip movement trajectory data storage unit 5 and extracts "a" and "ha" from the lip movement trajectory as candidates for the utterance contents. Therefore, in this case, it is known that the utterance contents are either "a" or "ha", but it is not possible to specify which of these utterances it is. Therefore, in such a case, two candidates for the utterance contents are extracted.

[0052] The frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound and extracts the frequency pattern. Figure 9(a) shows the frequency spectrum of the voice when "a" is spoken, and Figure 9(b) shows the frequency spectrum of the voice when "wa" is spoken.

[0053] The inaudible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of inaudible sounds and the contents of speech. For example, as shown in FIG. 9(a), when "a" is spoken, the whole tongue is lightly pressed against the upper palate, so no clear peak appears in the frequency pattern near 20 kHz. Also, as shown in FIG. 9(b), when "wa" is spoken, the center of the tongue is pressed against the upper palate, so a peak Ph appears in the frequency pattern near 20 kHz. In this way, the inaudible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of the inaudible sounds of "a" and "wa" and the contents of speech.

[0054] Next, the utterance determination unit 8 refers to the inaudible sound pattern memory unit 6 and compares the frequency pattern of the detected inaudible sound with the frequency patterns of the two candidate utterance contents ("a" and "ha") to determine a specific utterance content from among the two candidate utterance contents.

[0055] The utterance determination unit 8 can determine the utterance content based on the presence or absence of a peak in the frequency pattern and the position of the peak occurring in a specific frequency band. As shown in FIG. 9(a), when a power peak is not detected in the frequency pattern near 20 kHz, the detected utterance can be determined to be "a." Also, as shown in FIG. 9(b), when a power peak is detected in the frequency pattern near 20 kHz, the detected utterance can be determined to be "wa." In this manner, when the utterance candidate extraction unit 7 extracts two utterance content candidates ("a" and "wa"), the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 and determines one of "a" and "wa" as the specific utterance content from among the two utterance content candidates.

[0056] As described above, the combination of "na," "ta," and "da," the combination of "shi" and "chi," and the combination of "a" and "ha" have been shown as examples of multiple utterance contents with approximately the same lip movement trajectories, but the present invention is not limited to these examples, and the voice recognition device according to the embodiment of the present disclosure can also be used for other combinations of multiple utterance contents with approximately the same lip movement trajectories.

[0057] As described above, the voice recognition device according to the embodiment of the present disclosure can determine speech without using vocal cords (muttering) without contact, extract rough candidates for predicted speech from lip movement trajectories, and determine the speech of the speaker from among the predicted candidates based on the frequency pattern of inaudible sounds. Furthermore, by combining pattern determination based on lip movement and determination based on inaudible sounds, the prediction accuracy of the speech content can be improved. The voice recognition device according to the embodiment of the present disclosure can decipher conversations of elderly people spoken without using vocal cords. In addition, in vehicles where silence is required, the content of a call made in a low voice without using vocal cords can be understood. In this case, since the conversation can be conducted using inaudible sounds, the conversation can be conducted while protecting privacy or preventing information leakage. Furthermore, since there is no need to equip the speaker with dedicated equipment in advance, it can be used for multiple purposes.

[0058] [Example 1] Next, a voice recognition device according to a first embodiment will be described. Fig. 10 is a schematic diagram of a conversation system using the voice recognition device according to the first embodiment. When the speakers (120, 220) are elderly people or the like, they may not be able to hear what the other person is saying because they speak without using their vocal cords, which may result in a problem of poor communication. The voice recognition devices (100, 200) according to the first embodiment communicate by displaying the contents of their own utterances on their own voice recognition devices and showing them to the other person.

[0059] Here, it is assumed that the first speaker 120 has the first voice recognition device 100, and the second speaker 220 has the second voice recognition device 200. The first voice recognition device 100 and the second voice recognition device 200 can be information terminals such as tablet terminals. The first speaker 120 wears the first voice recognition device 100 around his neck and points the display unit 113 toward the second speaker 220, and enables the imaging unit 101 to capture an image of the lip area 220m of the second speaker 220. Similarly, the second speaker 220 wears the second voice recognition device 200 around his neck and points the display unit 213 toward the first speaker 120, and enables the imaging unit 201 to capture an image of the lip area 120m of the first speaker 120.

[0060] First, a procedure for displaying the analysis result on the display unit 213 of the second speech recognition device 200 after the image and inaudible sound when the second speaker 220 speaks are analyzed by the first speech recognition device 100 will be described. FIG. 11 is a block diagram of speech recognition devices (100, 200) according to a first embodiment. The first speech recognition device 100 and the second speech recognition device 200 include a receiving unit (111, 211), a transmitting unit (112, 212), and a display unit (113, 213) in addition to the configuration of the speech recognition device 1001 shown in FIG. 1. The other configurations are the same as those of the speech recognition device 1001 shown in FIG. 1.

[0061] The imaging unit 101 of the first speech recognition device 100 acquires an image including the lip area 220m when the second speaker 220 speaks, "What's for dinner today?" without using the vocal cords. The inaudible sound detection unit 103 detects inaudible sounds propagating through the air from the voice of the second speaker 220 when he / she speaks.

[0062] The first speech recognition device 100 determines that the speech content of the second speaker 220 is "What's for dinner today?" from the acquired image of the lip region 220m and the inaudible sound, and the output unit 109 outputs the determined result to the transmission unit 112. The transmission unit 112 transmits information related to the speech content of the second speaker 220 to the reception unit 211 of the second speech recognition device 200.

[0063] The receiving unit 211 of the second speech recognition device 200 receives information about the utterance content and transmits it to the display unit 213. Based on the acquired information, the display unit 213 displays "What's for dinner today?" on the display screen. By visually checking the display unit 213 of the second speech recognition device 200, the first speaker 120 recognizes that the utterance of the second speaker 220 is "What's for dinner today?". At this time, the utterance of the second speaker 220 has temporarily ended, and the display unit 213 may display that a reply from the first speaker 120 is being waited for, making it easier for the first speaker 120 to time his / her utterance.

[0064] Next, a procedure for displaying the analysis result on the display unit 113 of the first voice recognition device 100 after the image and inaudible sound when the first speaker 120 speaks are analyzed by the second voice recognition device 200 will be described. The imaging unit 201 of the second voice recognition device 200 acquires an image including the lip area 120m when the first speaker 120 speaks "Curry" without using the vocal cords. In addition, the inaudible sound detection unit 203 detects inaudible sound propagating through the air from the voice when the first speaker 120 speaks.

[0065] The second speech recognition device 200 determines that the speech content of the first speaker 120 is "Curry" from the acquired image of the lip region 120m and the inaudible sound, and the output unit 209 outputs the determination result to the transmission unit 212. The transmission unit 212 transmits information regarding the speech content of the first speaker 120 to the reception unit 111 of the first speech recognition device 100.

[0066] Receiving unit 111 of first speech recognition device 100 receives information related to the utterance content and transmits it to display unit 113. Display unit 113 displays "Curry" on the display screen based on the acquired information. By visually checking display unit 113 of first speech recognition device 100, second speaker 220 recognizes that the content uttered by first speaker 120 is "Curry".

[0067] As described above, by using the voice recognition device according to the first embodiment, the first speaker 120 and the second speaker 220 can understand each other's speech and can communicate with each other even when they speak without using their vocal cords.

[0068] Next, as a modification of the first embodiment, a procedure will be described in which the first speech recognition device 100' acquires information on an image and inaudible sound when the second speaker 220 speaks, and then the acquired information is transmitted to the second speech recognition device 200', where the second speech recognition device 200' analyzes the information and displays the analysis result. Fig. 12 shows a block diagram of a speech recognition device according to the modification of the first embodiment.

[0069] The imaging unit 101 of the first speech recognition device 100' acquires an image including the lip area 220m when the second speaker 220 speaks, "What's for dinner today?" without using the vocal cords. The inaudible sound detection unit 103 detects inaudible sounds propagating through the air from the voice of the second speaker 220 when he / she speaks.

[0070] The image data acquired by the imaging unit 101 of the first speech recognition device 100' and the inaudible sound data acquired by the inaudible sound detection unit 103 are supplied to the transmission unit 112, which transmits these data to the receiving unit 211 of the second speech recognition device 200'.

[0071] The second speech recognition device 200' determines from the image of the lip area 220m and the inaudible sound received by the receiving unit 211 that the speech of the second speaker 220 is “What's for dinner today?” and displays the determined result on the display unit 213.

[0072] The first speaker 120 recognizes by viewing the display unit 213 of the second speech recognition device 200' that the content of the utterance by the second speaker 220 is "What's for dinner today?" At this time, the utterance by the second speaker 220 may be temporarily terminated and the display unit 213 may display that a reply from the first speaker 120 is being awaited, thereby making it easier for the first speaker 120 to time his / her utterance.

[0073] Next, a procedure will be described in which the content of speech uttered by the first speaker 120 is acquired by the second speech recognition device 200', and then the acquired information is transmitted to the first speech recognition device 100', where it is analyzed and displayed.

[0074] The imaging unit 201 of the second speech recognition device 200' captures an image including the lip area 120m when the first speaker 120 speaks "Curry" without using the vocal cords. In addition, the inaudible sound detection unit 203 detects inaudible sounds propagating through the air from the voice of the first speaker 120 when he / she speaks.

[0075] The image data acquired by the imaging unit 201 of the second speech recognition device 200' and the inaudible sound data acquired by the inaudible sound detection unit 203 are supplied to the transmission unit 212, which transmits these data to the receiving unit 111 of the first speech recognition device 100'.

[0076] The first speech recognition device 100 ′ determines that the speech of the first speaker 120 is “Curry” from the image of the lip region 120 m and the inaudible sound received by the receiving unit 111 , and displays the determination result on the display unit 113 .

[0077] The second speaker 220 visually checks the display unit 113 of the first speech recognition device 100' and thereby recognizes that the content uttered by the first speaker 120 is "Curry."

[0078] As described above, by using the voice recognition device according to the modified example of the first embodiment, the first speaker 120 and the second speaker 220 can understand each other's speech and can communicate with each other even when they speak without using their vocal cords.

[0079] [Example 2] Next, a voice recognition device according to a second embodiment will be described. FIG. 13 is a schematic diagram of an interpretation device using a voice recognition device 1002 according to the second embodiment. A caregiver 30 tries to understand the emotion of an elderly person 40 who is a speaker from the contents of the conversation, but when the elderly person 40 speaks without using his / her vocal cords, the caregiver 30 cannot hear the contents of the speech, and it may be difficult to understand the emotion. The voice recognition device 1002 according to the second embodiment recognizes the contents of the speech of the elderly person 40 by displaying the contents of the speech of the elderly person 40 without using his / her vocal cords on a display unit 13.

[0080] Fig. 14 is a block diagram of a voice recognition device 1002 according to the second embodiment. The voice recognition device 1002 according to the second embodiment includes a display unit 13 in addition to the voice recognition device 1001 shown in Fig. 1. The other configurations are similar to those of the voice recognition device 1001.

[0081] The imaging unit 1 of the voice recognition device 1002 captures an image including a lip area 40m when an elderly person 40, who is a speaker, speaks without using the vocal cords. In addition, the inaudible sound detection unit 3 detects inaudible sounds propagating through the air from the voice of the elderly person 40 when he or she speaks.

[0082] The voice recognition device 1002 determines the speech content of the elderly person 40 based on the acquired image including the lip area 40m and the detected inaudible sound, and the output unit 9 outputs data related to the determined speech content to the display unit 13. The display unit 13 displays the speech content based on the acquired data.

[0083] According to the voice recognition device 1002 of the second embodiment, even if the elderly person 40 speaks without using the vocal cords, the speech content of the elderly person 40 can be displayed on the display unit 13, and therefore the speech content of the elderly person 40 can be recognized.

[0084] [Example 3] Next, a voice recognition device according to a third embodiment will be described. FIG. 15 is a schematic diagram of a voice device operation system using a voice recognition device 1003 according to the third embodiment. In recent years, devices that operate home appliances by voice or control car navigation systems have been developed. These devices can be controlled by recognizing received voice, but there is a problem that even if an elderly person or the like tries to operate the device by speaking without using the vocal cords, these devices cannot recognize such voices uttered without using the vocal cords. The voice recognition device 1003 according to the third embodiment recognizes the contents of a speech uttered by an elderly person 50 without using the vocal cords, converts it into an audible voice, and causes a device 60 such as a home appliance to recognize the voice converted into an audible voice.

[0085] Fig. 16 is a block diagram of a voice recognition device 1003 according to the third embodiment. The voice recognition device 1003 according to the third embodiment includes a voice reproducing unit 14 in addition to the voice recognition device 1001 shown in Fig. 1. The other configurations are similar to those of the voice recognition device 1001.

[0086] The imaging unit 1 of the voice recognition device 1003 acquires an image including a lip area 50m when an elderly person 50, who is a speaker, speaks without using the vocal cords. In addition, the inaudible sound detection unit 3 detects inaudible sounds propagating through the air from the voice of the elderly person 50 when he or she speaks.

[0087] The voice recognition device 1003 determines the speech content of the elderly person 50 based on the acquired image including the lip area 50m and the detected inaudible sound, and the output unit 9 outputs the speech content to the voice reproduction unit 14. The voice reproduction unit 14 reproduces the speech content of the elderly person 50 as an audible sound, and the device 60 recognizes the reproduced audible sound and executes a predetermined control.

[0088] According to the voice recognition device 1003 of the third embodiment, even if the elderly person 50 speaks without using the vocal cords, the speech content can be converted into an audible sound and output from the voice playback unit 14, thereby improving the voice recognition accuracy in the device 60. [Explanation of symbols]

[0089] 1. Imaging unit 2 Lip movement trajectory detection section 3. Inaudible sound detector 4. Frequency pattern extraction section 5 Lip movement trajectory data storage unit 6 Inaudible sound pattern storage section 7. Utterance candidate extraction unit 8. Speech Decision Unit 9 Output section 10 Face image recognition unit 13 Display section 14 Audio playback section

Claims

1. an imaging unit for acquiring an image including a lip region during a speaker's speaking action; a lip movement trajectory detection unit for detecting a trajectory of a speaker's lip movement from the image; an inaudible sound detection unit that detects inaudible sounds propagating through the air from the voice of the speaker when the speaker speaks without using the vocal cords; a frequency pattern extraction unit that analyzes the frequency characteristics of the inaudible sound and extracts a frequency pattern; a lip movement trajectory data storage unit that stores in advance a correspondence relationship between the lip movement trajectory and kana characters; an inaudible sound pattern storage unit that stores in advance a correspondence relationship between the frequency patterns of the inaudible sounds and kana characters; an utterance candidate extraction unit that refers to the lip movement trajectory data storage unit and extracts kana character candidates from the lip movement trajectory; an utterance determination unit that, when the utterance candidate extraction unit extracts a plurality of kana character candidates, determines a specific kana character from among the plurality of kana character candidates by referring to the inaudible sound pattern storage unit and comparing the extracted frequency pattern with each of the stored frequency patterns of the plurality of kana character candidates; an output unit that outputs information about the kana characters determined by the utterance determination unit; A speech recognition device comprising:

2. The speech recognition device according to claim 1 , wherein the inaudible sound detection section starts detecting the inaudible sound of the speaker when a start of lip movement of the speaker detected by the lip movement trajectory detection section is used as a trigger.

3. 3. The voice recognition device according to claim 1, wherein the inaudible sound detection unit detects, as the inaudible sound, a sound wave having a frequency of 20 kHz or more and 70 kHz or less.

4. The speech recognition device according to claim 1 , wherein the utterance determination unit determines a kana character based on the presence or absence of a peak in the frequency pattern and the position of a peak occurring in a specific frequency band.

5. The speech recognition device according to claim 1 , wherein the plurality of kana characters having substantially the same lip movement trajectory include at least two of "na", "ta", and "da".

6. The speech recognition device according to claim 1 , wherein the plurality of kana characters having substantially the same lip movement trajectories include "shi" and "chi."

7. The speech recognition device according to claim 1 , wherein the plurality of kana characters having substantially the same lip movement trajectories include "a" and "ha."

Images