Speech detection device

Abstract

To provide an utterance detection device that acquires an uttered voice without disconnection of a beginning part while preventing occurrence of howling and echo, mixture of noise, and the like.SOLUTION: In a calling system 1, an utterance detection device 10 that detects utterance of a user, and outputs the detected uttered voice includes: a lip motion detection unit 13 that detects a pre-utterance motion based on the movement of the lip of a face image obtained by imaging the face of the user; a voice detection unit 14 that detects voice based on a voice signal from a voice detection device; a determination unit 15 that determines that the voice signal is to be validated in a case where the pre-utterance motion is detected, and that the voice signal is to be invalidated in a case where no voice is detected within an invalidation determination time which is a predetermined time set in advance after validation of the voice signal; and a control unit 16 that controls validation and invalidation of the voice signal in accordance with the determination about validation or invalidation of the voice signal made by the determination unit 15.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a speech detection device that detects a user's speech and outputs the detected speech sound. [Background technology]

[0002] In systems that detect and acquire a user's voice, such as a telephone call system and a voice recognition system, in order to prevent the occurrence of howling and echo, as well as the inclusion of noise, it is preferable to disable the voice signal by controlling, for example, turning off acquisition of voice by a microphone when the user is not speaking, and to activate the voice signal only when the user is speaking. Generally, a technique for activating a voice signal when voice is detected is known. Also, a technique for detecting a speech section by detecting lip movement based on an image of a speaker is known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-79624 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a technology that activates a voice signal when voice is detected, the voice signal is activated after voice detection, so that the beginning of the speech is not acquired, that is, the beginning is cut off. Also, when detecting a speech section by lip movement as in the technology described in Patent Document 1, lip movement that is not associated with speech is erroneously detected, and there are cases where it is not possible to prevent the inclusion of noise.

[0005] Therefore, the present invention has been made in consideration of the above problems, and has an object to acquire speech without cutting off the beginning part of the speech in a system for detecting and acquiring speech, while preventing the occurrence of howling and echo and the introduction of noise, etc. [Means for solving the problem]

[0006] In order to solve the above problem, a speech detection device according to one embodiment of the present invention is a speech detection device that detects a user's speech and outputs the detected speech sound, and includes a lip movement detection unit that detects a pre-defined speech preparatory movement as lip movement during speech based on a facial image of the user's face, a voice detection unit that detects voice based on a voice signal from a voice detection device provided for detecting the user's speech, a judgment unit that determines whether to enable or disable the voice signal, the judgment unit enabling the voice signal when a speech preparatory movement is detected, and determining to disable the voice signal when voice is not detected within a predetermined disable judgment time after the activation of the voice signal, which is a predetermined time after the activation of the voice signal, and a control unit that controls the enabling and disabling of the voice signal in accordance with the judgment of the judgment unit to enable or disable the voice signal.

[0007] According to the above embodiment, when the movement of the lips occurs slightly before the speech during speech, the voice signal is activated when a preparatory movement for speech defined as the movement of the lips is detected based on the face image, so that the speech voice can be acquired without cutting off the beginning part. Furthermore, even if the voice signal is once activated based on the movement of the lips that is not accompanied by speech, the voice signal is disabled if no voice is detected within the disablement determination time after the voice signal is enabled, so that the occurrence of howling and echo, the introduction of noise, etc. can be prevented. Effect of the Invention

[0008] In a system for detecting and acquiring voice, it is possible to acquire a speech sound without cutting off the beginning portion while preventing the occurrence of howling and echo, as well as the inclusion of noise. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a device configuration when an utterance detection device according to an embodiment of the present invention is applied to a telephone call system. [Diagram 2] 1 is a block diagram showing a functional configuration of an utterance detection device according to an embodiment of the present invention; [Diagram 3] FIG. 2 is a hardware block diagram of the speech detection device. [Figure 4] 11A and 11B are diagrams illustrating examples of face images for detecting preparatory movements for speech. [Diagram 5] 11 is a flowchart showing an example of processing content of an utterance detection method in the utterance detection device. [Figure 6] 13 is a flowchart showing another example of the processing contents of the utterance detection method in the utterance detection device. [Figure 7] FIG. 4 is a diagram showing a configuration of an utterance detection program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a speech detection device according to the present invention will now be described with reference to the drawings. Whenever possible, the same reference numerals are used to designate the same parts, and duplicated explanations will be omitted.

[0011] FIG. 1 is a diagram showing an example of a device configuration when the speech detection device of this embodiment is applied to a call system. As shown in FIG. 1, a plurality of call systems 1 are configured to be able to communicate with each other via a network N. The speech detection device 10 is a device that detects a user's speech and outputs the detected speech voice. Therefore, the speech detection device 10 of this embodiment can be applied to the call system 1 as shown in FIG. 1. Specifically, the speech detection device 10 applied to the call system 1 detects the user's speech and transmits the detected speech voice to the call system 1 of the other party via the network N.

[0012] 2 is a block diagram showing the device configuration of the communication system 1 and the functional configuration of the speech detection device 10 according to this embodiment. As shown in FIG. 2, the communication system 1 includes the speech detection device 10, a camera 20, and a microphone 30 (voice detection device). The communication system 1 may further include a display 40 and a speaker 50.

[0013] Functionally, the speech detection device 10 comprises an image acquisition unit 11, a voice acquisition unit 12, a lip movement detection unit 13, a voice detection unit 14, a determination unit 15, and a control unit 16. Each of these functional units 11 to 16 may be configured in one device (computer) or may be distributed across multiple devices.

[0014] The camera 20 captures an image of the user's face. The camera 20 sends the captured face image of the user to the speech detection device 10. The microphone 30 detects voice. The microphone 30 sends an audio signal consisting of the detected voice to the speech detection device 10.

[0015] The display 40 displays, for example, an image of the other user who is communicating with the user through the communication system 1, received via the network N. The speaker 50 outputs, for example, the voice of the other user who is communicating with the user through the communication system 1, received via the network N.

[0016] The block diagram shown in FIG. 2 shows functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0017] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.

[0018] For example, the speech detection device 10 according to an embodiment of the present invention may function as a computer. Fig. 3 is a diagram showing an example of a hardware configuration of the speech detection device 10 according to this embodiment. The speech detection device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0019] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the speech detection apparatus 10 may be configured to include one or more of the apparatuses shown in FIG. 3, or may be configured to exclude some of the apparatuses.

[0020] Each function in the speech detection device 10 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, so that the processor 1001 performs calculations and controls communication via the communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.

[0021] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each of the functional units 11 to 16 shown in FIG. 2 may be realized by the processor 1001.

[0022] Furthermore, the processor 1001 reads out a program (program code), software modules, and data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes according to the programs. The programs used are those that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments. For example, the functional units 11-16 of the speech detection device 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented in one or more chips. The programs may be transmitted from a network via a telecommunication line.

[0023] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement the speech detection method according to one embodiment of the present invention.

[0024] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray® disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and / or storage 1003.

[0025] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0026] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0027] Furthermore, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

[0028] Furthermore, the speech detection device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

[0029] 2 again, the functional units of the speech detection device 10 will be described. The image acquisition unit 11 acquires a face image of a user captured by the camera 20. The voice acquisition unit 12 acquires a voice signal consisting of a voice captured by the microphone 30.

[0030] The lip movement detection unit 13 detects preparatory movements for speech defined in advance as movements of the lips during speech based on the face image acquired by the image acquisition unit 11. An example of the preparatory movements for speech will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of a face image for detecting the preparatory movements for speech.

[0031] The lip movement detection unit 13 extracts the lip part lp from the user's face image f, and sets landmarks mu, ml, cl, and cr on the extracted lip part lp, for example, on the upper lip (the point with the largest upward coordinate in the lip part lp), the lower lip (the point with the smallest upward coordinate in the lip part lp), and the left and right corners of the mouth (the left and right end points in the lip part lp). Any image processing technique may be applied to extract the lip part lp and set the landmarks, and for example, a machine-learned model may be applied to extract the face and lip parts from a captured image, or other well-known image processing techniques (e.g., pattern matching, etc.) may be used to extract the lip part lp and set the landmarks. The lip movement detection unit 13 may also obtain the face length fl and width fw from the face image f.

[0032] The lip movement detection unit 13 may detect, as a preparatory movement for speech, that the size of the area occupied by the lip part lp in the face image f is equal to or larger than a predetermined size. Specifically, the lip movement detection unit 13 may obtain the size of the area surrounded by the line segments connecting the landmarks mu, ml, cl, and cr as an index of the size of the lip part lp. The lip movement detection unit 13 may also obtain, as an index of the size of the lip part lp, the ratio of the area surrounded by the line segments connecting the landmarks mu, ml, cl, and cr to the area of ​​the face shown in the face image f.

[0033] The lip movement detection unit 13 may also obtain the length of the line segment connecting the landmark mu (upper end of the lips) and the landmark ml (lower end of the lips) as an index of the size of the lip portion lp, or obtain the ratio of the length of the line segment connecting the landmark mu and the landmark ml to the face length fl as an index of the size of the lip portion lp. The lip movement detection unit 13 may also obtain the length of the line segment connecting the landmark cl (left end of the lips) and the landmark cr (right end of the lips) as an index of the size of the lip portion lp, or obtain the ratio of the length of the line segment connecting the landmark cl and the landmark cr to the face width fw as an index of the size of the lip portion lp.

[0034] The lip movement detection unit 13 can determine that a preparatory movement for speech has been detected when any one, multiple, or all of the various indices of the size of the lip portion lp described above exceed a preset threshold. Considering that a lip portion area having a predetermined size or larger is highly likely to indicate a speaking state, the lip movement detection unit 13 can appropriately detect a preparatory movement for speech by acquiring an index of the size of the lip portion lp.

[0035] The lip movement detection unit 13 may detect, as a preparatory movement for speech, that the amount of change per unit time of at least one of the shape and size of the lip part lp in the face image f is equal to or greater than a predetermined amount. Specifically, the lip movement detection unit 13 may acquire, as an index of the amount of change in the shape of the lip part lp, the amount of change per unit time (for example, a predetermined number of consecutive frames of a captured moving image) of the coordinates of each of the landmark points mu, ml, cl, and cr, and the amount of change per unit time of the relative coordinates of each of the landmark points mu, ml, cl, and cr with respect to the center point of the lip part lp.

[0036] Furthermore, the lip movement detection section 13 may obtain the amount of fluctuation per unit time of the various indices of the size of the lip portion lp as described above as an index of the amount of fluctuation in the size of the lip portion lp.

[0037] The lip movement detection unit 13 can determine that a preparatory movement for speech has been detected when any one, multiple, or all of the above-mentioned various indexes of the amount of change in the shape and size of the lip part lp exceed a preset threshold. Considering that a change in the shape or size of the lip part is more than a predetermined amount, it is highly likely that the person is in a speaking state, the lip movement detection unit 13 can appropriately detect a preparatory movement for speech by acquiring an index of the amount of change in the shape and size of the lip part lp.

[0038] The lip movement detection unit 13 may detect, as a preparatory movement for speech, that at least one of the shape and movement of the lip part lp in the face image f corresponds to a predetermined shape pattern or a predetermined movement pattern.

[0039] Specifically, the lip movement detection unit 13 can determine that it has detected a preparatory movement for speech when indices such as the size, amount of change in shape, and amount of change in size of the above-mentioned lip part lp correspond to a predetermined shape pattern or a predetermined movement pattern.

[0040] The predetermined pattern of shape and movement may be, for example, a variation pattern when a vowel is uttered. The variation pattern when a vowel is uttered may be a pattern of opening the lips up and down when uttering "a", "e", or "o", a pattern of stretching the lips left and right when uttering "i", and a pattern of pursing the lips when uttering "u".

[0041] The predetermined pattern of shape and movement may be, for example, a variation pattern when a predetermined consonant is produced. The variation pattern when a consonant is uttered may be a pattern when the lips are pressed before a plosive sound, such as when uttering the "pa" row.

[0042] In this way, by detecting the shape or movement pattern of the lips that is highly likely to indicate a speaking state, it is possible to appropriately detect a preparatory movement for speech based on the face image f.

[0043] The lip movement detection unit 13 may detect a nodding action when the user makes a backchannel response as a preparatory movement for speech. Specifically, the lip movement detection unit 13 may detect a predetermined downward movement of a landmark set for the lip portion lp as a nodding action, and detect this as a preparatory movement for speech. The speech detection device 10 may also detect the direction of the user's gaze from the face image f, and detect a predetermined change in the detected gaze direction as a nodding action, and detect this as a preparatory movement for speech.

[0044] Referring back to FIG. 2, the voice detection unit 14 detects voice based on a voice signal from a microphone 30 (voice detection device) provided for detecting the user's speech.

[0045] The voice detection unit 14 may detect voice when the voice signal acquired by the voice acquisition unit 12 is equal to or higher than a predetermined level. Alternatively, the voice detection unit 14 may detect voice based on the voice signal acquired by the voice acquisition unit 12 using a technique such as VAD (Voice Activity Detection). Alternatively, the voice detection unit 14 may detect voice using other well-known voice detection techniques.

[0046] The determination unit 15 determines whether to enable or disable the audio signal. Specifically, the determination unit 15 determines to enable the audio signal acquired by the audio acquisition unit 12 when a preparatory movement for speech is detected by the lip movement detection unit 13, and determines to disable the audio signal when no audio is detected by the audio detection unit 14 within an invalidation determination time, which is a predetermined time set in advance after the audio signal is enabled.

[0047] The control unit 16 controls the enabling and disabling of the audio signal in response to the determination of the enabling or disabling of the audio signal by the determination unit 15. Specifically, the control unit 16 may control the disabling of the audio signal when the determination unit 15 determines that the audio signal should be disabled by disabling the audio detection function of the microphone 30. This ensures that the audio signal is disabled.

[0048] Furthermore, the control unit 16 may control the disabling of the audio signal when the determination unit 15 determines that the audio signal should be disabled, by changing the mode (control state) of the speech detection device 10 to a mode in which the enabling or disabling of the audio signal is determined based only on the presence or absence of detection of audio based on the audio signal. In this way, the determination that the audio signal should be disabled is made when a state in which no audio is detected continues, so that the control of disabling the audio signal is reliably performed.

[0049] Furthermore, the control unit 16 may control the invalidation of the voice signal when the determination unit 15 determines to invalidate the voice signal by not outputting the speech sound. When the speech detection device 10 constitutes a communication system for communicating with a counterpart party, the control unit 16 controls the invalidation of the voice signal by not transmitting the voice signal consisting of the speech sound to the counterpart party's communication system. The control unit 16 may also control the invalidation of the voice signal by not outputting the voice signal from a speaker, storing it in a predetermined storage means, or displaying it on a predetermined display means. Such control makes it possible to reliably invalidate the voice signal without the need to control the microphone 30.

[0050] Furthermore, the control unit 16 may execute the disabling control by disabling the audio signal at any stage of the path from the input to the output of the audio signal.

[0051] When speaking, lip movement occurs slightly before the speech, and thus the audio signal is enabled when a preparatory speech movement defined as lip movement is detected based on the face image f, so that the speech sound can be acquired without cutting off the beginning part. Furthermore, even if the audio signal is once enabled based on lip movement not accompanied by speech, the audio signal is disabled if no sound is detected within the disablement determination time after the audio signal is enabled, so that the occurrence of howling and echo, as well as the introduction of noise, can be prevented.

[0052] Next, the setting of the invalidation judgment time for judging invalidation of the voice signal after the voice signal is validated will be described. The judgment unit 15 may set the invalidation judgment time to a first time when a first preparatory movement to utterance is detected among the defined preparatory movements to utterance, and may set the invalidation judgment time to a second time longer than the first time when a second preparatory movement to utterance is detected among the defined preparatory movements to utterance.

[0053] The first speech preparatory movement is a lip movement that is predefined as a lip movement that is highly likely to cause speech to start immediately, and the second speech preparatory movement is a lip movement that is predefined as a lip movement that is difficult to determine as causing speech to start immediately.

[0054] The detection of the first preparatory movement for speech may be based on at least one of the following: the size of the lips being larger than / above a predetermined threshold, the amount of change in the size of the lips being larger than / above a predetermined threshold, and the movement of the lips conforming to a predetermined movement pattern.

[0055] The determination unit 15 may detect, as the first preparatory movement for speech, that the movement of the lips corresponds to a lip movement pattern specific to utterance. The lip movement pattern specific to utterance may be, for example, a movement pattern when a vowel is uttered and a movement pattern when a predetermined consonant is uttered.

[0056] The second preparatory movement for utterance may be detected by at least one of the following: the size of the lips is equal to or smaller than a predetermined threshold value, the amount of change in the size of the lips is equal to or smaller than the predetermined threshold value, and the movement of the lips does not correspond to a predetermined movement pattern. That is, the second preparatory movement for utterance may be a defined preparatory movement for utterance that does not correspond to the first preparatory movement for utterance.

[0057] In this way, when the voice signal is enabled based on lip movement that is likely to indicate the start of speech immediately, the time spent on determining whether to disable the voice signal after the voice signal is enabled is shorter than when the voice signal is enabled based on lip movement that is difficult to determine as the start of speech immediately, and the invalidation determination time that is the criterion for determining whether to disable the voice signal is set to be shorter. Therefore, it is possible to shorten the time that the voice signal is unnecessarily enabled.

[0058] 5 is a flowchart showing the processing contents of the speech detection method in the speech detection device 10. When the speech detection processing is started, the image acquisition processing by the image acquisition unit 11 and the sound acquisition processing by the sound acquisition unit 12 are performed continuously or intermittently at predetermined time intervals.

[0059] In step S1, the voice detection unit 14 determines whether or not voice has been detected. If it is determined that voice has been detected, the process proceeds to step S3. On the other hand, if it is determined that voice has not been detected, the process proceeds to step S2.

[0060] In step S2, the lip movement detection unit 13 judges whether or not a preparatory movement for speech has been detected. If it is judged that a preparatory movement for speech has been detected, the process proceeds to step S3. On the other hand, if it is not judged that a preparatory movement for speech has been detected, the process proceeds to step S4.

[0061] In step S3, the determination unit 15 determines to enable the audio signal and records the audio signal enable time Ts, which is the time when the audio signal is enabled. Then, the control unit 16 performs control to enable the audio signal.

[0062] In step S4, the decision unit 15 decides to maintain the state in which the audio signal is disabled, and the process returns to step S1.

[0063] In step S5, the voice detection unit 14 judges whether or not voice is detected. If voice is detected here, it means that the user actually speaks following the preparatory movement for speech. If it is judged that voice is detected, the process proceeds to step S7. On the other hand, if it is not judged that voice is detected, the process proceeds to step S6.

[0064] In step S6, the determination unit 15 determines whether the current time t is later than the time when the invalidation determination time T has elapsed since the audio signal activation time Ts. If it is determined that the current time t is later than the time when the invalidation determination time T has elapsed since the audio signal activation time Ts, the process proceeds to step S8. On the other hand, if it is not determined that the current time t is later than the time when the invalidation determination time T has elapsed since the audio signal activation time Ts, the process proceeds to step S7.

[0065] Prior to the judgment process of step S6, the judgment unit 15 may determine to set the invalidation judgment time T to either a first time T1 or a second time T2 longer than the first time T1, based on whether the detected speech preparatory movement corresponds to the first or second speech preparatory movement.

[0066] In steps S5 to S6, if no voice is detected and the invalidation determination time T has elapsed from the voice signal activation time Ts, this corresponds to a case where a preparatory movement for speech has been detected but the movement is not accompanied by actual speech.

[0067] In step S7, the decision unit 15 decides to maintain the enabled state of the audio signal, and the process returns to step S5.

[0068] In step S8, the determination unit 15 determines to invalidate the audio signal, and the process returns to step S1.

[0069] In addition, in the case where a voice such as noise that is not the user's speech continues to be detected at the stage of step S5, the voice signal is maintained in an enabled state. In order to prevent such an unnecessary state of the voice signal being maintained in an enabled state, if no lip movement is detected when the invalidation determination time T has elapsed from the voice signal activation time Ts while the voice continues to be detected, the processing procedure may be advanced to step S8, where the determination unit 15 may determine to invalidate the voice signal.

[0070] The processing content of the speech detection method in the speech detection device 10 may be executed as shown in the flowchart of Fig. 6. When the speech detection processing is started, the image acquisition processing by the image acquisition unit 11 and the sound acquisition processing by the sound acquisition unit 12 are performed continuously or intermittently at a predetermined time interval.

[0071] In step S11, the control unit 16 determines whether or not the audio signal is enabled. If it is determined that the audio signal is enabled, the process proceeds to step S17. On the other hand, if it is not determined that the audio signal is enabled, the process proceeds to step S12.

[0072] In step S12, the lip movement detection unit 13 judges whether or not a preparatory movement for speech has been detected. If it is judged that a preparatory movement for speech has been detected, the process proceeds to step S13. On the other hand, if it is not judged that a preparatory movement for speech has been detected, the process returns to step S11.

[0073] In step S13, the determination unit 15 determines to enable the audio signal, and the control unit 16 performs control to enable the audio signal. Then, the process proceeds to step S14.

[0074] In step S14, the judgment unit 15 sets an invalidation judgment time T and waits for the set time T. Then, the process proceeds to step S15. Note that the judgment unit 15 may determine to set the invalidation judgment time T to either a first time T1 or a second time T2 longer than the first time T1, based on whether the detected preparatory movement for speech corresponds to the first or second preparatory movement for speech.

[0075] In step S15, the voice detection unit 14 judges whether or not voice has been detected. If voice has been detected here, this corresponds to a case where the user actually speaks following the preparatory movement to speak, etc. If it is not judged that voice has been detected, this corresponds to a case where a preparatory movement to speak is detected but the movement is not accompanied by actual speech, etc. If it is judged that voice has been detected, the process returns to step S11. On the other hand, if it is not judged that voice has been detected, the process proceeds to step S16.

[0076] In step S16, the determination unit 15 determines to invalidate the audio signal, and the control unit 16 performs control to invalidate the audio signal. Then, the process returns to step S11.

[0077] In step S17, the lip movement detection unit 13 judges whether or not a preparatory movement for speech has been detected. If it is judged that a preparatory movement for speech has been detected, the process proceeds to step S14. On the other hand, if it is not judged that a preparatory movement for speech has been detected, the process proceeds to step S16.

[0078] According to the operation of the flowchart shown in FIG. 6, even if the speech detection unit 14 continues to detect speech such as noise that is not the user's speech, if lip movement is not detected, the speech is not activated or is immediately deactivated, thereby preventing speech such as noise from being unnecessarily maintained in an activated state.

[0079] Next, with reference to FIG. 7, an utterance detection program for causing a computer to function as the utterance detection device 10 of this embodiment will be described.

[0080] 7 is a diagram showing the configuration of the speech detection program. The speech detection program P1 includes a main module m10 that controls the speech detection process in the speech detection device 10, an image acquisition module m11, a voice acquisition module m12, a lip movement detection module m13, a voice detection module m14, a judgment module m15, and a control module m16. The modules m11 to m16 realize the functions of the image acquisition unit 11, the voice acquisition unit 12, the lip movement detection unit 13, the voice detection unit 14, the judgment unit 15, and the control unit 16.

[0081] The speech detection program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG.

[0082] According to the speech detection device 10, speech detection method, and speech detection program P1 of the present embodiment described above, when lip movement occurs slightly before speech during speech, the audio signal is enabled when a preparatory movement for speech defined as lip movement is detected based on a face image, so that speech can be acquired without cutting off the beginning part. Furthermore, even if the audio signal is once enabled based on lip movement not accompanied by speech, the audio signal is disabled if no speech is detected within a predetermined time after the audio signal is enabled, so that the occurrence of howling and echo, as well as the introduction of noise, can be prevented.

[0083] In addition, in another embodiment of the speech detection device, the judgment unit sets the invalidation judgment time to a first time when it detects a first speech preparatory movement among the specified speech preparatory movements, and sets the invalidation judgment time to a second time longer than the first time when it detects a second speech preparatory movement among the speech preparatory movements, wherein the first speech preparatory movement is a lip movement that is predefined as a lip movement that is likely to result in instantaneous speech start, and the second speech preparatory movement is a lip movement that is predefined as a lip movement that is difficult to determine as instantaneous speech start.

[0084] According to the above embodiment, when the audio signal is enabled based on lip movement that is highly likely to indicate the start of speech immediately, the invalidation determination time that is the criterion for determining whether to invalidate the audio signal when no voice is detected after the audio signal is enabled is set shorter than when the audio signal is enabled based on lip movement that is difficult to determine as the start of speech immediately. Therefore, it is possible to shorten the time for which the audio signal is unnecessarily enabled.

[0085] In addition, in another embodiment of the speech detection device, the detection of a first preparatory movement for speech may be at least one of the following: the size of the lips is larger than / above a predetermined threshold, the amount of change in the size of the lips is larger than / above a predetermined threshold, and the movement of the lips corresponds to a predetermined movement pattern; and the detection of a second preparatory movement for speech may be at least one of the following: the size of the lips is smaller than / below a predetermined threshold, the amount of change in the size of the lips is smaller than / below a predetermined threshold, and the movement of the lips does not correspond to a predetermined movement pattern.

[0086] According to the above aspect, it is possible to appropriately detect lip movement, which is difficult to determine with certainty when speech is started instantly.

[0087] In the speech detection device according to another aspect, the lip movement detection unit may detect, as a preparatory movement for speech, that a size of an area occupied by the lips in the face image is equal to or larger than a predetermined size.

[0088] According to the above aspect, since there is a high possibility that the person is in a speaking state when the area of ​​the lip portion is equal to or larger than a predetermined size, it is possible to appropriately detect a preparatory movement for speech based on a face image.

[0089] In addition, in another embodiment of the speech detection device, the lip movement detection unit may detect, as a preparatory movement for speech, the amount of change per unit time in at least one of the shape and size of the lip portion in the facial image being greater than or equal to a predetermined amount.

[0090] According to the above aspect, when the amount of change in the shape or size of the lips is equal to or greater than a predetermined amount, it is highly likely that the person is in a speaking state, so that a preparatory movement for speech can be appropriately detected based on a face image.

[0091] In addition, in another embodiment of the speech detection device, the lip movement detection unit may detect as a preparatory movement for speech that at least one of the shape and movement of the lip portion in the facial image corresponds to a predetermined shape pattern or a predetermined movement pattern.

[0092] According to the above aspect, when the shape or movement of the lips corresponds to a predetermined pattern, it is highly likely that the person is in a speaking state, so that the preparatory movement for speech can be appropriately detected based on the face image.

[0093] In addition, in another embodiment of the speech detection device, the control unit may be configured to control the disabling of the voice signal when the judgment unit determines that the voice signal should be disabled by disabling the voice detection function of the voice detection device.

[0094] According to the above embodiment, the audio signal is reliably invalidated.

[0095] In addition, in another embodiment of the speech detection device, the control unit may perform control of disabling the audio signal when the judgment unit determines that the audio signal should be disabled, by changing the control state in the speech detection device to a control state that determines whether the audio signal is enabled or disabled based solely on whether or not audio is detected based on the audio signal.

[0096] According to the above aspect, since it is determined to invalidate the audio signal when a state in which no audio is detected continues, the control of invalidating the audio signal is reliably performed.

[0097] In addition, in a speech detection device according to another embodiment, the control unit may perform control of disabling the audio signal when the judgment unit determines that the audio signal should be disabled by not outputting the speech sound.

[0098] According to the above aspect, it is possible to reliably invalidate the audio signal without the need for control of the audio detection device.

[0099] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as a modified or altered form without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

[0100] Each aspect / embodiment described herein may be applied to systems utilizing LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), or other suitable systems and / or next generation systems enhanced thereon.

[0101] The steps, sequences, flow charts, etc. of each aspect / embodiment described herein may be reordered unless inconsistent, For example, the methods described herein present elements of various steps in an example order, and are not limited to the particular order presented.

[0102] The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0103] The determination may be based on a value represented by a single bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).

[0104] Each aspect / embodiment described in this specification may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0105] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0106] Additionally, software, instructions, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies, such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.

[0107] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0108] In addition, terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0109] As used herein, the terms "system" and "network" are used interchangeably.

[0110] Furthermore, the information, parameters, etc. described in this specification may be expressed as absolute values, may be expressed as relative values ​​from a predetermined value, or may be expressed as other corresponding information.

[0111] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0112] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0113] When designations such as "first," "second," and the like are used herein, any reference to an element is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

[0114] To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

[0115] In this specification, a plurality of devices is also included unless the context or technical basis clearly indicates that only one device is present.

[0116] Throughout this disclosure, the plural is intended to be included unless the context clearly indicates the singular. [Explanation of symbols]

[0117] 1...communication system, 10...speech detection device, 11...image acquisition unit, 12...voice acquisition unit, 13...lip movement detection unit, 14...voice detection unit, 15...judgment unit, 16...control unit, 20...camera, 30...microphone, 40...display, 50...speaker, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus, M1...recording medium, m10...main module, m11...image acquisition module, m12...voice acquisition module, m13...lip movement detection module, m14...voice detection module, m15...judgment module, m16...control module, N...network, P1...speech detection program.

Claims

1. A speech detection device that detects a user's speech and outputs the detected speech, a lip movement detection unit that detects a preparatory movement for speech defined in advance as a movement of lips during speech based on a face image obtained by capturing the face of the user; a voice detection unit that detects voice based on a voice signal from a voice detection device provided to detect the user's speech; a determination unit that determines whether to enable or disable the voice signal, the determination unit determining to enable the voice signal when the preparatory movement for speech is detected, and to disable the voice signal when the voice is not detected within a disable determination time, which is a predetermined time after the enablement of the voice signal; a control unit that controls the activation and deactivation of the audio signal in response to a determination of the activation or deactivation of the audio signal by the determination unit, the determination unit sets the invalidation judgment time to a first time when a first preparatory movement for speech among the defined preparatory movements for speech is detected, and sets the invalidation judgment time to a second time longer than the first time when a second preparatory movement for speech among the defined preparatory movements for speech is detected; the first preparatory movement for speech is a movement of lips that is predefined as a movement of lips that is highly likely to cause an instantaneous start of speech, The second speech preparatory movement is a lip movement that is predefined as a lip movement that is difficult to determine as an instantaneous speech start. Speech detection device.

2. The detection of the first preparatory movement for speech corresponds to at least one of the following: a size of lips being larger than / equal to a predetermined threshold, an amount of change in the size of lips being larger than / equal to a predetermined threshold, and a movement of the lips corresponding to a predetermined movement pattern; The detection of the second preparatory movement for speech is at least one of the following: a size of lips is equal to or smaller than a predetermined threshold value; a variation in the size of lips is equal to or smaller than the predetermined threshold value; and the movement of lips does not correspond to the predetermined movement pattern. The speech detection device according to claim 1 .

3. the lip movement detection unit detects, as the preparatory movement for speech, that a size of an area occupied by lips in the face image is equal to or larger than a predetermined size; 3. The speech detection device according to claim 1 or 2.

4. the lip movement detection unit detects, as the preparatory movement for speech, a change amount per unit time of at least one of a shape and a size of a lip portion in the face image being equal to or greater than a predetermined amount; The speech detection device according to any one of claims 1 to 3.

5. the lip movement detection unit detects, as the preparatory movement for speech, that at least one of a shape and a movement of a lip part in the face image corresponds to a predetermined shape pattern or a predetermined movement pattern; The speech detection device according to any one of claims 1 to 4.

6. The control unit performs control of disabling the voice signal when the determination unit determines that the voice signal should be disabled by disabling a voice detection function of the voice detection device. The speech detection device according to any one of claims 1 to 5.

7. the control unit changes a control state of the speech detection device to a control state in which the voice signal is enabled or disabled based only on the presence or absence of detection of voice based on the voice signal, thereby performing control of disabling the voice signal when the determination unit determines that the voice signal should be disabled. The speech detection device according to any one of claims 1 to 6.

8. The control unit performs control of invalidating the voice signal when the determination unit determines that the voice signal is to be invalidated by not outputting the speech sound. The speech detection device according to any one of claims 1 to 7.

Images