Voice section detection device and voice section detection method

Abstract

To reduce a calculation amount in detection of a voice section.SOLUTION: A voice section detection device 1 includes: a sound signal acquisition part 11 for acquiring a sound signal indicating sound; a voice signal detection part 12 for detecting a voice signal indicating voice of a person in the sound signal based on the sound signal; a video signal acquisition part 13 for acquiring a video signal indicating a video; a video determination part 14 for determining presence or absence of an event on a target person in the video based on the video signal; a voice determination part 15 for determining presence or absence of voice of the target person based on a detection result of the voice signal detection part 12 and a determination result of the video determination part 14; a quality determination part 16 for determining quality of the video indicated by the video signal acquired by the video signal acquisition part 13 based on determination results of the video determination part 14 and the voice determination part 15; and a voice section detection part 17 for detecting a voice section of the target person from the voice signal based on the determination result of the voice determination part 15.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present disclosure relates to a voice activity detection device and a voice activity detection method. [Background technology]

[0002] In the field of voice communication and voice recognition, there is known a technique for detecting a voice section where a person (user)'s voice exists. The voice section detection device described in Patent Document 1 detects a lip movement signal by applying lip movement video feature information to a movement area detected in a video frame based on a video signal captured by a video capture device. This voice section detection device detects the voice section using a sound signal (acoustic signal) input by a person via a microphone or the like and the detected lip movement signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-156493 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above voice activity detection device, the voice activity is detected using a video signal captured by a video capture device and an input audio signal. Therefore, compared with a general voice activity detection device that detects a voice activity using only an audio signal, the above voice activity detection device may require a larger amount of calculation to perform video processing. Therefore, it is required to reduce the amount of calculation required for voice activity detection.

[0005] The present disclosure has been made in consideration of the above-described circumstances, and has an object to reduce the amount of calculation required for detecting a voice activity. [Means for solving the problem]

[0006] A voice section detection device according to one aspect of the present disclosure includes a sound signal acquisition unit that acquires a sound signal indicative of sound, an audio signal detection unit that detects an audio signal indicative of a person's voice in the sound signal based on the sound signal, a video signal acquisition unit that acquires a video signal indicative of a video, a video judgment unit that determines whether or not an event related to a target person in the video is present based on the video signal, an audio judgment unit that determines whether or not the target person's voice is present based on the detection result of the audio signal detection unit and the judgment result of the video judgment unit, a quality judgment unit that determines the quality of the video represented by the video signal acquired by the video signal acquisition unit based on the judgment results of the video judgment unit and the audio judgment unit, and a voice section detection unit that detects the voice section of the target person from the audio signal based on the judgment result of the audio judgment unit.

[0007] A voice section detection method according to one aspect of the present disclosure includes a sound signal acquisition step of acquiring a sound signal indicative of sound, an audio signal detection step of detecting an audio signal indicative of a person's voice in the sound signal based on the sound signal, a video signal acquisition step of acquiring a video signal indicative of a video, a video judgment step of determining whether or not an event related to a target person in the video is present based on the video signal, an audio judgment step of determining whether or not the target person's voice is present based on the detection result of the audio signal detection step and the judgment result of the video judgment step, a quality determination step of determining the quality of the video represented by the video signal acquired by the video signal acquisition step based on the judgment results of the video judgment step and the audio judgment step, and a voice section detection step of detecting the voice section of the target person from the audio signal based on the judgment result of the audio judgment step.

[0008] In the voice activity detection device and the voice activity detection method according to an aspect of the present disclosure, the quality of the video shown by the acquired video signal is determined based on the determination result of the presence or absence of an event related to the target person and the presence or absence of the target person's voice. For example, the presence or absence of an object related to the target person, such as the face of the target person, is determined as the event related to the target person. For example, if it is determined that there is an object related to the target person and that there is a voice of the target person, it is highly likely that the target person is present and is speaking. In this case, since there is no need to detect the timing when the target person starts to make a sound, the voice activity can be appropriately detected even without high quality of the video. Also, for example, if it is determined that there is no object, it is highly likely that the target person has left the seat and the voice of the target person will not be detected, so the voice activity can be appropriately detected even if the quality of the video is low. In this way, the quality of the video shown by the acquired video signal can be reduced according to the situation of the target person, so the amount of calculation in detecting the voice activity can be reduced. Effect of the Invention

[0009] According to the present disclosure, the amount of calculation required for detecting a speech segment can be reduced. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a voice activity detection system including a voice activity detection device according to an embodiment. [Diagram 2] 4 is a flowchart illustrating a voice activity detection method according to the embodiment. [Diagram 3] 3 is a flowchart showing a video quality determination process shown in FIG. 2. [Figure 4] 13 is a flowchart showing a video quality determination process according to a modified example. [Diagram 5] FIG. 1 is a diagram illustrating a hardware configuration of a speech activity detection device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0012] 1 is a diagram showing a voice activity detection system 10 including a voice activity detection device 1 according to an embodiment. As shown in FIG. 1, the voice activity detection system 10 includes the voice activity detection device 1, a sound input device 2, and an imaging device 3.

[0013] The voice activity detection device 1 is a device that detects a voice section in which the voice of a target person is present. The target person refers to a person whose voice section is detected by the voice activity detection device 1. The voice refers to a voice uttered by a person. The voice section refers to a section (time period) in which voice is present. The voice activity detection device 1 is used, for example, in a dialogue via multiple terminals connected to each other via a network. The voice activity detection device 1 is used, for example, in a dialogue between multiple people including a target person. The voice activity detection device 1 is applied, for example, to a smartphone, a mobile phone, a tablet terminal, a personal computer, and the like, which are examples of terminals used by multiple people who converse with each other.

[0014] The sound input device 2 acquires sound and outputs a sound signal indicating the acquired sound to the voice activity detection device 1. "Sound" includes various sounds such as voice and non-voice environmental sounds. The sound input device 2 is, for example, a microphone mounted on a terminal of a target person included in the people.

[0015] The imaging device 3 captures a video and outputs a video signal indicating the captured video to the voice activity detection device 1. The "video" includes the target person and the background of the target person. The imaging device 3 is, for example, a camera mounted on the terminal of the target person.

[0016] Next, a functional configuration of the speech section detection device 1 according to the embodiment will be described. The speech section detection device 1 includes, as its functional configuration, a sound signal acquisition unit 11, an audio signal detection unit 12, a video signal acquisition unit 13, a video determination unit 14, a speech determination unit 15, a quality determination unit 16, and a speech section detection unit 17.

[0017] The sound signal acquiring unit 11 acquires a sound signal output from the sound input device 2. The sound signal acquiring unit 11 outputs the acquired sound signal to the voice signal detection unit 12. The sound signal acquiring unit 11 may, for example, convert the acquired sound signal into a digital signal and output the converted digital signal to the voice signal detection unit 12. The sound signal acquiring unit 11 may perform signal processing such as noise canceling on the acquired sound signal before outputting the sound signal to the voice signal detection unit 12. The signal processing may be executed outside the voice activity detection device 1.

[0018] In this embodiment, the sound signal acquisition unit 11 continuously acquires sound signals. "Continuously" means that the sound signals are acquired continuously while a plurality of people are having a conversation. However, the sound signal acquisition unit 11 may not continuously acquire sound signals, but may acquire sound signals intermittently at appropriate timings set in advance.

[0019] The voice signal detection unit 12 detects a voice signal indicating a person's voice in the sound signal based on the sound signal acquired by the sound signal acquisition unit 11. For example, the voice signal detection unit 12 detects the voice signal by determining whether or not the sound indicated by the sound signal acquired by the sound signal acquisition unit 11 is a person's voice. The voice signal detection unit 12 may determine whether or not the sound indicated by the sound signal is a voice by using a known means. The voice signal detection unit 12 outputs the detection result of the voice signal to the voice determination unit 15.

[0020] In this embodiment, the audio signal detection unit 12 continuously detects the audio signal. However, the audio signal detection unit 12 may not continuously detect the audio signal, but may intermittently detect the audio signal at appropriate timing set in advance.

[0021] The video signal acquisition unit 13 acquires a video signal (a signal indicating a video) output from the imaging device 3. The video signal acquisition unit 13 acquires the video signal with the video quality determined by the quality determination unit 16.

[0022] The video signal acquisition unit 13 outputs the acquired video signal to the video determination unit 14. The video signal acquisition unit 13 may, for example, convert the acquired video signal into a digital signal and output the converted digital signal to the video determination unit 14. Before outputting the video signal to the video determination unit 14, the video signal acquisition unit 13 may perform signal processing on the acquired video signal, such as detecting edges of an imaging target. The signal processing may be executed outside the voice activity detection device 1.

[0023] The video judgment unit 14 judges the presence or absence of an event related to the target person in the video based on the video signal acquired by the video signal acquisition unit 13. In this embodiment, the video judgment unit 14 judges the presence or absence of an object related to the target person as the presence or absence of an event. The video judgment unit 14 judges the presence or absence of the face of the target person as the object. For example, the video judgment unit 14 may judge the presence or absence of the face of the target person in the video by judging whether or not the face of the target person is detected from the video shown by the video signal acquired by the video signal acquisition unit 13. The video judgment unit 14 may judge whether or not the face of the target person is detected from the video shown by the video signal using a known means. In this embodiment, the video judgment unit 14 outputs the judgment result of the presence or absence of the face of the target person to the audio judgment unit 15 and the quality determination unit 16.

[0024] Note that the object related to the target person is not limited to the face of the target person, and may be, for example, the ears, eyes, shoulders, etc. of the target person. In addition, the object is not limited to a part of the body of the target person, and may be an object owned by the target person, such as a smartphone.

[0025] In this embodiment, the video determination unit 14 determines the presence or absence of lip movement of the target person as the presence or absence of an event based on the video signal acquired by the video signal acquisition unit 13 and the determination result of the presence or absence of an object. In this embodiment, if the video determination unit 14 determines that the target person's face (target object) is not present, it determines that the target person's lip movement is not present. If the video determination unit 14 determines that the target person's face is present, it detects the target person's lips from the video shown by the video signal. The video determination unit 14 may detect the target person's lips from the video shown by the video signal using known means.

[0026] In this embodiment, the image determination unit 14 determines whether the degree of opening of the detected lips is equal to or greater than a predetermined value. The image determination unit 14 detects the length of the face of the target person (the length from the top of the head to the tip of the chin when viewed from the front of the face) from the image shown by the image signal. The image determination unit 14 also detects the distance from the upper end to the lower end of the lips from the image shown by the image signal. If the ratio of the distance from the upper end to the lower end of the lips to the detected face length is equal to or greater than a predetermined value, the image determination unit 14 determines that the degree of opening of the detected lips is equal to or greater than a predetermined value. In other cases, the image determination unit 14 determines that the degree of opening of the detected lips is not equal to or greater than a predetermined value. The predetermined value is, for example, set in advance and can be changed as appropriate.

[0027] Furthermore, the image judgment unit 14 judges whether the detected movement of the lips is equal to or greater than a predetermined value. When the amount of fluctuation per unit time of the ratio of the distance from the upper end to the lower end of the lips to the detected face length is equal to or greater than a predetermined value, the image judgment unit 14 judges that the detected movement of the lips is equal to or greater than a predetermined value. In other cases, the image judgment unit 14 judges that the detected movement of the lips is not equal to or greater than a predetermined value. The predetermined value is, for example, set in advance and can be changed as appropriate. The image judgment unit 14 may use known means to judge whether the opening and movement of the lips is equal to or greater than a predetermined value.

[0028] When the video judgment unit 14 judges that the target person's lips are moving when the degree of opening of the detected lips is equal to or greater than a predetermined value, or when the detected lip movement is equal to or greater than a predetermined value, the video judgment unit 14 judges that the target person's lips are not moving in any other case. However, the method by which the video judgment unit 14 judges whether the target person's lips are moving is not limited to the above-described method, and can be changed as appropriate. The video judgment unit 14 outputs the judgment result of whether or not there is lip movement to the audio judgment unit 15.

[0029] The voice determination unit 15 determines whether or not there is a voice of the target person based on the detection result of the voice signal detection unit 12 and the determination result of the video determination unit 14. In this embodiment, the voice determination unit 15 determines whether or not there is a voice of the target person based on the detection result of the voice signal detection unit 12 as well as the determination result of the video determination unit 14 on the presence or absence of lip movement. The voice determination unit 15 determines that there is a voice of the target person when a voice signal is detected by the voice signal detection unit 12 and the video determination unit 14 determines that there is lip movement. The voice determination unit 15 determines that there is no voice of the target person when no voice signal is detected by the voice signal detection unit 12 or when the video determination unit 14 determines that there is no lip movement. The voice determination unit 15 outputs the determination result of the presence or absence of the voice of the target person to the quality determination unit 16 and the voice section detection unit 17.

[0030] The quality determination unit 16 executes a video quality determination process. In the video quality determination process, the quality determination unit 16 determines the quality of the video represented by the video signal acquired by the video signal acquisition unit 13 based on the determination results of the video determination unit 14 and the audio determination unit 15. "Quality" includes, for example, at least one of the frame rate, resolution, and angle of view of the video. In this embodiment, the quality of the video is the resolution of the video. The quality determination unit 16 outputs the determined video quality to the video signal acquisition unit 13.

[0031] A specific example of the video quality determination process executed by the quality determination unit 16 will be described below. The quality determination unit 16 acquires the determination results of the video determination unit 14 and the audio determination unit 15. When the video determination unit 14 determines that the face of the target person is present and the audio determination unit 15 determines that the audio of the target person is present, the quality determination unit 16 determines the quality of the video to be a reference quality, which is a predetermined quality. More specifically, the quality determination unit 16 determines the resolution of the video to be a first value, which is a predetermined value.

[0032] When the video determination unit 14 determines that the face of a target person is present and the audio determination unit 15 determines that the audio of the target person is present, it can be said that there is a high possibility that the target person is present in front of the terminal (for example, is present at a desk) and is making audio (speaking). In this case, since there is no need to detect the timing at which the target person starts making audio, audio segments can be detected appropriately without requiring high video quality.

[0033] When the video determination unit 14 determines that the target person's face is not present, the quality determination unit 16 determines the video quality to be low quality that is lower than the reference quality. More specifically, the quality determination unit 16 determines the video resolution to be a second value that is lower than the first value.

[0034] If the video determination unit 14 determines that the target person's face is not present, it is highly likely that the target person is not in front of the terminal (for example, has left the desk). In this case, since there is a high possibility that the target person's voice will not be detected, the voice section can be detected appropriately even if the video quality is low.

[0035] When the video determination unit 14 determines that the face of the target person is present and the audio determination unit 15 determines that the audio of the target person is absent, the quality determination unit 16 determines the quality of the video to be high quality higher than the reference quality. More specifically, the quality determination unit 16 determines the resolution of the video to be a third value that is higher than the first value.

[0036] When the image determination unit 14 determines that the face of the target person is present and the audio determination unit 15 determines that the target person's audio is not present, it is highly likely that the target person is in front of the terminal and is not making any audio. In this case, by increasing the quality of the image, it is possible to detect with high accuracy the timing at which the target person starts making audio. The above first value, second value, and third value are, for example, preset and can be changed as appropriate.

[0037] The voice section detection unit 17 detects a voice section of the target person from the voice signal detected by the voice signal detection unit 12 based on the determination result of the voice determination unit 15. In this embodiment, the voice section detection unit 17 detects, as a voice section, a section in the voice signal that is determined by the voice determination unit 15 to contain the voice of the target person.

[0038] In this embodiment, the voice activity detection unit 17 outputs the detected voice activity. As the output, for example, the voice activity detection unit 17 may transmit the detected voice activity to an external device of the voice activity detection device 1, or may display it on a display device included in the voice activity detection device 1. The external device may be, for example, a smartphone or a tablet terminal, or may be a display of a personal computer. The external device is configured to be able to communicate with the voice activity detection device 1, for example.

[0039] Next, a voice activity detection method according to this embodiment (including an operation method of the voice activity detection device 1) will be described. Fig. 2 is a flowchart showing the voice activity detection method according to this embodiment. First, a sound generated in an environment in which a target person is present is acquired by the sound input device 2. Then, a video of the target person is acquired by imaging the image of the target person by the imaging device 3.

[0040] Next, the sound signal output from the sound input device 2 is acquired by the sound signal acquisition unit 11 (sound signal acquisition step, step S1). In step S1, the acquired sound signal is output to the voice signal detection unit 12. In step S1, for example, before the acquired sound signal is output to the voice signal detection unit 12, signal processing may be performed on the sound signal.

[0041] Next, the voice signal detection unit 12 detects a voice signal indicating a person's voice in the sound signal based on the sound signal acquired in step S1 (voice signal detection step, step S2). In step S2, it is determined whether the sound indicated by the sound signal acquired in step S1 is a person's voice, and the voice signal is detected.

[0042] Next, the video signal acquiring unit 13 acquires the video signal output from the imaging device 3 (video signal acquiring step, step S3). In step S3, the video signal acquiring unit 13 acquires a video signal with the video quality determined in step S7 described below. In step S3, a video signal is acquired based on the video quality acquired by the video signal acquiring unit 13. In step S3, the video signal acquiring unit 13 acquires a video signal indicating the video of the acquired quality.

[0043] In step S3, for example, the acquired video signal is output to the video determination unit 14. In step S3, for example, before the acquired video signal is output to the video determination unit 14, signal processing may be performed on the video signal.

[0044] Next, the video determination unit 14 determines whether or not an event related to the target person has occurred based on the video signal acquired in step S3 (video determination step, step S4). In this embodiment, in step S4, the video determination unit 14 determines whether or not an object related to the target person has occurred as the presence or absence of an event. In step S4, the video determination unit 14 determines whether or not the target person's face is present as an object. In step S4, for example, it may be determined whether or not the target person's face has been detected from the video represented by the video signal acquired in step S3, and the presence or absence of the target person's face in the video may be determined.

[0045] Next, the video determination unit 14 determines whether or not there is an event, based on the video signal acquired in step S3 and the determination result of step S4, whether or not there is lip movement of the target person (video determination step, step S5). In step S5, if it is determined in step S4 that the target person does not have a face, the video determination unit 14 determines that there is no lip movement of the target person. If it is determined in step S4 that the target person has a face, the video determination unit 14 detects the target person's lips from the video shown by the video signal. In this case, it is determined whether or not the lip opening degree detected by the video determination unit 14 is equal to or greater than a predetermined degree. In addition, it is determined whether or not the lip movement detected by the video determination unit 14 is equal to or greater than a predetermined degree. If the video determination unit 14 determines that the lip opening degree is equal to or greater than a predetermined degree and that the lip movement is equal to or greater than a predetermined degree, it is determined that there is lip movement of the target person. In other cases, the video determination unit 14 determines that there is no lip movement of the target person.

[0046] Next, the voice determination unit 15 determines whether or not there is a voice of the target person based on the detection result of step S2 and the determination result of step S5 (voice determination step, step S6). In this embodiment, in step S6, the presence or absence of the voice of the target person is determined by further considering the determination result of the presence or absence of lip movement in step S5 in addition to the detection result of step S2. In step S6, if the voice determination unit 15 detects a voice signal in step S2 and determines that there is lip movement in step S5, it determines that there is a voice of the target person. In step S6, if the voice signal is not detected in step S2 or if it is determined that there is no lip movement in step S5, it determines that there is no voice of the target person.

[0047] Next, the quality determination unit 16 executes a video quality determination process (video quality determination step, step S7). In the video quality determination process, the quality determination unit 16 determines the quality of the video represented by the video signal acquired by the video signal acquisition unit 13 in step S3 based on the determination results of steps S4 and S6. In step S7, the video quality determined by the quality determination unit 16 is output to the video signal acquisition unit 13. The video quality determination process in step S7 will be described later.

[0048] Next, based on the determination result of step S6, the voice section detection unit 17 detects the voice section of the target person from the voice signal detected in step S2 (voice section detection step, step S8). In step S8, the voice section detection unit 17 detects, as a voice section, the section in the voice signal determined in step S6 to contain the voice of the target person. Next, in this embodiment, the voice section detection unit 17 outputs the voice section detected in step S8.

[0049] Next, an example of the video quality determination process in the above step S7 will be described. Fig. 3 is a flowchart showing the video quality determination process shown in Fig. 2. In the video quality determination process, first, the quality determination unit 16 acquires the determination result of the video determination unit 14 in step S4 (step S71). If the video determination unit 14 determines in step S4 that the target person's face is not present (step S71: NO), the quality determination unit 16 determines the video quality to be low (step S72). If the video determination unit 14 determines in step S4 that the target person's face is present (step S71: YES), the quality determination unit 16 executes step S73.

[0050] In step S73, the quality determination unit 16 acquires the determination result of the voice determination unit 15 in step S6. If the voice determination unit 15 determines in step S6 that there is no voice of the target person (step S73: NO), the quality determination unit 16 determines the quality of the video to be high (step S74). If the voice determination unit 15 determines in step S6 that there is a voice of the target person (step S73: YES), the quality determination unit 16 determines the quality of the video to be standard quality (step S75).

[0051] Next, the effects of the voice activity detection device 1 and the voice activity detection method according to this embodiment will be described.

[0052] In the voice activity detection device 1 and the voice activity detection method according to the present embodiment, the quality of the video shown by the acquired video signal is determined based on the determination result of the presence or absence of an event related to the target person and the presence or absence of the target person's voice. For example, the presence or absence of an object related to the target person, such as the face of the target person, is determined as the event related to the target person. For example, if it is determined that there is an object related to the target person and that there is the target person's voice, it is highly likely that the target person is present and is speaking. In this case, since it is not necessary to detect the timing when the target person starts to emit a voice, the voice activity can be appropriately detected even if the quality of the video is not high. Also, for example, if it is determined that there is no object, it is highly likely that the target person has left the seat and the target person's voice will not be detected, so the voice activity can be appropriately detected even if the quality of the video is low. In this way, the quality of the video shown by the acquired video signal can be reduced according to the situation of the target person, so the amount of calculation in detecting the voice activity can be reduced.

[0053] In a section (voice section) where it is determined that the voice of a target person is present, power is required to transmit a voice signal corresponding to the voice section to, for example, an external device of the voice section detection device 1. For example, when the voice section detection device 1 is applied to a terminal with a small battery capacity such as a smartphone, the increased power consumption may cause problems. In this regard, the voice section detection device 1 according to this embodiment can reduce the amount of calculation in detecting a voice section, thereby suppressing the power consumption of the terminal to which the voice section detection device 1 is applied.

[0054] In the above-mentioned voice section detection device 1, the video determination unit 14 determines the presence or absence of an object related to a target person as the presence or absence of an event. When the video determination unit 14 determines that a face, which is a target, is present and the audio determination unit 15 determines that audio is present, the quality of the video is determined to be a predetermined quality, which is a reference quality. When the video determination unit 14 determines that there is no face, the quality determination unit 16 determines that the quality of the video is low, which is a quality lower than the reference quality. When it is determined that there is no face of the target person, for example, it can be said that there is a high possibility that the target person has left his / her seat and that there is no audio section of the target person. Therefore, even if the quality of the video shown by the acquired video signal is determined to be low quality, the voice section can be appropriately detected. As described above, the amount of calculation in detecting the voice section can be reduced.

[0055] In the voice section detection device 1 described above, when the video determination unit 14 determines that a face is present and the audio determination unit 15 determines that no audio is present, the quality determination unit 16 determines the quality of the video to be high quality, which is higher than the reference quality. When it is determined that the face of the target person is present and that the audio of the target person is absent, it can be said that, for example, the target person is highly likely to be present and not speaking. At this time, by determining the quality of the video represented by the acquired video signal to be high quality, the timing at which the target person begins to make audio can be detected with high accuracy. As described above, the voice section can be detected with high accuracy.

[0056] In the above-mentioned voice section detection device 1, the video determination unit 14 determines the presence or absence of lip movement of the target person as an event based on the video signal and the determination result of the presence or absence of the target object. The audio determination unit 15 determines the presence or absence of the target person's voice based on the determination result of the presence or absence of lip movement by the video determination unit 14. This makes it possible to determine the presence or absence of the target person's voice by taking into consideration whether or not there is lip movement of the target person in addition to whether or not an audio signal indicating the person's voice has been detected. As a result, the voice section can be detected with even higher accuracy.

[0057] The present disclosure has been described in detail above based on the embodiments. However, the present disclosure is not limited to the above embodiments. The present disclosure can be modified in various ways without departing from the spirit and scope of the present disclosure.

[0058] In the above embodiment, an example has been described in which the voice determination unit 15 determines the presence or absence of the voice of the target person based on the detection result of the voice signal detection unit 12 and the determination result of the presence or absence of lip movement of the target person by the video determination unit 14. However, the voice determination unit 15 may determine the presence or absence of the voice of the target person based on the detection result of the voice signal detection unit 12 and the determination result of the presence or absence of an event by the video determination unit 14, without using the determination result of the presence or absence of lip movement. In this case, the video determination unit 14 does not need to determine the presence or absence of lip movement of the target person.

[0059] As an example, the voice determination unit 15 may determine the presence or absence of the voice of the target person based on the detection result of the voice signal detection unit 12 and the determination result of the presence or absence of the target object by the video determination unit 14. In this case, the voice determination unit 15 may determine that the voice of the target person is present when a voice signal is detected by the voice signal detection unit 12 and the video determination unit 14 determines that there is an target object. In other cases, the voice determination unit 15 may determine that there is no voice of the target person.

[0060] In the above embodiment, an example has been described in which the quality determination unit 16 determines the quality of the video represented by the video signal acquired by the video signal acquisition unit 13 based on the determination results of the video determination unit 14 and the audio determination unit 15. However, the quality determination unit 16 may determine the quality of the video by further taking into consideration the detection result of a new audio signal by the audio signal detection unit 12 in addition to the determination results of the video determination unit 14 and the audio determination unit 15.

[0061] In this modification, the sound signal acquisition unit 11 continuously acquires a sound signal. The audio signal detection unit 12 continuously detects an audio signal based on the sound signal continuously acquired by the sound signal acquisition unit 11. In addition, in this modification, the quality determination unit 16 executes a video quality determination process according to the modification. In the video quality determination process according to the modification, the quality determination unit 16 determines whether or not a new audio signal has been detected by the audio signal detection unit 12 when the video determination unit 14 determines that the face of the target person is present and the audio determination unit 15 determines that the audio of the target person is present. The "new audio signal" refers to an audio signal newly detected by the audio signal detection unit 12 within a predetermined time after the audio determination unit 15 makes a determination. The predetermined time is, for example, set in advance and can be changed as appropriate.

[0062] When the video judgment unit 14 judges that the face of the target person is present and the audio judgment unit 15 judges that the audio of the target person is present, if a new audio signal is detected by the audio signal detection unit 12, the quality determination unit 16 determines the quality of the video to be a predetermined quality, which is a reference quality, and if no new audio signal is detected by the audio signal detection unit 12, the quality determination unit 16 determines the quality of the video to be a high quality, which is higher than the reference quality.

[0063] Next, a speech activity detection method according to the modified example (including an operation method of the speech activity detection device according to the modified example) will be described. In the speech activity detection method according to the modified example, in step S1, a sound signal is continuously acquired by the sound signal acquisition unit 11. In step S2, a speech signal is continuously detected by the speech signal detection unit 12 based on the sound signal continuously acquired by the sound signal acquisition unit 11 in step S1.

[0064] Fig. 4 is a flowchart showing a video quality determination process according to the modified example. In the video quality determination process according to the modified example, the quality determination unit 16 executes steps S81 to S87. The processes of steps S81 to S84 shown in Fig. 4 are the same as the processes of steps S71 to S74 shown in Fig. 3, and therefore will not be described.

[0065] If the voice determination unit 15 determines in step S6 that there is a voice of the target person (step S83: YES), the quality determination unit 16 determines whether or not a new voice signal has been detected by the voice signal detection unit 12 in step S2 (step S85). If the voice signal detection unit 12 determines in step S2 that a new voice signal has not been detected (step S85: NO), the quality determination unit 16 determines the quality of the video to be high (step S86). If the voice signal detection unit 12 determines in step S2 that a new voice signal has been detected (step S85: YES), the quality determination unit 16 determines the quality of the video to be standard quality (step S87).

[0066] In this modified example, the sound signal acquisition unit 11 continuously acquires sound signals. The audio signal detection unit 12 continuously detects audio signals based on the continuously acquired sound signals. When the video determination unit 14 determines that the face of the target person is present and the audio determination unit 15 determines that the audio of the target person is present, if the audio signal detection unit 12 detects a new audio signal, the quality determination unit 16 determines the quality of the video to be a reference quality, which is a predetermined quality, and when the audio signal detection unit 12 does not detect a new audio signal, the quality determination unit 16 determines the quality of the video to be a high quality, which is a quality higher than the reference quality.

[0067] In this case, the quality of the video is determined by taking into consideration the result of judging whether the target person has a face and whether the target person has a voice, as well as the result of judging whether the target person has a voice. If it is judged that the target person has a face and a voice, and no new voice signal is detected, it is highly likely that the target person has stopped speaking and is not making a sound with his / her mouth open. In this case, by determining the quality of the video represented by the acquired video signal to be high quality, the timing when the target person starts making a sound again can be detected with high accuracy. As a result, the voice section can be detected with even higher accuracy.

[0068] The quality determination unit 16 may determine the quality of the video based on the computational resources of the terminal of the target person. The computational resources of the terminal include, for example, CPU (Central Processing Unit) usage, memory usage, and power consumption. In this case, the quality determination unit 16 may determine the quality of the video to be low, for example, when the CPU usage of the terminal is equal to or higher than a predetermined value. The predetermined value is, for example, set in advance and can be changed as appropriate. Even in this case, the amount of calculation required for detecting a voice segment can be reduced. Furthermore, when the voice segment detection device 1 is applied to a terminal with small computational resources, or when the computational resources of the terminal are temporarily reduced, the voice segment can be stably detected.

[0069] In the above embodiment, an example has been described in which the voice activity detection device 1 is applied to a terminal used by multiple people who converse with each other. However, all or part of the processing executed by the voice activity detection device 1 may be executed in an external device (for example, a server) connected to the terminal via a network.

[0070] In this modification, as an example, the speech section detection device 1 further includes a sound processing unit and a video processing unit. The processes executed by the sound signal acquisition unit 11 and the video signal acquisition unit 13 are executed in the above-mentioned terminal. Moreover, the processes executed by the functional units of the speech section detection device 1 other than the sound signal acquisition unit 11 and the video signal acquisition unit 13 are executed in the above-mentioned external device.

[0071] The sound signal acquisition unit 11 converts the acquired sound signal into a digital signal and outputs the converted digital signal to the sound processing unit. The sound processing unit acquires the digital signal output from the sound signal acquisition unit 11, for example, and outputs the acquired digital signal to the audio signal detection unit 12.

[0072] The video signal acquisition unit 13 converts the acquired video signal into a digital signal and outputs the converted digital signal to the video processing unit. The video processing unit acquires the digital signal output from the video signal acquisition unit 13. The video processing unit performs digital signal processing to adjust video quality related to audio detection, for example. The video processing unit performs digital signal processing on the acquired digital signal, for example, to acquire a digital signal of a quality corresponding to the video quality determined by the quality determination unit 16. As an example, the video processing unit executes processing to adjust the frame rate (quality) of the received video signal stream. The video processing unit outputs the quality-adjusted digital signal to the video determination unit 14.

[0073] In the above modified example, the voice activity detection device 1 includes a sound processing unit. However, the voice activity detection device 1 does not necessarily have to include a sound processing unit.

[0074] The voice activity detection device 1 may have a multimodal speech detection function. For example, the voice activity detection device 1 may switch the sound input device 2 of the target person's terminal on or off depending on the detected voice activity. More specifically, when the voice activity detection device 1 detects the voice activity of the target person, it may switch the sound input device 2 of the target person's terminal on, and when it does not detect the voice activity of the target person, it may switch the sound input device 2 of the target person's terminal off. In this case, the target person is spared the trouble of directly operating the sound input device 2. In addition, noise other than when the target person is speaking is removed, making the dialogue smoother.

[0075] The voice section detection device 1 may display the status of the target person on the terminal of the person who will have a conversation with the target person, for example, based on the judgment result of the video judgment unit 14. The "status" includes, for example, a state in which the target person is present and can have a conversation (for example, a casual chat), and a state in which the target person is away from his / her desk and cannot have a conversation. In this case, when the video judgment unit 14 judges that the face of the target person is present, the voice section detection device 1 may display, on the terminal of the person who will have a conversation with the target person, the status of the target person that the target person is present and can have a conversation. Also, when the video judgment unit 14 judges that the face of the target person is not present, the voice section detection device 1 may display, on the terminal of the person who will have a conversation with the target person, the status of the target person that the target person is away from his / her desk and cannot have a conversation. In this way, multiple people who are having a conversation with each other can recognize each other as people who can have a conversation.

[0076] The voice activity detection device 1 may, for example, display on the terminal the number of people participating in the conversation. The voice activity detection device 1 may also, for example, display on the terminal the remaining time available for conversation. In this case, the voice activity detection device 1 may display on the terminal the remaining time available for conversation based on, for example, a person's previously set schedule. More specifically, the voice activity detection device 1 may display on the terminal, for example, the time from the current time to the time of the person's next schedule as the remaining time available for conversation. In this case, an atmosphere in which people can easily converse can be created.

[0077] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

[0078] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.

[0079] For example, the voice activity detection device 1 according to an embodiment of the present disclosure may function as a computer that performs the information processing of the present disclosure. Fig. 5 is a diagram showing an example of a hardware configuration of the voice activity detection device 1 according to an embodiment of the present disclosure. The above-mentioned voice activity detection device 1 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. The hardware configuration of the terminal 20 may also be as described here.

[0080] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the speech activity detection apparatus 1 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.

[0081] Each function in the voice activity detection device 1 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communication by a communication device 1004, and control at least one of reading and writing of data in the memory 1002 and the storage 1003.

[0082] The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each function in the above-mentioned voice activity detection device 1 may be realized by the processor 1001.

[0083] Furthermore, the processor 1001 reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. As the programs, programs that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments are used. For example, each function in the voice activity detection device 1 may be realized by a control program stored in the memory 1002 and operated in the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

[0084] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to perform information processing according to an embodiment of the present disclosure.

[0085] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may be called an auxiliary storage device. The storage medium provided in voice activity detection device 1 may be, for example, a database including at least one of memory 1002 and storage 1003, a server, or other appropriate medium.

[0086] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0087] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0088] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0089] Furthermore, the voice activity detection device 1 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0090] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0091] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0092] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0093] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0094] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0095] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0096] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0097] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0098] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0099] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0100] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0101] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0102] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0103] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0104] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0105] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0106] Finally, various exemplary embodiments included in the present disclosure are described in [E1] to [E6] below.

[0107] [E1] A voice activity detection device for detecting a voice activity in which a voice of a target person is present, a sound signal acquisition unit that acquires a sound signal indicating a sound; a voice signal detection unit that detects a voice signal indicating a person's voice in the sound signal based on the sound signal; a video signal acquisition unit that acquires a video signal representing a video; an image determination unit that determines whether or not an event related to the target person exists in the image based on the image signal; a voice determination unit that determines the presence or absence of the voice of the target person based on a detection result of the voice signal detection unit and a determination result of the video determination unit; a quality determination unit that determines a quality of the video represented by the video signal acquired by the video signal acquisition unit based on the determination results of the video determination unit and the audio determination unit; a voice activity detection unit that detects the voice activity of the target person from the voice signal based on a determination result of the voice determination unit. A voice activity detection device.

[0108] [E2] The image determination unit determines the presence or absence of an object related to the target person as the presence or absence of the event, The quality determination unit When the image determination unit determines that the object is present and the audio determination unit determines that the audio of the object is present, the quality of the image is determined to be a reference quality, which is a predetermined quality; When the image determination unit determines that the object is not present, the image quality is determined to be low quality that is lower than the reference quality. The voice activity detection device according to [E1].

[0109] [E3] the quality determination unit determines a quality of the video to be a high quality that is higher than the reference quality when the video determination unit determines that the object is present and the audio determination unit determines that the audio of the object is absent. The voice activity detection device according to [E2].

[0110] [E4] The sound signal acquisition unit continuously acquires the sound signal, The audio signal detection unit continuously detects the audio signal based on the sound signal continuously acquired, The quality determination unit When the video determination unit determines that the object is present and the audio determination unit determines that the audio of the object is present, When a new audio signal is detected by the audio signal detection unit, the quality of the video is determined to be a reference quality, which is a predetermined quality; When the audio signal detection unit does not detect a new audio signal, the quality of the video is determined to be a high quality that is higher than the reference quality. The speech activity detection device according to [E2] or [E3]. [E5] the video determination unit determines the presence or absence of lip movement of the target person as the presence or absence of the event based on the video signal and a result of the determination of the presence or absence of the target object; the audio determination unit determines the presence or absence of the audio of the target person based on a result of the determination by the video determination unit of the presence or absence of the lip movement. The voice activity detection device according to any one of [E2] to [E4]. [E6] A voice activity detection method for detecting a voice activity in which a voice of a target person is present, comprising the steps of: a sound signal acquiring step of acquiring a sound signal indicative of a sound; a voice signal detection step of detecting a voice signal indicating a person's voice in the sound signal based on the sound signal; a video signal acquisition step of acquiring a video signal representing a video; an image determination step of determining whether or not an event related to the target person exists in the image based on the image signal; a voice determination step of determining whether or not there is a voice of the target person based on a detection result of the voice signal detection step and a determination result of the video determination step; a quality determination step of determining a quality of the image represented by the image signal acquired by the image signal acquisition step based on the determination results of the image determination step and the audio determination step; a voice activity detection step of detecting the voice activity of the target person from the voice signal based on a result of the voice determination step. Voice activity detection method. [Explanation of symbols]

[0111] 1...voice activity detection device, 11...sound signal acquisition unit, 12...voice signal detection unit, 13...video signal acquisition unit, 14...video judgment unit, 15...voice judgment unit, 16...quality determination unit, 17...voice activity detection unit.

Claims

1. a sound signal acquisition unit that acquires a sound signal indicating a sound; a voice signal detection unit that detects a voice signal indicating a person's voice in the sound signal based on the sound signal; a video signal acquisition unit that acquires a video signal representing a video; an image determination unit that determines whether or not an event related to a target person exists in the image based on the image signal; a voice determination unit that determines the presence or absence of the voice of the target person based on a detection result of the voice signal detection unit and a determination result of the video determination unit; a quality determination unit that determines a quality of the video represented by the video signal acquired by the video signal acquisition unit based on the determination results of the video determination unit and the audio determination unit; a voice activity detection unit that detects a voice activity of the target person from the voice signal based on a determination result of the voice determination unit. A voice activity detection device.

2. The image determination unit determines the presence or absence of an object related to the target person as the presence or absence of the event, The quality determination unit When the image determination unit determines that the object is present and the audio determination unit determines that the audio of the object is present, the quality of the image is determined to be a reference quality, which is a predetermined quality; When the image determination unit determines that the object is not present, the image quality is determined to be low quality that is lower than the reference quality. The voice activity detection device according to claim 1 .

3. the quality determination unit determines a quality of the video to be a high quality that is higher than the reference quality when the video determination unit determines that the object is present and the audio determination unit determines that the audio of the object is absent. The voice activity detection device according to claim 2 .

4. The sound signal acquisition unit continuously acquires the sound signal, The audio signal detection unit continuously detects the audio signal based on the sound signal continuously acquired, The quality determination unit When the video determination unit determines that the object is present and the audio determination unit determines that the audio of the object is present, When a new audio signal is detected by the audio signal detection unit, the quality of the video is determined to be a reference quality, which is a predetermined quality; When the audio signal detection unit does not detect a new audio signal, the quality of the video is determined to be a high quality that is higher than the reference quality. The voice activity detection device according to claim 2 .

5. the video determination unit determines the presence or absence of lip movement of the target person as the presence or absence of the event based on the video signal and a result of the determination of the presence or absence of the target object; the audio determination unit determines the presence or absence of the audio of the target person based on a result of the determination by the video determination unit of the presence or absence of the lip movement. The voice activity detection device according to claim 2 .

6. a sound signal acquiring step of acquiring a sound signal indicative of a sound; a voice signal detection step of detecting a voice signal indicating a person's voice in the sound signal based on the sound signal; a video signal acquisition step of acquiring a video signal representing a video; a video determination step of determining whether or not an event related to a target person exists in the video based on the video signal; a voice determination step of determining whether or not there is a voice of the target person based on a detection result of the voice signal detection step and a determination result of the video determination step; a quality determination step of determining a quality of the image represented by the image signal acquired by the image signal acquisition step based on the determination results of the image determination step and the audio determination step; a voice activity detection step of detecting a voice activity of the target person from the voice signal based on a result of the voice determination step. Voice activity detection method.

Images