Voice activity detection device, learning device, and voice activity detection program

Abstract

To provide a sound section detection device capable of reducing a deterioration in detection accuracy by an abnormal image input, a learning device and a sound section detection program.SOLUTION: An acquisition unit acquires an acoustic signal and a non-acoustic signal related to the same sound generation source. An acoustic feature calculation unit calculates an acoustic feature on the basis of the acoustic signal. A non-acoustic feature calculation unit calculates a non-acoustic feature on the basis of the non-acoustic signal. A confidence weight calculation unit calculates confidence weight on the basis of difference between the acoustic signal and the non-acoustic signal. An integrated feature calculation unit calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the confidence weight. A sound existence probability calculation unit calculates a sound existence probability on the basis of the integrated feature. A detection unit detects a sound section being a time section in which sound is generated and / or a non-sound section being a time section in which sound is not generated on the basis of comparison to a threshold of the sound existence probability.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a voice activity detection device, a learning device, and a voice activity detection program. [Background technology]

[0002] Voice Activity Detection (VAD) is a technology that detects voice activity that includes the user's speech from an input signal. VAD is mainly used to improve the recognition accuracy of voice recognition, and in the field of voice coding, it is used to assist data compression in non-voice segments.

[0003] Voice activity detection requires a process of detecting a voice activity containing a predetermined voice from a time period of an input signal. For example, a pre-trained model is used to detect a predetermined voice activity from an input acoustic signal, based on whether a frame to be processed is a voice activity containing a voice such as an utterance.

[0004] In order to improve the accuracy of voice activity detection in a difficult-to-process environment such as a noisy environment, a voice activity detection device has been proposed that uses both an audio signal and a non-audio signal such as a lip video signal as input signals to detect a voice activity. For example, the technology disclosed in Patent Document 1 uses features of audio information extracted from an audio sound collection device and visual features related to the horizontal and vertical lengths of the lips in the lip area extracted from a video imaging unit to calculate the probability of speech based on a Bayesian network and detect the voice activity. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-191423 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology in Patent Document 1 is premised on a normal lip video signal as the input signal, and therefore there is a problem that the accuracy of voice activity detection can be significantly reduced by abnormal video input in actual processing due to the speaker wearing a mask, facing sideways, failure to estimate the mouth area, video loading errors, etc.

[0007] Specifically, the technology disclosed in Patent Document 1 calculates the conditional probability of an acoustic feature and a visual feature for the presence or absence of a voice, so that the detection accuracy decreases due to an abnormal visual feature calculated from an abnormal image input.

[0008] An object of the present invention is to provide a voice activity detection device, a learning device, and a voice activity detection program that are capable of reducing a decrease in detection accuracy caused by an abnormal image input. [Means for solving the problem]

[0009] A voice activity detection device according to an embodiment includes an acquisition unit, an acoustic feature calculation unit, a non-acoustic feature calculation unit, a confidence weight calculation unit, an integrated feature calculation unit, a voice presence probability calculation unit, and a detection unit. The acquisition unit acquires an acoustic signal and a non-acoustic signal related to the same sound source. The acoustic feature calculation unit calculates an acoustic feature based on the acoustic signal. The non-acoustic feature calculation unit calculates a non-acoustic feature based on the non-acoustic signal. The confidence weight calculation unit calculates the confidence weight of the acoustic signal. Features and the non-acoustic Features and calculates a confidence weight based on a difference between the confidence weight and the non-acoustic feature. An integrated feature calculation unit calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the confidence weight. A voice presence probability calculation unit calculates a voice presence probability based on the integrated feature. A detection unit detects a voice interval, which is a time interval during which voice is being emitted, and / or a non-voice interval, which is a time interval during which voice is not being emitted, based on a comparison of the voice presence probability with a threshold. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 shows an example of the configuration of a voice activity detection device. [Diagram 2]FIG. 13 is a diagram showing an example of the flow of a voice activity detection process; [Diagram 3] FIG. 2 is a diagram showing a schematic diagram of a voice activity detection process; [Figure 4] FIG. 1 shows an example of an input signal (video signal), an audio signal, and an image signal. [Diagram 5] FIG. 1 shows an example of the configuration of a learning device. [Figure 6] FIG. 1 is a diagram showing an example of a flow of a learning process; [Figure 7] A diagram showing an example of the configuration of an integrated neural network [Figure 8] FIG. 1 is a diagram showing an example of an image signal (image data) according to an embodiment; [Figure 9] Table showing the results of validation under matched noise conditions using the CHiME4 dataset. [Figure 10] Table showing the results of the experiment under mismatched noise conditions using the East Anglia dataset. [Figure 11] Average ROC curves for the audio-only model and the baseline model under matched noise conditions. [Figure 12] FIG. 1 is a diagram showing an example of the configuration of a voice activity detection device according to an application example; [Figure 13] FIG. 13 is a diagram showing an example of the flow of a voice activity detection process according to an application example. [Figure 14] FIG. 10 is a diagram showing a learning process according to an application example; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a speech activity detection device, a learning device, and a speech activity detection program according to the present embodiment will be described with reference to the drawings.

[0012] (Voice Activity Detection Device) Fig. 1 is a diagram showing an example of the configuration of a voice activity detection device 100. The voice activity detection device 100 is a computer that detects the voice activity of an input signal. As shown in Fig. 1, the voice activity detection device 100 includes a processing circuit 11, a storage device 12, an input device 13, a communication device 14, a display device 15, and an audio device 16.

[0013] The processing circuit 11 has a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). The processing circuit 11 executes a voice activity detection program stored in the storage device 12 to perform a voice activity detection process for detecting a voice activity of an input signal. The voice activity detection program is recorded in a non-transitory computer-readable recording medium. The processing circuit 11 reads out the voice activity detection program from the recording medium and executes it to realize an input signal acquisition unit 111, an acoustic feature calculation unit 112, a non-acoustic feature calculation unit 113, a confidence weight calculation unit 114, an integrated feature calculation unit 115, a voice presence probability calculation unit 116, a voice activity detection unit 117, and an output control unit 118. The voice activity detection program may have multiple modules in which the functions of the units 111 to 118 are divided and implemented.

[0014] The hardware implementation of the processing circuit 11 is not limited to the above-mentioned embodiment. For example, the processing circuit 11 may be configured by a circuit such as an application specific integrated circuit (ASIC) that realizes the input signal acquisition unit 111, the acoustic feature calculation unit 112, the non-acoustic feature calculation unit 113, the confidence weight calculation unit 114, the integrated feature calculation unit 115, the voice presence probability calculation unit 116, the voice activity detection unit 117, and / or the output control unit 118. The input signal acquisition unit 111, the acoustic feature calculation unit 112, the non-acoustic feature calculation unit 113, the confidence weight calculation unit 114, the integrated feature calculation unit 115, the voice presence probability calculation unit 116, the voice activity detection unit 117, and / or the output control unit 118 may be implemented in a single integrated circuit or individually in multiple integrated circuits.

[0015] The input signal acquisition unit 111 acquires an acoustic signal and a non-acoustic signal related to the same voice source. The acoustic signal and the non-acoustic signal are time-series signals and are synchronized in time on a frame-by-frame basis. The acoustic signal is a signal related to the voice of a speaker who is the voice source. The non-acoustic signal is a signal other than the acoustic signal related to the speaker that is collected approximately simultaneously with the acoustic signal. For example, the non-acoustic signal is an image signal related to the speaker who is speaking, or a sensor signal related to physiological responses of the speaker's lips and facial muscles due to speech, brain waves, etc.

[0016] The acoustic feature calculation unit 112 calculates a feature amount of the acoustic signal (hereinafter referred to as an acoustic feature). The acoustic feature has a value based on the acoustic signal and has a value correlated with the speech of the speaker. The acoustic feature is calculated for each frame. As an example, the acoustic feature is calculated using a first trained model. The first trained model is a neural network trained to input an acoustic signal and output the acoustic feature. The first trained model is stored in the storage device 12 or the like.

[0017] The non-acoustic feature calculation unit 113 calculates a feature amount of the non-acoustic signal (hereinafter referred to as a non-acoustic feature). The non-acoustic feature has a value based on the non-acoustic signal and has a characteristic value correlating with the speech of the speaker. The non-acoustic feature is calculated for each frame. As an example, the non-acoustic feature is calculated using a second trained model. The second trained model is a neural network trained to input a non-acoustic signal and output a non-acoustic feature. The second trained model is stored in the storage device 12 or the like.

[0018] Note that the first learned model and the second learned model are neural networks trained to penalize differences between acoustic and non-acoustic features for the same speech source.

[0019] The reliability weight calculation unit 114 calculates the reliability weight based on the difference between the acoustic feature and the non-acoustic feature. The reliability weight is calculated for each frame as a judgment value for judging that an abnormal non-acoustic signal has been input. As an example, the reliability weight has a first reliability weight calculated as a difference between the acoustic feature and the non-acoustic feature, a value between a median value and an upper limit value, and a second reliability weight calculated as a value obtained by subtracting the first reliability weight from the upper limit value. For example, when the first reliability weight and the second reliability weight are in a numerical range from 0 to 1, the median value may be set to 0.5 and the upper limit value may be set to 1.

[0020] The integrated feature calculation unit 115 calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight calculated by the reliability weight calculation unit 114. The integrated feature is calculated for each frame. As an example, an integrated feature of the acoustic feature corrected by the first reliability weight and the non-acoustic feature corrected by the second reliability weight is calculated.

[0021] The voice presence probability calculation unit 116 calculates the voice presence probability based on the integrated features. The voice presence probability is used as a measure for distinguishing between a voice section and a non-voice section. The voice section is a time section of the input signal in which voice is being emitted, and the non-voice section is a time section of the input signal in which voice is not being emitted. The voice presence probability is calculated for each frame. As an example, the voice presence probability is calculated using a third trained model. The third trained model is a neural network trained to input the integrated features and output the voice presence probability. Note that the third trained model is a neural network trained to impose a penalty on the difference between the correct answer label for the voice section and the non-voice section and the voice presence probability. The third trained model is stored in the storage device 12 or the like.

[0022] The voice activity detection unit 117 detects a voice activity, which is a time period during which speech is being produced, and / or a non-voice activity, which is a time period during which speech is not being produced, based on a comparison of the voice presence probability with a threshold value.

[0023] The output control unit 118 displays various information via the display device 15 and the audio device 16. For example, the output control unit 118 displays an image signal on the display device 15 and outputs an audio signal via the audio device 16.

[0024] The storage device 12 is configured by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The storage device 12 stores various calculation results by the processing circuit 11, a voice activity detection program executed by the processing circuit 11, etc. The storage device 12 is an example of a computer-readable recording medium.

[0025] The input device 13 inputs various commands from a user. As the input device 13, a keyboard, a mouse, various switches, a touch pad, a touch panel display, etc. can be used. An output signal from the input device 13 is supplied to the processing circuit 11. Note that the input device 13 may also be a computer connected to the processing circuit 11 via a wired or wireless connection.

[0026] The communication device 14 is an interface for communicating information with external devices connected via a network to the voice activity detection device 100. The communication device 14 receives, for example, acoustic signals and non-acoustic signals from a device that collects acoustic signals and non-acoustic signals, and receives a first trained model, a second trained model, and a third trained model from a learning device described later.

[0027] The display device 15 displays various information. As the display device 15, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art can be appropriately used. The display device 15 may be a projector.

[0028] The acoustic device 16 converts an electrical signal into sound and radiates the sound. As the acoustic device 16, a magnetic speaker, a dynamic speaker, a condenser speaker, or any other speaker known in the art can be appropriately used.

[0029] Next, a description will be given of an example of the voice activity detection process performed by the processing circuit 11 of the voice activity detection device 100. For the sake of concreteness in the following description, it is assumed that the non-acoustic signal is an image signal.

[0030] Fig. 2 is a diagram showing an example of the flow of the voice activity detection process by the processing circuit 11. Fig. 3 is a diagram showing a schematic diagram of the voice activity detection process. The voice activity detection process is executed by the processing circuit 11 operating in accordance with a voice activity detection program stored in the storage device 12 or the like.

[0031] 2 and 3, the input signal acquisition unit 111 acquires an input signal including an audio signal and an image signal (step SA1). The input signal is a video signal including an audio signal and an image signal related to the same sound source.

[0032] Fig. 4 is a diagram showing an example of an input signal (video signal), an audio signal, and an image signal. As shown in Fig. 4, the video signal is a time-series signal including a time-series audio signal and a time-series image signal that are synchronized in time. The length of the time period of the video signal is not particularly limited, but is assumed to be, for example, a frame length of about 10 seconds.

[0033] The video signal is collected by a video camera device including a microphone and an imaging device. The acoustic signal is collected by a microphone. The microphone collects the sound related to the speaker's speech, converts the sound pressure of the collected sound into an analog electric signal (acoustic signal), and converts the acoustic signal into a digital time domain electric signal (acoustic signal) by A / D conversion. The time domain acoustic signal is acquired by an input signal acquisition unit 111 and converted into a frequency domain acoustic signal by short-time Fourier transform or the like. The image signal is collected substantially simultaneously with the acoustic signal. The image signal is collected by an imaging device including multiple imaging elements such as CCDs (Charge Coupled Devices). The imaging device optically captures a speaker who is speaking and generates a digital spatial domain image signal (image data) related to the speaker on a frame-by-frame basis. The image signal is required to correlate with the speaker's speech. The image frame only needs to include at least the lip area, the shape of which changes in response to speech, as a captured subject, and may also include the speaker's entire face area. The image signal is acquired by the input signal acquisition unit 111 on a frame-by-frame basis.

[0034] Here, the time-series audio signal A and the time-series image signal V are defined according to the following formula (1). The time-series audio signal A is an audio signal having T dimensions in the time domain and F dimensions in the frequency domain of the frame to be processed. The image signal V is an image signal having dimensions of time T, vertical width H, horizontal width W, and color channel C.

[0035]

number

[0036] When step SA1 is performed, the acoustic feature calculation unit 112 uses the first trained model to derive an acoustic feature E from the acoustic signal A acquired in step SA1. A (Step SA2). A is calculated for each frame based on the acoustic signal A. A is the time series data. The first trained model inputs the acoustic signal A and extracts the acoustic features E AThe neural network is trained to output the following: A The first trained model is generated by a learning device described below.

[0037] The relationship between the acoustic signal and the acoustic feature here is as follows: The acoustic signal is waveform data of a time series of sound pressure values ​​of the voice uttered by the speaker. The acoustic signal correlates with the voice uttered by the speaker. For example, the peak value of the acoustic signal has a relatively high value when the speaker is speaking, and a relatively low value when the speaker is not speaking. The acoustic feature is designed so that its value correlates with the peak value of the acoustic signal, in other words, so as to distinguish between speech components and silent components contained in the acoustic signal. For example, the higher the peak value of the acoustic signal, the higher the value of the acoustic feature, and the lower the peak value of the acoustic signal, the lower the value of the acoustic feature.

[0038] When step SA2 is performed, the non-acoustic feature calculation unit 113 uses the second trained model to derive the image feature E from the image signal V acquired in step SA1. V (Step SA3). V is calculated for each frame based on the image signal V. That is, the image feature E V is a time series data. The second trained model inputs an image signal V and extracts image features E V The neural network is trained to output the following: V The second trained model is generated by a learning device, which will be described later.

[0039] The relationship between the image signal and the image feature here is as follows: The image signal correlates with the shape of the facial region when the speaker is speaking. The image feature is designed to distinguish between speech components and non-speech components contained in the image signal. Specifically, the lip region of the speaker represented by the image signal has a different shape when the speaker is making a sound and when he is not making a sound. The image feature is designed so that its value correlates with the shape of the facial region of the speaker. For example, the more the speaker's mouth is open, the higher the value of the image feature, and the more the speaker's mouth is closed, the lower the value of the image feature.

[0040] The order of steps SA2 and SA3 is not particularly limited, and step SA2 may be performed after step SA3, or steps SA2 and SA3 may be performed in parallel.

[0041] After step SA3, the confidence weight calculation unit 114 calculates a confidence weight based on the difference between the acoustic feature calculated in step SA2 and the image feature calculated in step SA3 (step SA4). A and image feature E V A first confidence weight Λ is calculated as a difference between a middle value (e.g., 0.5) and an upper limit value (e.g., 1) between the A and a second reliability weight Λ calculated by subtracting the first reliability weight from the upper limit. V The first reliability weight Λ is calculated. A and the second confidence weight Λ V is expressed by the following equations (2) and (3). That is, the first reliability weight Λ A is the acoustic feature E A and image feature E V The first confidence weight Λ is based on the Kullback-Leibler divergence, which is a measure of the distance between A is the acoustic feature E A and image feature E V The Kullback-Leibler divergence based on the above is nonlinearly transformed into a scale ranging from an intermediate value (e.g., 0.5) to an upper limit value (e.g., 1). As shown in equation (3), the second reliability weight ΛV is the first reliability weight Λ from the upper limit A It is a scale with a value subtracted from

[0042]

number

[0043] Here, E A (t、d) E is an acoustic feature vector at frame time t ∈ {1, 2, ..., T} and coordinate d ∈ {1, 2, ..., D} compressed to D dimensions, and is an example of an acoustic feature. V (t、d) is an image feature vector at frame time t ∈ {1, 2, ..., T} and coordinate d ∈ {1, 2, ..., D} compressed to D dimensions, and is an example of an image feature. δ is an acoustic feature E A and image feature E V The first confidence weight Λ is a scale factor that expands or reduces the magnitude of the difference between A and the second confidence weight Λ V is a weight that depends on the frame time t.

[0044] As described later, the first trained model and the second trained model have acoustic features E A and image feature E V The first trained model and the second trained model are trained so that the difference between them is small. In order to use the first trained model and the second trained model trained in this way, when an abnormal image is input, the acoustic feature E A and image feature E V The difference between the first reliability weight Λ becomes large, and the first reliability weight Λ becomes large according to the formulas (2) and (3). A is close to the upper limit, and the second confidence weight Λ V will have a value close to the lower limit.

[0045] After step SA4, the integrated feature calculation unit 115 calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight calculated in step SA4 (step SA5). In step S5, the integrated feature calculation unit 115 calculates a first reliability weight Λ A and the second confidence weight Λ V Using the acoustic features E A and image feature E V Calculate the integrated feature of Z AV is calculated for each frame time t. Specifically, the integrated feature Z AV is expressed by the following equation (4), the first reliability weight Λ A and acoustic feature E A and the second reliability weight Λ V and image feature E V It is calculated as the sum of the integrated value of and.

[0046]

number

[0047] When step SA5 is performed, the voice presence probability calculation unit 116 uses the third trained model to calculate the integrated feature Z AV The presence of voice probability Y is calculated from the integrated feature Z AV The third trained model is the integrated feature Z AV The neural network is trained to input the integrated feature Z AV The third trained model is generated by a learning device, which will be described later.

[0048] When step SA6 is performed, the voice activity detection unit 117 detects a voice activity based on a comparison of the voice presence probability Y calculated in step SA6 with a threshold value η (step SA7).

[0049] For each frame time, the value of the presence of voice probability Y is compared with a threshold value η. The threshold value η is set as the boundary between values ​​corresponding to speech and values ​​not corresponding to speech. For example, when the presence of voice probability Y ranges from "0" to "1", the threshold value η is set to "0.5". When the value of the presence of voice probability Y is greater than the threshold value η, the frame time is determined to be a voice section, and when the value of the presence of voice probability Y is smaller than the threshold value η, the frame time is determined to be a non-voice section. By performing this determination process for each frame time, the voice section and the non-voice section are detected within the time section corresponding to the input signal. A label of a voice section or a non-voice section is assigned to each frame time of the input signal.

[0050] After step SA7 is performed, the output control unit 118 outputs the speech section and / or non-speech section detected in step SA7 (step SA8). Various output modes are possible. As one example, in step SA8, the output control unit 118 displays the speech section and / or non-speech section on the display device 15. At this time, it is preferable that the output control unit 118 displays the speech section and / or non-speech section in visual association with the audio signal and / or image signal.

[0051] This completes the voice activity detection process by the processing circuit 11. After the voice activity detection, the input signal is subjected to processes such as voice recognition and data compression.

[0052] As described above, the voice activity detection device 100 according to this embodiment includes an acoustic feature calculation unit 112, a non-acoustic feature calculation unit 113, a confidence weight calculation unit 114, an integrated feature calculation unit 115, a voice presence probability calculation unit 116, and a voice activity detection unit 117. The acoustic feature calculation unit 112 calculates an acoustic feature based on an acoustic signal. The acoustic feature has a value correlating with pronunciation. The non-acoustic feature calculation unit 113 calculates a non-acoustic feature based on a non-acoustic signal. The non-acoustic feature has a value correlating with pronunciation. The confidence weight calculation unit 114 calculates a confidence weight based on a difference between the acoustic feature and the non-acoustic feature. The integrated feature calculation unit 115 calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the confidence weight. The voice presence probability calculation unit 116 calculates a voice presence probability from the integrated feature. The voice activity detection unit 117 detects a voice activity, which is a time period during which speech is being produced, and / or a non-voice activity, which is a time period during which speech is not being produced, based on a comparison of the voice presence probability with a threshold value.

[0053] According to this embodiment, when calculating the voice presence probability from the integrated feature of the acoustic feature based on the acoustic signal and the non-acoustic feature based on the non-acoustic signal, a confidence weight is calculated for each of the acoustic feature and the non-acoustic signal, and the integrated feature and the voice presence probability are calculated by taking the confidence weight into account. Therefore, even when a continuous or sudden abnormal non-acoustic signal occurs, it is possible to detect a voice section with high accuracy. In addition, preferably, the acoustic feature is calculated from the acoustic signal using a first trained model, and the non-acoustic feature is calculated from the non-acoustic signal using a second trained model. The first trained model and the second trained model are generated by training using a loss function defined to impose a penalty on the difference between the acoustic feature and the non-acoustic feature output for the same voice source. By obtaining the acoustic feature and the non-acoustic feature using the first trained model and the second trained model obtained by such training, the difference between the acoustic feature and the non-acoustic feature becomes small for the input of a normal non-acoustic signal, and the difference between the acoustic feature and the non-acoustic feature becomes large for the input of an abnormal non-acoustic signal. By using the difference value between such acoustic features and non-acoustic features as a confidence weight and obtaining an integrated feature of the weighted acoustic features and the non-acoustic features, it is possible to further improve the detection accuracy of speech segments and / or non-speech segments even when an abnormal non-acoustic signal is input.

[0054] (Learning device) Fig. 5 is a diagram showing an example of the configuration of a learning device 200. The learning device 200 is a computer that generates a first trained model used to calculate an acoustic feature, a second trained model used to calculate an image feature, and a third trained model used to detect a voice section and / or a non-voice section. As shown in Fig. 9, the learning device 200 has a processing circuit 21, a storage device 22, an input device 23, a communication device 24, a display device 25, and an audio device 26.

[0055] The processing circuit 21 has a processor such as a CPU and a memory such as a RAM. The processing circuit 21 executes a learning process for generating a first trained model, a second trained model, and a third trained model by executing a learning program stored in the storage device 22. The learning program is recorded in a non-transitory computer-readable recording medium. The processing circuit 21 realizes an input signal acquisition unit 211, an acoustic feature calculation unit 212, a non-acoustic feature calculation unit 213, a reliability weight calculation unit 214, an integrated feature calculation unit 215, a voice presence probability calculation unit 216, an update unit 217, an update end determination unit 218, and an output control unit 219 by reading and executing the learning program from the recording medium. The learning program may have a plurality of modules in which the functions of the units 211 to 219 are divided and implemented.

[0056] The hardware implementation of the processing circuit 21 is not limited to the above-mentioned embodiment. For example, the processing circuit 21 may be configured by a circuit such as an ASIC that realizes the input signal acquisition unit 211, the acoustic feature calculation unit 212, the non-acoustic feature calculation unit 213, the reliability weight calculation unit 214, the integrated feature calculation unit 215, the voice presence probability calculation unit 216, the update unit 217, the update end determination unit 218, and the output control unit 219. The input signal acquisition unit 211, the acoustic feature calculation unit 212, the non-acoustic feature calculation unit 213, the reliability weight calculation unit 214, the integrated feature calculation unit 215, the voice presence probability calculation unit 216, the update unit 217, the update end determination unit 218, and / or the output control unit 219 may be implemented in a single integrated circuit, or may be implemented individually in multiple integrated circuits.

[0057] The acquiring unit 211 acquires training data having a plurality of training samples. The training sample is an input signal including a set of an acoustic signal and a non-acoustic signal. The input signal is a time-series signal, and includes a time-series acoustic signal and a time-series non-acoustic signal. As described above, the non-acoustic signal is an image signal related to a speaker who is speaking, a sensor signal related to a physiological response of the speaker's lips or facial muscles due to the speaking, or the like.

[0058] The acoustic feature calculation unit 212 calculates acoustic features from the acoustic signal using a first neural network. The acoustic features calculated by the acoustic feature calculation unit 212 are similar to the acoustic features calculated by the acoustic feature calculation unit 112. A first trained model is generated by training the first neural network.

[0059] The non-acoustic feature calculation unit 213 calculates non-acoustic features from the non-acoustic signal using a second neural network. The non-acoustic features calculated by the non-acoustic feature calculation unit 213 are similar to the non-acoustic features calculated by the non-acoustic feature calculation unit 113. A second trained model is generated by training the second neural network.

[0060] The first and second neural networks are trained to penalize differences between acoustic and non-acoustic features of the same audio source.

[0061] The confidence weight calculation unit 214 calculates the confidence weight based on the difference between the acoustic feature and the non-acoustic feature. The confidence weight calculated by the confidence weight calculation unit 214 is similar to the confidence weight calculated by the confidence weight calculation unit 114.

[0062] The integrated feature calculation unit 215 calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight calculated by the reliability weight calculation unit 214. The integrated feature calculated by the integrated feature calculation unit 215 is similar to the integrated feature calculated by the integrated feature calculation unit 115.

[0063] The presence of voice probability calculation unit 216 calculates the presence of voice probability from the integrated features using a third neural network. The presence of voice probability calculated by the presence of voice probability calculation unit 216 is similar to the presence of voice probability calculated by the presence of voice probability calculation unit 116. The third neural network is trained to penalize the difference between the ground truth labels for the voice and non-voice segments and the presence of voice probability. A third trained model is generated by training the third neural network.

[0064] The update unit 217 updates the first neural network, the second neural network and the third neural network using a third loss function including a first loss function for penalizing a difference between acoustic features and non-acoustic features for the same voice source, and a second loss function for penalizing a difference between ground truth labels and voice presence probabilities for voice segments and non-voice segments.

[0065] The update end determination unit 218 determines whether or not the stop condition of the learning process is satisfied. If it is determined that the stop condition is not satisfied, the calculation of the acoustic feature by the acoustic feature calculation unit 212, the calculation of the non-acoustic feature by the non-acoustic feature calculation unit 213, the calculation of the reliability weight by the reliability weight calculation unit 214, the calculation of the integrated feature by the integrated feature calculation unit 215, the calculation of the voice presence probability by the voice presence probability calculation unit 216, and the update of the first neural network, the second neural network, and the third neural network by the update unit 217 are repeated. If it is determined that the stop condition is satisfied, the first neural network at this point is output as the first trained model, the second neural network is output as the second trained model, and the third neural network is output as the third trained model.

[0066] The output control unit 219 displays various information via the display device 25 and the audio device 26. For example, the output control unit 219 displays an image signal on the display device 25 and outputs an audio signal via the audio device 26.

[0067] The storage device 22 is configured with a ROM, HDD, SSD, integrated circuit storage device, etc. The storage device 22 stores various calculation results by the processing circuit 21, learning programs executed by the processing circuit 21, etc. The storage device 22 is an example of a computer-readable recording medium.

[0068] The input device 23 inputs various commands from a user. As the input device 23, a keyboard, a mouse, various switches, a touch pad, a touch panel display, etc. can be used. An output signal from the input device 23 is supplied to the processing circuit 21. Note that the input device 23 may be a computer connected to the processing circuit 21 via a wired or wireless connection.

[0069] The communication device 24 is an interface for communicating information with an external device connected to the learning device 200 via a network.

[0070] The display device 25 displays various information. As the display device 25, a CRT display, a liquid crystal display, an organic EL display, an LED display, a plasma display, or any other display known in the art can be appropriately used. The display device 25 may be a projector.

[0071] The acoustic device 26 converts an electrical signal into sound and radiates the sound. As the acoustic device 26, a magnetic speaker, a dynamic speaker, a condenser speaker, or any other speaker known in the art can be appropriately used.

[0072] Next, a description will be given of an example of the learning process performed by the processing circuitry 21 of the learning device 200. For the sake of specificity in the following description, it is assumed that the non-acoustic signal is an image signal.

[0073] 6 is a diagram showing an example of the flow of a learning process by the processing circuit 21. The learning process is executed by the processing circuit 21 operating in accordance with a learning program stored in the storage device 22 or the like. In the learning process, the processing circuit 21 trains the first neural network NN1, the second neural network NN2, and the third neural network NN3 in parallel. More specifically, the processing circuit 21 performs supervised learning of an integrated neural network including the first neural network NN1, the second neural network NN2, and the third neural network NN3 based on training data.

[0074] 7 is a diagram showing an example of the configuration of an integrated neural network. As shown in FIG. 7, the integrated neural network includes a first neural network NN1, a second neural network NN2, a confidence weight calculation module NM1, an integrated feature calculation module NM2, and a third neural network NN3. The first neural network NN1 inputs an acoustic signal A and calculates an acoustic feature E A The second neural network NN2 inputs the image signal V and outputs the image feature E V The confidence weight calculation module NM1 calculates the confidence weight of the acoustic feature E output from the first neural network NN1. A and the image features E output from the second neural network NN2 V and input the reliability weight Λ, i.e., the first reliability weight Λ A and the second confidence weight Λ V The reliability weight calculation module NM1 is a module corresponding to the function of the reliability weight calculation unit 214. The integrated feature calculation module NM2 calculates the acoustic feature E A and the image features E output from the second neural network NN2 V and the first reliability weight Λ output from the reliability weight calculation module NM1. A and the second confidence weight Λ V Enter and integrate the features Z AV The integrated feature calculation module NM2 is a module corresponding to the function of the integrated feature calculation unit 215. The third neural network NN3 calculates the integrated feature Z AV is input and a voice presence probability Y^ is output. It is assumed that initial values ​​of learning parameters are assigned to the first neural network NN1, the second neural network NN2, and the third neural network NN3. The learning parameters are weights, biases, etc. Note that the learning parameters may include any hyperparameters.

[0075] The first neural network NN1 extracts acoustic features E from the acoustic signal A. A The first neural network NN1 has an encoder network architecture capable of calculating the following: V The third neural network NN3 has an encoder network architecture capable of calculating the integrated feature Z AV The architecture of the detection network is capable of calculating the voice presence probability Y^ from the above. As an example, the third neural network NN3 includes three dense layers. All convolutional layers and dense layers of the first neural network NN1, the second neural network NN2, and the third neural network NN3 are followed by normalized linear function units, and a normalized linear function is applied to the output of the layers. Note that the final layers of the first neural network NN1, the second neural network NN2, and the third neural network NN3 are followed by sigmoid activation function units, rather than normalized linear function units, and a sigmoid function is applied to the output of the final layers.

[0076] The integrated neural network receives an audio signal A and an image signal V and outputs a voice presence probability Y^. Learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3 are trained so as to minimize a third loss function that evaluates the error between the voice presence probability Y^ and a correct label (ground truth) Y. Hereinafter, the learning of the first neural network NN1, the second neural network NN2, and the third neural network NN3 will be described with reference to Figs. 6 and 7.

[0077] 6 and 7, the acquisition unit 211 acquires an input signal including an acoustic signal A and an image signal V (step SB1). In step SB1, the input signal, which is one learning sample, is acquired. The frame length of the time section of the input signal is not particularly limited, but is assumed to be, for example, about 10 frames. The acoustic signal A and the image signal V included in the input signal are synchronized in time.

[0078] After step SB1 is performed, the acoustic feature calculation unit 212 uses the first neural network NN1 to calculate an acoustic feature fs from the acoustic signal A acquired in step SB1 (step SB2). It is assumed that the first neural network NN1 in step SB2 has not yet completed learning. It is assumed that the acoustic signal A input to the first neural network NN1 has been transformed from the time domain to the frequency domain.

[0079] After step SB2, the non-acoustic feature calculation unit 213 uses the second neural network NN2 to calculate image features fv from the image signal V acquired in step SB1 (step SB3). It is assumed that the learning of the second neural network NN2 in step SB3 has not been completed.

[0080] The order of steps SB2 and SB3 is not particularly limited, and step SB2 may be performed after step SB3, or steps SB2 and SB3 may be performed in parallel.

[0081] When step SB3 is performed, the confidence weight calculation unit 214 uses the confidence weight calculation module NM1 to calculate the acoustic feature E A and the image feature E calculated in step SB3 V In step SB4, the confidence weight calculation unit 214 first calculates a confidence weight based on the difference between the acoustic feature E A and image feature E VA first confidence weight Λ is calculated as a difference between a middle value (e.g., 0.5) and an upper limit value (e.g., 1) between the A and a second reliability weight Λ calculated by subtracting the first reliability weight from the upper limit. V Calculate.

[0082] When step SB4 is performed, the integrated feature calculation unit 215 uses the integrated feature calculation module NM2 to calculate the reliability weight Λ calculated in step SB4. A ,Λ V Based on E A and image feature E V Integration with Z AV In step S5, the integrated feature calculation unit 115 calculates the first reliability weight Λ A and the second confidence weight Λ V Using the acoustic features E A and image feature E V Calculate the integrated feature of Z AV is calculated for each frame time t. In addition, when the image signal does not include an abnormal image and includes only normal images in the learning stage, the first reliability weight Λ A and the second confidence weight Λ V may be fixed at an intermediate value of 0.5.

[0083] When step SB5 is performed, the voice presence probability calculation unit 216 uses the third neural network NN3 to calculate the integrated feature Z AV The voice presence probability Y^ is calculated from (step SB6).

[0084] When step SB6 is performed, the update unit 217 calculates a first loss function L for penalizing differences between acoustic features and non-acoustic features related to the same speech source. KLD and a second loss function L to penalize the discrepancy between the ground truth labels and the speech presence probabilities for voice and non-voice segments. BCE and the third loss function L TOTALThe first neural network NN1, the second neural network NN2, and the third neural network NN3 are updated using the third loss function L at frame time t and coordinate d as shown in the following formula (5). TOTAL is the first loss function L KLD and the second loss function L BCE The first loss function L KLD Specifically, as shown in the following equation (6), the first loss function L KLD is the acoustic feature E A and image feature E V The Kullback-Leibler divergence is defined as the Kullback-Leibler divergence based on the acoustic feature E A and image feature E V As shown in the following equation (7), the second loss function L BCE is defined as the binary cross entropy (BCE), which is used as a measure to evaluate the difference between the correct labels y and the speech presence probability y^ for speech and non-speech segments.

[0085]

number

[0086] The update unit 217 calculates the third loss function L according to an arbitrary optimization method. TOTAL The learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3 are updated so as to minimize the acoustic feature E A and image feature E VThe learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3 are updated so as to comprehensively minimize the difference between the correct label y and the voice presence probability y^. Any optimization method such as stochastic gradient descent or Adam (adaptive moment estimation) may be used.

[0087] After step SB7 is performed, the update end determination unit 218 determines whether or not a stop condition is satisfied (step SB8). The stop condition may be set, for example, that the number of updates of the learning parameters has reached a predetermined number, that the update amount of the learning parameters is less than a threshold, etc. If it is determined that the stop condition is not satisfied (step SB8: NO), the input signal acquisition unit 211 acquires other acoustic signals and image signals (step SB1). Then, for the other acoustic signals and image signals, the following steps are executed in order: calculation of acoustic features by the acoustic feature calculation unit 212 (step SB2), calculation of non-acoustic features by the non-acoustic feature calculation unit 213 (step SB3), calculation of reliability weights by the reliability weight calculation unit 214 (step SB4), calculation of integrated features by the integrated feature calculation unit 215 (step SB5), calculation of the presence of voice probability by the presence of voice probability calculation unit 216, updating of the first neural network, the second neural network and the third neural network by the update unit 217 (step SB6), and determination of whether the stop condition is satisfied by the update termination determination unit 218 (step SB7).

[0088] Note that steps SB2 to SB7 may be repeated for one training sample (batch training), or steps SB2 to SB7 may be repeated for multiple training samples (mini-batch training).

[0089] Then, when it is determined in step SB8 that the stopping condition is satisfied (step SB8: YES), the update end determination unit 218 outputs the first neural network NN1 at the time when the stopping condition is satisfied as the first trained model, the second neural network NN2 as the second trained model, and the third neural network NN3 as the third trained model (step SB9). The first trained model, the second trained model, and the third trained model are transmitted to the voice activity detection device 100 via the communication device 24 or the like, and stored in the storage device 12. Note that the update end determination unit 218 may output an integrated neural network including the first trained model, the second trained model, the third trained model, the confidence weight calculation module NM1, and the integrated feature calculation module NM2.

[0090] When step SB8 is performed, the learning process by the processing circuit 21 ends. As described above, the voice activity detection device 100 calculates the voice presence probability from the audio signal and the image signal by using the first trained model, the second trained model, and the third trained model generated by the training device 200. The voice activity detection device 100 may input the audio signal and the image signal to the integrated neural network to calculate the voice presence probability.

[0091] As described above, the learning device 200 according to this embodiment includes an input signal acquisition unit 211, an acoustic feature calculation unit 212, a non-acoustic feature calculation unit 213, a confidence weight calculation unit 214, an integrated feature calculation unit 215, a voice presence probability calculation unit 216, and an update unit 217. The input signal acquisition unit 211 acquires an acoustic signal and a non-acoustic signal related to the same sound source. The acoustic feature calculation unit 212 calculates an acoustic feature from the acoustic signal using a first neural network. The non-acoustic feature calculation unit 213 calculates a non-acoustic feature from the non-acoustic signal using a second neural network. The confidence weight calculation unit 214 calculates a confidence weight based on the difference between the acoustic feature and the non-acoustic feature. The integrated feature calculation unit 215 calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the confidence weight. The voice presence probability calculation unit 216 calculates a voice presence probability from the integrated feature using a third neural network. The update unit 217 updates the first neural network, the second neural network, and the third neural network using a third loss function including a first loss function that penalizes a difference between acoustic features and non-acoustic features, and a second loss function that penalizes a difference between correct labels and voice presence probabilities for voice segments and non-voice segments.

[0092] By obtaining acoustic features and non-acoustic features using the first trained model and the second trained model obtained by such learning, the difference between the acoustic features and the non-acoustic features becomes small for a normal non-acoustic signal input, and the difference between the acoustic features and the non-acoustic features becomes large for an abnormal non-acoustic signal input. By using such difference values ​​between the acoustic features and the non-acoustic features as confidence weights and obtaining integrated features of the weighted acoustic features and the non-acoustic features, it becomes possible to further improve the detection accuracy of the voice section and / or the non-speech section even when an abnormal non-acoustic signal input occurs.

[0093] (Example) The performance of the integrated neural network according to this embodiment (hereinafter referred to as the proposed model), which includes the first neural network, the second neural network, and the third neural network generated by the above embodiment, was verified. The audio-only model, the AV-catenation model, the baseline model (AV-summation model), and the proposed model were trained and evaluated using the GRID-AV sentence corpus (hereinafter simply referred to as the GRID corpus), which consists of AV recordings in which 34 speakers (18 men and 16 women) spoke 1,000 sentences each.

[0094] To train the model in noise, background noises provided by the 4th CHiME Challenge were randomly selected and mixed into all acoustic training data from the GRID corpus with SNRs ranging from -5 to +20 dB. The CHiME4 noises were recorded from four locations: a bus, a cafeteria, a pedestrian area, and a road intersection. To evaluate performance under matched noise conditions, the CHiME4 noises were mixed into all test acoustic recordings from the GRID corpus with SNRs ranging from -5 to +15 dB, at 5 dB intervals. Similarly, to evaluate performance under unmatched noise conditions, background noises from the East Anglia dataset were randomly mixed into all test acoustic recordings from the GRID corpus with SNRs ranging from -5 to +15 dB, at 5 dB intervals. The East Anglia dataset provides environmental noises collected from 10 locations: a bar, a beach, a bus, a car, a football match, a laundromat, a lecture, an office, a train station, and a street.

[0095] FIG. 8 is a diagram showing an example of an image signal (image data) according to the embodiment. Since the GRID corpus does not include abnormal images, three types of abnormal images were generated as shown in FIG. 8. The abnormal images were used only for evaluation. The first type is a random pixel image in which random pixels are uniformly distributed, assuming a video reading error or crash. The second type is a non-lip area image cut out from the GRID corpus, assuming a face landmark detection error. The third type is 10 types of face mask images, assuming that the speaker is wearing a mask.

[0096] Each speaker's recordings in the GRID corpus were split into training, validation, and test datasets in a ratio of 6:2:2. The validation dataset was used to identify the appropriate hyperparameters for each model and each experimental condition. For input speech, all speech recordings were resampled at a sampling rate of 16 kHz and a 128-dimensional log-compressed mel-spectrogram was computed using a short-time Fourier transform with a window size of 1,280 samples (80 ms) and a hop length of 640 samples (40 ms). For input images, all video recordings were cropped to the lip region using a 68-coordinate facial landmark detector and transformed to 25 frames / s (40 ms) with a resolution of H × W = 40 × 64 pixels, and RGB channels were normalized between 0 and 1. All models were trained with Gradient Clipping using Adam optimization. The learning rate was initialized at 0.0001 and the batch size was set to 1. The ground truth labels for VAD in (7) were generated by applying the VAD method to the corresponding good quality speech data in the GRID corpus. The ground truth value for k=1 was set to 1 for input acoustic frames containing speech and 0 for frames with silence or noise only. The scale factor δ in (2) was set to 100 for all experimental conditions in the proposed model.

[0097] FIG. 9 is a table showing the results of validation under matched noise conditions using the CHiME4 dataset. FIG. 10 is a table showing the results of validation under mismatched noise conditions using the East Anglia dataset. In FIG. 9 and FIG. 10, the area under the receiver operating characteristic (AUROC) is used as an index for quantitative performance evaluation of AV-VAD. The validation results for each model with and without abnormal image input are shown. The best value is indicated in bold, and the second best value is underlined.

[0098] In the absence of abnormal image input, the AV model generally performs better than the audio-only model (A-only), especially at low SNR. The baseline model performs best on average among the AV models, except for the proposed model. However, in the presence of abnormal image input, the baseline model performs worse than the audio-only model, with a maximum relative improvement of -13.75%.

[0099] Figure 11 shows the average ROC curves of the Audio-only model and the baseline model under matched noise conditions. As shown in Figure 11, the False Acceptance Rate (FAR) tends to increase for abnormal input images (random pixels) whose pixel values ​​change frequently between frames, while the False Rejection Rate (FRR) tends to increase for abnormal input images (masks) whose pixel values ​​do not change frequently between frames. This result indicates that pixel value changes related to mouth movement are used as an important visual feature in AV-VAD.

[0100] The proposed model significantly outperforms the baseline model in all experimental conditions, providing a relative improvement of up to 2.07% for normal image input and up to 15.97% for abnormal input images. Furthermore, the proposed model provides a relative improvement of up to 2.37% for abnormal input images compared to the audio-only model. By using the loss function shown in equation (5), the proposed model also achieves better results than the baseline model for normal input images. This result indicates that there is a high similarity between audio features and image features in the latent space of AV-VAD. This improves the overall learning efficiency of the neural network for AV-VAD, contributing to improved performance.

[0101] (Application example) The following describes the voice activity detection device 100 according to the application example. In the following description, components having substantially the same functions as those in this embodiment are given the same reference numerals, and will be described repeatedly only when necessary.

[0102] Fig. 12 is a diagram showing an example of the configuration of a voice activity detection device 100 according to an application example. As shown in Fig. 12, the processing circuit 11 realizes a normal / abnormal determination unit 119 in addition to an input signal acquisition unit 111, an acoustic feature calculation unit 112, a non-acoustic feature calculation unit 113, a confidence weight calculation unit 114, an integrated feature calculation unit 115, a voice presence probability calculation unit 116, a voice activity detection unit 117, and an output control unit 118.

[0103] The normal / abnormal determination unit 119 determines whether the non-acoustic signal acquired by the input signal acquisition unit 111 is a normal signal or an abnormal signal, based on the reliability weight calculated by the reliability weight calculation unit 114.

[0104] The output control unit 118 outputs the result of the determination made by the normal / abnormal determination unit 119 as to whether the non-acoustic signal is a normal signal or an abnormal signal to the display device 15 or the like.

[0105] Next, an example of the voice activity detection process by the processing circuit 11 of the voice activity detection device 100 according to the application example will be described. For the sake of specificity in the following description, it is assumed that the non-acoustic signal is an image signal.

[0106] Fig. 13 is a diagram showing an example of the flow of voice activity detection processing by the processing circuit 11 according to the application example. The voice activity detection processing is executed by the processing circuit 11 operating in accordance with a voice activity detection program stored in the storage device 12 or the like. Note that steps SC1 to SC7 shown in Fig. 13 are the same as steps SA1 to SA7 shown in Fig. 2, respectively, and therefore will not be described here.

[0107] As shown in Fig. 13, after step SC4 is performed, steps SC5 to SC7 are performed, and the normal / abnormal determination unit 119 determines whether the non-acoustic signal (image signal) acquired in step SC1 is a normal signal or an abnormal signal based on the reliability weight calculated in step SC4 (step SC8). In step SC8, the normal / abnormal determination unit 119 determines whether the image signal is a normal signal or an abnormal signal based on a comparison between a predetermined threshold value and the reliability weight. Specifically, the threshold value θ A and the first confidence weight Λ A Compare with θ A <Λ A If θ A ≧Λ A If θ is greater than θ, the image signal is determined to be normal. A may be experimentally or empirically set from any natural number between the intermediate value (e.g., 0.5) and the upper limit value (e.g., 1.0). A The threshold θ may be set to about 0.7. V and the second confidence weight Λ V You can compare it with θ V <Λ V If θ V ≧Λ V If θ is exceeded, the image signal is determined to be abnormal. Vmay be experimentally or empirically set from any natural number between the intermediate value (e.g., 0.5) and the lower limit value (e.g., 0.0). A may be set to about 0.3.

[0108] After steps SC7 and SC8 have been performed, the output control unit 118 outputs the voice section detection result from step SC7 and the normal / abnormal determination result from step SC8 (step SC9). In step SC9, the output control unit 118 displays the voice section detection result and the normal / abnormal determination result on the display device 15. As an example, the output control unit 118 displays a display screen including the voice section detection result and the normal / abnormal determination result on the display device 15.

[0109] FIG. 14 is a diagram showing an example of a display screen I1 including a voice section detection result and a normal / abnormal judgment result. As shown in FIG. 14, the display screen I1 displays an input signal, a voice section detection result, and a normal / abnormal image judgment result in a time series. The input signal, which is a video signal, is displayed in a time series horizontally at a predetermined frame interval. As the voice section detection result, a label indicating that the frame is a voice section or a non-voice section is displayed in a time series. In FIG. 14, a label "voice" indicating that it is a voice section and a label "non-voice" indicating that it is a non-voice section are displayed. As the normal / abnormal image judgment result, a label indicating that the image signal of the input signal of the frame is a normal signal or an abnormal signal is displayed in a time series. In FIG. 14, a label "normal" indicating that it is a normal signal and a label "abnormal" indicating that it is an abnormal signal are displayed.

[0110] By observing the display screen I1, the operator can know whether the image signal is abnormal or normal. By knowing whether the image signal is abnormal or normal, the operator can estimate the reliability of the voice activity detection result.

[0111] With the above, the voice activity detection process by the processing circuit 11 according to the application example is completed.

[0112] The flow of the voice activity detection process according to the above application example is an example, and is not limited to the flow shown in FIG. 13. The determination process by the normal / abnormal determination unit 119 in step SC8 may be performed at any timing as long as it is after the reliability weight calculation process by the reliability weight calculation unit 114 in step SC4. For example, it may be performed after the reliability weight calculation process by the reliability weight calculation unit 114 in step SC4 and before steps SC5 to SC7 are performed. In this case, when the image signal is determined to be normal, steps SC5 to SC7 are performed, and when the image signal is determined to be abnormal, steps SC5 to SC7 may not be performed. In this way, by stopping the subsequent processing when it is determined to be abnormal, it is possible to reduce the overall amount of processing. Furthermore, since the voice activity detection is performed only when it is determined to be normal, the reliability of the voice activity detection result is improved.

[0113] The above embodiment is merely an example, and various modifications are possible. For example, the acoustic signal is a voice signal, but may be a vocal cord signal or vocal tract signal obtained by decomposing the voice signal. Also, the acoustic signal is waveform data of sound pressure values ​​in the time domain or frequency domain, but may be data obtained by converting the waveform data into an arbitrary space.

[0114] Thus, according to some of the above-described embodiments, it is possible to reduce the deterioration of detection accuracy due to the input of an abnormal image.

[0115] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0116] 11...processing circuit, 12...memory device, 13...input device, 14...communication device, 15...display device, 16...acoustic device, 100...voice activity detection device, 111...input signal acquisition unit, 112...acoustic feature calculation unit, 113...non-acoustic feature calculation unit, 114...confidence weight calculation unit, 115...integrated feature calculation unit, 116...voice presence probability calculation unit, 117...voice activity detection unit, 118...output control unit.

Claims

1. an acquisition unit that acquires an acoustic signal and a non-acoustic signal related to the same sound source; an acoustic feature calculation unit that calculates an acoustic feature based on the acoustic signal; a non-acoustic feature calculation unit that calculates a non-acoustic feature based on the non-acoustic signal; a confidence weight calculation unit that calculates a confidence weight based on a difference between the acoustic feature and the non-acoustic feature; an integrated feature calculation unit that calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight; a voice presence probability calculation unit for calculating a voice presence probability based on the integrated features; a detection unit that detects a voice interval, which is a time interval during which speech is being produced, and / or a non-voice interval, which is a time interval during which speech is not being produced, based on a comparison of the voice presence probability with a threshold value; A voice activity detection device comprising:

2. The acoustic feature calculation unit calculates the acoustic feature from the acoustic signal using a first trained model; The non-acoustic feature calculation unit calculates the non-acoustic feature from the non-acoustic signal using a second trained model; the speech presence probability calculation unit calculates the speech presence probability from the integrated features using a third trained model; the first trained model and the second trained model are generated by training that penalizes differences between the acoustic features and the non-acoustic features for a same speech source; The third trained model is generated by training that penalizes a difference between a correct answer label for the voice section and the non-voice section and the voice presence probability.

2. The voice activity detection device according to claim 1.

3. The reliability weight calculation unit a first confidence weight calculated as a difference between the acoustic feature and the non-acoustic feature between a median value and an upper limit value; a second reliability weight calculated as the first reliability weight minus the upper limit value; 2. The voice activity detection device according to claim 1.

4. The voice activity detection device according to claim 3 , wherein the integrated feature calculation unit calculates the integrated feature by using the first confidence weight and the second confidence weight.

5. The voice activity detection device according to claim 1 , wherein the non-acoustic signal is an image signal that is time-synchronized with the acoustic signal.

6. a determination unit that determines whether the non-acoustic signal is a normal signal or an abnormal signal based on the reliability weight; and a notification unit that displays a determination result as to whether the non-acoustic signal is a normal signal or an abnormal signal.

2. The voice activity detection device according to claim 1.

7. an acquisition unit that acquires an acoustic signal and a non-acoustic signal related to the same sound source; an acoustic feature calculation unit that calculates an acoustic feature from the acoustic signal by using a first neural network; a non-acoustic feature calculation unit that calculates non-acoustic features from the non-acoustic signal using a second neural network; a confidence weight calculation unit that calculates a confidence weight based on a difference between the acoustic feature and the non-acoustic feature; an integrated feature calculation unit that calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight; a voice presence probability calculation unit for calculating a voice presence probability from the integrated features using a third neural network; an update unit that updates the first neural network, the second neural network, and the third neural network using a third loss function including a first loss function that penalizes a difference between the acoustic features and the non-acoustic features, and a second loss function that penalizes a difference between a correct label for a voice segment and a non-voice segment and the voice presence probability; A learning device comprising:

8. The processor: A function of calculating an acoustic feature based on the acoustic signal; A function of calculating a non-acoustic feature based on a non-acoustic signal obtained from the same source as the acoustic signal; calculating a confidence weight based on the difference between the acoustic features and the non-acoustic features; a function of calculating an integrated feature of the acoustic feature and the non-acoustic feature based on the confidence weight; calculating a probability of speech presence based on the integrated features; a function of detecting a voice interval, which is a time interval during which speech is being produced, and / or a non-voice interval, which is a time interval during which speech is not being produced, based on a comparison of the voice presence probability with a threshold value; A speech activity detection program comprising:

Images