Utterance estimation device, utterance estimation method, and utterance estimation program

Abstract

The present invention enables a speaker who wishes to speak to be estimated. This utterance estimation device is provided with an utterance estimator, a feature extraction unit, an utterance estimation unit, and an output unit. The utterance estimator has learned in advance a category relating to whether or not an utterance estimation target person will utter a voice soon after from video features of the utterance estimation target person. The feature extraction unit extracts video features from a certain period from video data in which the utterance estimation target person has been captured. The utterance estimation unit inputs the video features, which have been extracted by the feature extraction unit, to the utterance estimator, and acquires an utterance estimation result output by the utterance estimator. The output unit outputs, on the basis of the utterance estimation result acquired by the utterance estimation unit, a determination result of whether the utterance estimation target person will utter a voice immediately after the certain period.

Description

[Technical field]

[0001] One aspect of the present invention relates to an utterance estimation device, an utterance estimation method, and an utterance estimation program. [Background technology]

[0002] The demand for web conferencing systems is on the rise due to the increase in people working from home during the COVID-19 pandemic. When multiple speakers discuss using a web conferencing system, communication delays and camera shooting range are limited. This makes it difficult to read each other's attitudes, and it is difficult to read the other person's speaking timing, which often results in speech collisions where multiple speakers speak at the same time. This is thought to be why speakers often miss opportunities to speak.

[0003] Conventionally, there is a technology for estimating the speech of a target speaker (target speaker) when multiple speakers are speaking at the same time. For example, in Patent Document 1, auxiliary information is created using video information in addition to the speech signal of the target speaker. Therefore, even if there are multiple speakers with similar voice characteristics, the speech signal of the target speaker can be estimated with stable accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 033587 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology cannot estimate a speaker who tried to speak but was unable to actually speak. For example, during a conference, there may be cases where multiple speakers try to speak at the same time, but only one speaker actually speaks. In this case, the conventional technology cannot estimate the speaker who was unable to speak.

[0006] The present invention has been made in light of the above circumstances, and has an object to provide an utterance estimation device, an utterance estimation method, and an utterance estimation program capable of estimating a speaker who wants to speak. [Means for solving the problem]

[0007] In order to solve the above problem, one aspect of the present invention includes an utterance estimator, a feature extraction unit, an utterance estimation unit, and an output unit. The utterance estimator learns in advance a classification of whether or not the utterance estimation target will speak immediately after a certain period of time based on the video features of the utterance estimation target. The feature extraction unit extracts video features for a certain period of time from video data in which the utterance estimation target is shown. The utterance estimation unit inputs the video features extracted by the feature extraction unit to the utterance estimator and acquires the utterance estimation result output by the utterance estimator. The output unit outputs a determination result of whether or not the utterance estimation target will speak immediately after the certain period of time based on the utterance estimation result acquired by the utterance estimation unit. The video features are feature vectors extracted by a trained video feature extractor for each of a certain number of frame images contained in a certain period of time. Effect of the Invention

[0008] According to one aspect of the present invention, by determining whether or not the utterance estimation target will speak immediately after based on the video of the utterance estimation target, it is possible to know that the utterance estimation target wanted to speak even if the utterance estimation target was not able to actually speak. Thus, it is possible to provide an utterance estimation device, an utterance estimation method, and an utterance estimation program that are capable of estimating a speaker who wants to speak. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an utterance estimation device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the software configuration of the utterance estimation device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart showing a learning process routine in the utterance estimation device according to the first embodiment. [Figure 4]FIG. 4 is a flowchart showing the details of the pre-utterance learning process subroutine in FIG. [Diagram 5] FIG. 5 is a conceptual diagram showing an example of the configuration of video data, video features, utterance labels, and an utterance estimator when the learning process is executed. [Figure 6] FIG. 6 is a flowchart showing the details of the silent learning process subroutine in FIG. [Figure 7] FIG. 7 is a flowchart showing an inference processing routine in the utterance estimation device according to the first embodiment. [Figure 8] FIG. 8 is a block diagram showing an example of a software configuration of an utterance estimation device according to the second embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart showing an inference processing routine in the utterance estimation device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] FIG. 1 is a block diagram showing an example of a hardware configuration of an utterance estimation device according to a first embodiment of the present invention.

[0011] The utterance estimation device is, for example, a PC (Personal Computer) or the like, and has a processor 11A such as a CPU (Central Processing Unit). The processor 11A may be a multi-core / multi-thread processor, and can execute multiple processes in parallel. The utterance estimation device is configured by connecting a program memory 11B, a data memory 12, an input / output interface 13, and a communication interface 14 to the processor 11A via a bus 15.

[0012] The program memory 11B is a storage medium that combines a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), with a non-volatile memory such as a read only memory (ROM). The program memory 11B stores programs required for the processor 11A to execute various processes. The programs include an utterance estimation program according to the first embodiment, in addition to application programs such as an operating system (OS) and a web conference application.

[0013] The data memory 12 is a storage that uses a combination of a non-volatile memory such as an HDD or SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium. The data memory 12 is used to store data acquired and created in the course of performing various processes.

[0014] The input / output interface 13 is an interface with an input device 16 and an output device 17 .

[0015] The input device 16 includes, for example, a camera for capturing an image of the user's face and a microphone for capturing the user's voice. The input device 16 also includes a keyboard, a pointing device, and the like for the user to input instructions to the processor 11A. The input device 16 may further include a reader for reading document files and data to be stored in the data memory 12 from a memory medium such as a USB memory, and a disk device for reading such files and data from a disk medium.

[0016] The output device 17 includes a display that displays output data to be presented to the user from the processor 11A, a printer that prints the output data, etc. Furthermore, the output device 17 may include a speaker that outputs voice data or music data.

[0017] The communication interface 14 is a wired or wireless communication unit for connecting to a network such as a LAN (Local Area Network) (not shown) or the Internet.

[0018] FIG. 2 is a block diagram showing the software configuration of the utterance estimation device in relation to the hardware configuration shown in FIG.

[0019] The processing unit 11 is composed of the processor 11A and the program memory 11B, and includes, as software processing function units, an acquisition unit 111, a feature extraction unit 112, an utterance estimation unit 113, a parameter update unit 114, and an output unit 115. All of these processing function units are realized by causing the processor 11A to execute a program stored in the program memory 11B. The processing unit 11 may also be realized in various other forms, including integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-programmable gate array).

[0020] The storage area of ​​the data memory 12 includes an input data temporary storage unit 121 and an utterance estimator storage unit 122 .

[0021] The acquiring unit 111 acquires data necessary for utterance estimation, and stores the acquired data in the input data temporary storage unit 121 of the data memory 12. For example, the acquiring unit 111 acquires video data X. The video data X may be any video data that includes a person who is a target of utterance estimation.

[0022] For example, the video data X may be a video displayed on a screen of a Web conference system. The acquisition unit 111 can acquire the video data X from a Web conference application 116 that realizes the Web conference system. That is, the processor 11A constituting the processing unit 11 executes the Web conference application 116 in a separate process, creates a screen of a Web conference in which a video of a user operating the PC input from the input device 16 via the input / output interface 13 and a video of other speakers who are Web conference participants acquired via a network (not shown) via the communication interface 14 are arranged side by side, and displays and outputs this screen to the output device 17 via the input / output interface 13. Furthermore, during execution of the Web conference application 116, the processor 11A outputs the voice of the user operating the PC input from the input device 16 via the input / output interface 13 and the voice of other speakers who are Web conference participants acquired via a network (not shown) via the communication interface 14 from the output device 17 via the input / output interface 13.

[0023] The acquisition unit 111 acquires video data of the utterance estimation target person displayed on the screen of the Web conference. Here, the utterance estimation target person may be the user of the PC, or may be another speaker participating in the Web conference. There may be a plurality of utterance estimation targets, and in this case, each person is acquired as separate video data. The acquisition unit 111 may extract video data X from a recording file of the Web conference input from the input device 16 acquired by the input unit 117.

[0024] Alternatively, the video data X may be video data of the user of the PC, which is acquired by the input unit 117 via the input / output interface 13 as data input from the input device 16. In this case, the utterance estimation target person is the user of the PC.

[0025] In addition, when performing a learning process for learning the speech estimator stored in the speech estimator storage unit 122, the acquisition unit 111 can acquire the speech section information Tp of the speech estimation target. The speech section information Tp may be any data that describes the time during which the speech estimation target speaks. For example, it may be text data that records the time from the start time of the Web conference to the speech start time of the speech estimation target and the time from the start time of the conference to the speech end time. Specifically, when the time from the start time of the Web conference to the speech start time is 5 minutes 30 seconds and the time from the start time of the conference to the speech end time is 6 minutes, the text data records "5 minutes 30 seconds to 6 minutes". The acquisition unit 111 acquires the input data "5 minutes 30 seconds to 6 minutes" input by the keyboard of the input device 16 from the input unit 117 as text data. In addition, the acquisition unit 111 may extract audio data from the recording file of the Web conference acquired by the input unit 117, and by being equipped with a voice detection function or by providing a separate speech detection unit, may automatically detect the speaking time of the person to be estimated to speak from the audio data.

[0026] When executing the learning process, the feature extraction unit 112 receives the video data X stored in the input data temporary storage unit 121 and the utterance period information Tp of the utterance estimation target person, and creates a video feature Y for a part of time in the video data X. Then, the feature extraction unit 112 creates pair data of the video feature Y and an utterance label Z indicating whether or not an utterance will occur immediately after the video feature Y. Furthermore, when executing an utterance inference process for estimating the utterance of the utterance estimation target person, the feature extraction unit 112 receives the video data X stored in the input data temporary storage unit 121, and repeatedly creates a video feature Y for a certain period from the video data X.

[0027] The video feature Y may be any video feature that corresponds to the time length of a preset fixed period in the video data X. For example, when the preset fixed period is one second, the video feature Y is a collection of frames for one second from an arbitrary time point in the video data X. Specifically, when the vertical and horizontal sizes of the video data X are 256 pixels and the video is a 24 fps video, the video feature Y is 24 images of 256×256 pixels. Alternatively, the feature extraction unit 112 may include a trained video feature extractor, and the video feature Y may be a feature amount extracted by the video feature extractor. For example, when the video feature extractor outputs a vector with a length of 512 for each frame, the video feature Y is a vector with a length of 512×24 for the 24 fps video data X.

[0028] The speech label Z may be any label indicating whether or not the person will speak immediately after the video feature Y. For example, the feature extraction unit 112 identifies a time when the person is silent for 10 seconds or more from the video data X and the utterance section information Tp, creates a video feature Y from one second of video data from an arbitrary time point during the silent time, and sets the speech label Z to "0" as a label indicating no speech. The feature extraction unit 112 also identifies a time when the person is silent for 5 seconds or more from the video data X and the utterance section information Tp, creates a video feature Y from one second of video data immediately before the utterance time, and sets the speech label Z to "1" as a label indicating speech.

[0029] The speech estimator storage unit 122 of the data memory 12 stores a neural network as a speech estimator A that has learned classification of whether or not a person to be estimated for utterance will utter immediately after an utterance is estimated based on the video features of the person. The speech estimator A may be any neural network that receives a video feature Y as input and outputs a speech estimation result W that estimates whether or not the person will utter immediately after the video feature Y. For example, when the input video feature Y is a vector of length 1024 and the speech estimation result W is the probability of two classes of whether or not the person will utter, the speech estimator A is a fully connected neural network with an input layer of 1024 and an output layer of 2.

[0030] When executing the learning process, the utterance estimation unit 113 receives a pair of a video feature Y and a speech label Z from the feature extraction unit 112, receives the utterance estimator A from the utterance estimator storage unit 122, inputs the video feature Y to the utterance estimator A, and outputs the utterance estimation result W. When executing the utterance inference process, the utterance estimation unit 113 receives the utterance estimator A from the utterance estimator storage unit 122, receives video features Y from the feature extraction unit 112 one after another at regular intervals, inputs the video features Y to the utterance estimator A, and outputs the utterance estimation results W one after another.

[0031] The parameter update unit 114 operates when the learning process is executed. The parameter update unit 114 receives the utterance estimation result W from the utterance estimation unit 113, receives the utterance estimator A from the utterance estimator storage unit 122, and updates the parameters of each neural network so as to satisfy the constraint. The constraint is to update the parameters so that the utterance estimation result W shows the same label as the utterance label Z, and any learning method set to satisfy this constraint may be used. For example, assume that the utterance estimation result W is a vector with a length of 2 and a range of [0,1], and the utterance label Z is "1" indicating the case of speech. In this case, the parameter update unit 114 converts the utterance label Z to a vector (0,1) of the same size and range as the utterance estimation result W, and calculates the squared error between the utterance estimation result W and the vector (0,1) converted from the utterance label Z. Then, the parameter update unit 114 updates the parameters of the utterance estimator A so as to reduce this squared error.

[0032] The output unit 115 operates when the utterance inference process is executed. The output unit 115 receives the utterance estimation result W from the utterance estimation unit 113, and outputs the result of judging whether or not to utter based on the utterance estimation result W to the outside of the utterance estimation device. The method of judging whether or not to utter in the output unit 115 may be any method based on the utterance estimation result W. For example, when the utterance estimation result W is a vector with a length of 2 and a range of [0, 1], and the 0th element of the vector indicates the probability of utterance, the output unit 115 judges that utterance will be performed if the 0th element of the vector is 0.5 or more, and outputs "utterance" as the judgment result as a standard output. Also, when the 0th element of the vector is less than 0.5, the output unit 115 judges that utterance will not be performed, and outputs "not utterance" as the judgment result as a standard output.

[0033] The output unit 115 inputs the determination result to, for example, a Web conference application 116 that realizes a Web conference system, and displays / outputs on the screen of the Web conference system whether the utterance estimation target will speak / not speak. In this case, the utterance estimation target may be the user of the PC, or may be another speaker participating in the Web conference. Alternatively, the output unit 115 transmits the determination result to another speaker via a network (not shown) by the communication interface 14, thereby outputting a notification to the other speaker that the utterance estimation target will speak / not speak. In this case, the utterance estimation target is the user of the PC. The PC of the participant of the Web conference that receives this notification transmits the notification to the Web conference application, and can display / output or record on the screen of the Web conference whether the utterance estimation target will speak / not speak. If it is recorded, it is possible to confirm who wanted to speak and when after the Web conference ends, and it is also possible to inquire about the utterance estimation target at a later date.

[0034] The utterance estimation device according to the present embodiment can be realized by a processor 11A, which is a computer, and an utterance estimation program stored in advance in a program memory 11B. However, it is also possible to provide the utterance estimation program to the utterance estimation device by recording it in a non-transitory computer-readable medium or through a network. The utterance estimation program thus provided can be stored in the program memory 11B. In addition, the utterance estimation program provided can be stored in a data memory 12, which is a storage, and executed by the processor 11A as necessary, so that the processor 11A can function as the processing unit 11.

[0035] Next, the processing operation of the utterance estimation device configured as above will be described, which will be divided into a learning process and an utterance inference process.

[0036] <<Learning process>> 3 is a flowchart showing a learning process routine in the utterance estimation device according to the first embodiment. The processor 11A of the utterance estimation device can perform the process shown in this flowchart by executing an utterance estimation program stored in advance in, for example, the program memory 11B.

[0037] The processor 11A first operates as the acquisition unit 111 to acquire learning video data X and utterance section information Tp for each utterance estimation target person, and stores them in the input data temporary storage unit 121 (step S11).

[0038] Thereafter, processor 11A executes a subroutine for learning immediately before utterance to learn the neural network of the speech estimator using the video feature Y immediately before utterance (step S12). Thereafter, processor 11A executes a subroutine for learning during silence to learn the neural network of the speech estimator using the video feature Y during silent periods (step S13).

[0039] Fig. 4 is a flowchart showing details of the immediately before utterance learning processing subroutine. This immediately before utterance learning processing subroutine is a process in which video data X and utterance section information Tp are input to create a video feature Y immediately before an utterance, and parameters of the utterance estimator A are updated so that the utterance estimation result W, which is the output when the video feature Y is input to the utterance estimator A, is the same as the utterance label Z. The processing of this immediately before utterance learning processing subroutine will be described with reference to a conceptual diagram shown in Fig. 5, which shows an example of the configuration of the video data X, the video feature Y and the utterance label Z created by the feature extraction unit 112, and the utterance estimator A when the learning process is executed.

[0040] First, the processor 11A operates as the feature extraction unit 112 and extracts video features Y for a certain period Td before speech, here 1 second, from the video data X used for learning stored in the input data temporary storage unit 121 (step S121). That is, the feature extraction unit 112 receives the video data X and the utterance section information Tp from the input data temporary storage unit 121, and identifies a speech period x1 in which speech continues for 5 seconds or more from the video data X. Then, the feature extraction unit 112 creates video features Y for a feature extraction period D1, which is 1 second immediately before the start time of the identified utterance period x1. For example, the feature extraction unit 112 creates video features Y, which are vectors of length 512×24, by concatenating vectors y1, y2, ..., y24 of length 512 for each of the 24 frames output by the trained video feature extractor from the feature extraction period D1 of the 24 fps video data X.

[0041] Next, the processor 11A operates as the feature extraction unit 112 and creates pair data of a video feature Y and an utterance label Z (step S122). That is, the feature extraction unit 112 creates pair data of the video feature Y extracted in the above step S121 and an utterance label Z indicating that a speech will be made immediately after the video feature Y. In this case, the feature extraction unit 112 sets the utterance label Z to "1" as a label indicating that a speech will be made. The feature extraction unit 112 transmits the created pair data of the video feature Y and the utterance label Z to the utterance estimation unit 113.

[0042] Next, the processor 11A operates as the utterance estimation unit 113, inputs the video feature Y to the utterance estimator A, and acquires the utterance estimation result W output from the utterance estimator A (step S123). That is, the utterance estimation unit 113 receives the pair data of the video feature Y and the utterance label Z transferred in the above step S122, and the utterance estimator A stored in the utterance estimator storage unit 122. Then, the utterance estimation unit 113 inputs the video feature Y to the utterance estimator A, and acquires the utterance estimation result W. The utterance estimation unit 113 transfers the acquired utterance estimation result W and the above utterance label Z to the parameter update unit 114.

[0043] Next, the processor 11A operates as the parameter update unit 114, receives the utterance estimation result W and the utterance label Z obtained in the above step S123, and the utterance estimator A stored in the utterance estimator storage unit 122, and calculates a loss indicating the distance between the utterance estimation result W and the utterance label Z, as shown by the dashed bidirectional arrow in Figure 5 (step S124).

[0044] Then, the processor 11A operates as the parameter update unit 114 and updates the parameters of the utterance estimator A so as to reduce the loss calculated in the above step S124 (step S125). That is, the parameter update unit 114 updates the parameters of the utterance estimator A so as to reduce the loss function set so that the utterance estimation result W and the utterance label Z are the same. Then, the parameter update unit 114 stores the utterance estimator A with the updated parameters in the utterance estimator storage unit 122.

[0045] 6 is a flowchart showing the details of the silence learning process subroutine. This silence learning process routine is a process in which video data X and speech section information Tp are input to create video features Y for silence periods, and parameters of utterance estimator A are updated so that the utterance estimation result W, which is the output when the video features Y are input to utterance estimator A, becomes the same as the utterance label Z.

[0046] First, the processor 11A operates as the feature extraction unit 112 and extracts a video feature Y for a certain period Td during which silence continues, here 1 second, from the video data X used for learning stored in the input data temporary storage unit 121 (step S131). That is, the feature extraction unit 112 receives the video data X and the speech section information Tp from the input data temporary storage unit 121, and identifies a silent period x2 during which silence continues for 10 seconds or more from the video data X. Then, the feature extraction unit 112 creates a video feature Y for a feature extraction period D2, which is an arbitrary 1 second period within the identified silent period x2. For example, the feature extraction unit 112 creates a video feature Y, which is a vector of length 512×24, by concatenating vectors y1, y2, ..., y24 of length 512 for each of the 24 frames output by the trained video feature extractor from the feature extraction period D2 of the 24 fps video data X.

[0047] Next, the processor 11A operates as the feature extraction unit 112 and creates pair data of a video feature Y and an utterance label Z (step S132). That is, the feature extraction unit 112 creates pair data of the video feature Y extracted in the above step S131 and an utterance label Z indicating no speech. In this case, the feature extraction unit 112 sets the utterance label Z to "0" as a label indicating no speech. The feature extraction unit 112 transmits the created pair data of the video feature Y and the utterance label Z to the utterance estimation unit 113.

[0048] Next, the processor 11A operates as the speech estimation unit 113 and, similar to step S123 in the processing of the pre-utterance learning processing subroutine, inputs the video feature Y to the speech estimator A and obtains the speech estimation result W output from the speech estimator A (step S133).

[0049] Next, the processor 11A operates as the parameter update unit 114 and calculates a loss indicating the distance between the utterance estimation result W and the utterance label Z, similarly to step S124 in the processing of the immediately before utterance learning processing subroutine (step S134).

[0050] Then, the processor 11A operates as the parameter update unit 114 and updates the parameters of the utterance estimator A so as to reduce the calculated loss (step S135), similarly to step S125 in the processing of the pre-utterance learning processing subroutine.

[0051] 4 and 6 show processing for one utterance estimation target, and the processor 11A executes such an immediately before utterance learning processing subroutine and a silent learning subroutine for the video data X of each utterance estimation target stored in the input data temporary storage unit 121.

[0052] <<Inference processing>> The inference processing routine differs from the learning processing routine in that the feature extraction unit 112 creates video features Y for a fixed period immediately before the start time of a speech period or for any fixed period during a silent period in the learning processing routine, but in the inference processing routine, the feature extraction unit 112 continues to create video features Y for a fixed period of video data X. Specifically, the feature extraction unit 112 accepts video data X stored in the input data temporary storage unit 121, and repeatedly creates video features Y for a fixed period from the video data X. Then, speech estimation is performed based on each video feature Y.

[0053] 7 is a flowchart showing an inference processing routine in the utterance estimation device according to the embodiment 1. The processor 11A of the utterance estimation device can perform the processing shown in this flowchart by executing an utterance estimation program stored in advance in, for example, the program memory 11B.

[0054] First, the processor 11A operates as the acquisition unit 111 to acquire video data X showing an utterance estimation target person, and stores the video data X in the input data temporary storage unit 121 (step S201). For example, the acquisition unit 111 can acquire the video data X in real time from the Web conference application 116. Alternatively, the acquisition unit 111 may acquire non-real time video data X input from the input device 16 acquired by the input unit 117.

[0055] Next, the processor 11A operates as the feature extraction unit 112 and extracts a video feature Y for a certain period, here, one second, from the video data X stored in the input data temporary storage unit 121 (step S202). That is, the feature extraction unit 112 accepts the video data X from the input data temporary storage unit 121 and creates one second's worth of video feature Y from the video data X. For example, the feature extraction unit 112 creates a video feature Y, which is a vector with a length of 512×24, by concatenating vectors y1, y2, ..., y24 of length 512 for each of the 24 frames output by the trained video feature extractor from one second's worth of video data X at 24 fps. The feature extraction unit 112 transmits the created video feature Y to the utterance estimation unit 113.

[0056] Next, the processor 11A operates as the utterance estimation unit 113, inputs the video feature Y to the utterance estimator A, and acquires the utterance estimation result W output from the utterance estimator A (step S203). That is, the utterance estimation unit 113 receives the video feature Y transmitted in the above step S202 and the utterance estimator A stored in the utterance estimator storage unit 122. Then, the utterance estimation unit 113 inputs the video feature Y to the utterance estimator A, and acquires the utterance estimation result W. The utterance estimation unit 113 transmits the acquired utterance estimation result W to the output unit 115.

[0057] Then, the processor 11A operates as the output unit 115, receives the utterance estimation result W transferred in step S203, and outputs the result of judging whether the utterance estimation target will speak or not based on the utterance estimation result W (step S204). For example, the output unit 115 inputs this judgment result to the Web conference application 116, thereby displaying / outputting whether the utterance estimation target will speak / not speak on the screen of the other speaker participating in the Web conference system. Alternatively, the output unit 115 transmits this judgment result to the other speaker via the communication interface 14 through a network (not shown), thereby notifying the other speaker of whether the utterance estimation target will speak / not speak.

[0058] Then, the processor 11A judges whether or not to end the inference processing routine (step S205). For example, the processor 11A can judge this by judging whether or not an end instruction is given by the user of the PC via the input device 16. When it is judged not to end the inference processing routine, the processor 11A returns to the process of the above-mentioned step S201.

[0059] In this manner, the processor 11A acquires video data of the utterance estimation target person and repeatedly executes utterance estimation of the utterance estimation target person until an end instruction is given.

[0060] As described above in detail, in the first embodiment of the present invention, an utterance estimator A is prepared in advance that has learned classification of whether or not the utterance estimation target will speak immediately after based on the video features of the utterance estimation target, and the feature extraction unit 112 extracts video features for a certain period of time, for example, one second, from video data showing the utterance estimation target, and the utterance estimation unit 113 inputs the extracted video features to the utterance estimator A to obtain the utterance estimation result output by the utterance estimator A. Then, the output unit 115 outputs a determination result of whether or not the utterance estimation target will speak immediately after the certain period based on the utterance estimation result. Therefore, according to the first embodiment, by learning to predict whether or not an utterance will be made from the behavior pattern of the utterance estimation target before the utterance, it is possible to estimate that the utterance estimation target did not speak but was about to speak. Therefore, by designating a speaker who was unable to speak, it is possible to avoid loss of an opportunity to speak.

[0061] In addition, the video features are a fixed number of frame images contained in the fixed period, the speech estimation result is the probability of two classes of speech or not, and the speech estimator A includes a fully connected neural network with the fixed number of input layers and two output layers. Therefore, according to the first embodiment, the frame images contained in the video data can be used as they are as video features for learning, eliminating the need to transform the frame images into a different data format for learning.

[0062] Alternatively, the video features are feature vectors extracted by a trained video feature extractor for each of a fixed number of frame images included in the above-mentioned fixed period, the speech estimation result is a probability of two classes of whether or not to speak, and the speech estimator A includes a fully connected neural network with the above-mentioned fixed number of input layers and two output layers. Therefore, according to the first embodiment, by training using feature vectors extracted from frame images contained in video data, it is possible to reduce the amount of training data required to train a fully connected neural network, thereby shortening the training time.

[0063] In the first embodiment, the apparatus further includes an acquisition unit 111 that acquires the video data from the screen of the Web conference system. Therefore, according to the first embodiment, it is possible to estimate speakers who wanted to speak but missed their chance to speak from among the Web conference participants.

[0064] In this case, the output unit 115 outputs the determination result to the Web conference system. Therefore, according to the first embodiment, it is possible to notify participants viewing the screen of the Web conference of speakers who wanted to speak but missed their chance to speak.

[0065] [Second embodiment] Next, a second embodiment of the present invention will be described. In this embodiment, utterance estimation can be performed only when necessary. In the following description, the same reference symbols as those used in the first embodiment are used for the same parts, and the description thereof will be omitted.

[0066] 8 is a block diagram showing an example of a software configuration of an utterance estimation device according to a second embodiment of the present invention. In the second embodiment, in addition to the configuration of the first embodiment, an utterance detection unit 118 is provided. The utterance detection unit 118 operates during an inference processing routine. Also, the operations of the acquisition unit 111 and the feature extraction unit 112 during the inference processing routine are different from those of the first embodiment.

[0067] That is, during the inference processing routine, the acquisition unit 111 acquires video data X of the person to be estimated as speaking and audio data V of the Web conference from the Web conference application 116 and stores them in the input data temporary storage unit 121 .

[0068] During the inference processing routine, the speech detection unit 118 detects whether any of the participants of the Web conference, i.e., any of the multiple speakers including the speech estimation target, has started speaking a new speech based on the voice data V stored in the input data temporary storage unit 121. In other words, the speech detection unit 118 detects whether a speaker different from the speaker who had been speaking up until that point has started speaking. The speech detection unit 118 can distinguish speakers by applying a technique such as that disclosed in Patent Document 1, for example. Then, when the speech detection unit 118 detects the start of a new speaker's speech, it transmits a voice trigger signal VT to the feature extraction unit 112.

[0069] During the inference processing routine, feature extraction unit 112 does not operate all the time, but starts operating in response to a voice trigger signal VT from speech detection unit 118. Feature extraction unit 112 acquires from input data temporary storage unit 121 video data X stored in input data temporary storage unit 121, video data X from a certain period of time, for example, one second before the time when the voice trigger signal VT is received, and creates a video feature Y for that certain period of time.

[0070] FIG. 9 is a flowchart showing an inference processing routine in the utterance estimation device according to the second embodiment.

[0071] First, the processor 11A operates as the acquisition unit 111 to acquire video data X showing the estimated utterance target and audio data V of multiple speakers including the estimated utterance target, and stores the data in the input data temporary storage unit 121 (step S211). For example, the acquisition unit 111 can acquire the video data X and audio data V in real time from the Web conference application 116. Alternatively, the acquisition unit 111 may acquire recorded data including the video data X and audio data V input from the input device 16 acquired by the input unit 117.

[0072] Next, processor 11A operates as speech detection unit 118 and detects whether or not a new speaker has started speaking from the voice data V stored in input data temporary storage unit 121 (step S212). If a new speaker has not started speaking, that is, if the same speaker who had been speaking until then is still speaking, or if there is silence during which no speaker is speaking, processor 11A proceeds to the process of step S205 described later.

[0073] When it is determined in step S212 that a new speaker has started speaking, processor 11A operates as feature extraction unit 112 and extracts video features Y for a certain period of time immediately preceding the start of speech, in this case, for the immediately preceding one second, from video data X stored in input data temporary storage unit 121 (step S213). That is, feature extraction unit 112 accepts video data X for the immediately preceding one second from input data temporary storage unit 121, and creates video features Y for one second from the video data X. Feature extraction unit 112 transmits the created video features Y to utterance estimation unit 113.

[0074] The subsequent processes in steps S203 to S205 are similar to those in the first embodiment.

[0075] As described above in detail, in the second embodiment of the present invention, the acquisition unit 111 acquires audio data V together with the video data X, and in response to the speech detection unit 118 detecting the start of a new utterance from among a plurality of speakers including the utterance estimation target based on the audio data V, the feature extraction unit 112 extracts a video feature Y from the video data X for a certain period immediately prior to the detection of the utterance. Therefore, according to the second embodiment, utterance estimation can be performed only when any speaker starts speaking.

[0076] [Other embodiments] It should be noted that the present invention is not limited to the above-described embodiment.

[0077] For example, the present invention is applicable not only to Web conferences, but also to situations where there may be multiple speakers, such as actual face-to-face conferences, lectures, classes, etc.

[0078] In the second embodiment, utterance estimation is performed even when the utterance estimation target person starts speaking. However, the utterance detection unit 118 may be configured to apply a technique such as that disclosed in Patent Document 1 to determine whether the speaker who started speaking is the utterance estimation target person or another speaker, so that utterance estimation is performed only when a speaker other than the utterance estimation target person starts speaking.

[0079] Furthermore, the flow of each process described with reference to the flowchart is not limited to the procedures described, and the order of some steps may be changed, some steps may be performed simultaneously in parallel, or the processing content of some steps may be modified.

[0080] The methods described in each embodiment can be stored as a processing program (software means) that can be executed by a calculator (computer) on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The programs stored on the medium side include a setting program that configures the software means (including not only execution programs but also tables and data structures) that the calculator executes within the computer. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium referred to in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0081] In short, this invention is not limited to the above-mentioned embodiment as it is, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components across different embodiments may be appropriately combined. [Explanation of symbols]

[0082] 11 Processing section 11A Processor 11B program memory 111 Acquisition Department 112 Feature Extraction Unit 113 Speech Estimation Unit 114 Parameter Update Section 115 Output section 116 Web conferencing apps 117 Input section 118 Speech detection unit 12 Data Memory 121 Input data temporary storage unit 122 Speech Estimator Memory Unit 13 Input / Output Interface 14 Communication Interface 15 Bus 16 Input Devices 17 Output Devices A. Speech Estimator D1, D2 feature extraction period Td fixed period Tp Speech section information V Audio Data VT Audio Trigger Signal W Utterance Estimation Result X Video data Y Video Features Z utterance label x1 Speech period x2 silent period y1,y2,…,y24 vector

Claims

1. an utterance estimator that has learned in advance a classification as to whether or not a person who is an utterance estimation target will immediately speak based on video characteristics of the person; A feature extraction unit that extracts video features for a certain period of time from video data in which the utterance estimation target person is captured; an utterance estimation unit that inputs the video features extracted by the feature extraction unit to the utterance estimator and obtains an utterance estimation result output by the utterance estimator; an output unit that outputs a determination result of whether or not the utterance estimation target person will make an utterance immediately after the certain period of time based on the utterance estimation result acquired by the utterance estimation unit; Equipped with The video features are feature vectors extracted by a trained video feature extractor for each of a certain number of frame images included in the certain period.

2. The utterance estimation result is a probability of two classes, ie, whether or not an utterance is made, The utterance estimation device according to claim 1 , wherein the utterance estimator includes a fully connected neural network having the fixed number of input layers and two output layers.

3. The utterance estimation device according to claim 1 , further comprising an acquisition unit that acquires the video data from a screen of a Web conference system.

4. The utterance estimation device according to claim 3 , wherein the output unit outputs the determination result to the Web conference system.

5. The acquisition unit acquires audio data together with the video data, the utterance estimation device further includes an utterance detection unit that detects a start of a new utterance from among a plurality of speakers including the utterance estimation target person based on the voice data; 4. The speech estimation device according to claim 3, wherein the feature extraction unit, in response to the speech detection unit detecting a start of the speech, extracts the video features from the video data during the certain period immediately before the detection of the speech.

6. An utterance estimation method in an utterance estimation device that estimates whether an utterance estimation target person will speak, comprising: Extracting image features for a certain period of time from image data including at least the person to be estimated as an utterance; inputting the extracted video features into an utterance estimator that has previously learned classification of whether or not the utterance estimation target person will immediately speak based on the video features of the utterance estimation target person, and acquiring an utterance estimation result output by the utterance estimator; outputting a determination result as to whether or not the utterance estimation target person will make an utterance immediately after the certain period of time based on the acquired utterance estimation result; Including, The speech estimation method, wherein the video features are feature vectors extracted by a trained video feature extractor for each of a certain number of frame images included in the certain period.

7. An utterance estimation program that causes a computer to execute processing by each unit of the utterance estimation device according to any one of claims 1 to 5.

Images