Simultaneous interpretation device, simultaneous interpretation system, simultaneous interpretation processing method, and program

Abstract

To provide a simultaneous interpretation device, a simultaneous interpretation processing method and a program, which realize a simultaneous interruption system performing machine interpretation processing and speaker specification processing in real time.SOLUTION: In a simultaneous interpretation system 1000, a simultaneous interpretation device 100 includes: a segment processing unit 2 for performing high-speed and high-accurate segment processing and acquiring time range data with sentence data by dividing a word string included in data D_words into sentences; a machine interpretation processing unit 4 for performing machine interpretation processing on sentence data of data Ds_src (language (original language) of translation source) outputted from the segment processing unit 2; and a speaker prediction processing unit 3 for predicting a speaker uttered in a period specified by data including time range information based on an input video stream and data D_t_rng including acquired time range information. The simultaneous interpretation device performs machine interpretation processing and speaker specification processing in parallel and outputs data to a display processing device Dev2.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a multi-mode simultaneous interpretation technology, and more particularly to a technology for performing translation processing while identifying speakers in real time using, for example, AV-synchronized audio and video signals. [Background technology]

[0002] In recent years, simultaneous interpretation systems (real-time interpretation systems) have been developed that take a video stream (audio and video signals synchronized with AV) as input, extract audio from the video stream, perform automatic speech recognition and machine translation, and then display the machine translation results (for example, as subtitles) on the video with audio obtained from the input video stream.

[0003] In such a simultaneous interpretation system, the results of machine translation are displayed on top of video with audio obtained from the input video stream, allowing the user watching the video to understand what is being said.

[0004] However, with the above-mentioned simultaneous interpretation system, it can be difficult to understand who is saying what in a scene where multiple people are conversing. In other words, in such a case, with the above-mentioned simultaneous interpretation system, even if the user only looks at the machine translation results displayed on the video with audio (for example, as subtitles), they cannot identify the speaker, which can result in confusion for the user.

[0005] Meanwhile, a conversation recorder has been developed that records audio data from a conference or the like in which multiple people speak, analyzes the recorded audio data, identifies speakers, and makes it possible to easily distinguish between the content of the speech and the speakers (see, for example, Patent Document 1). Specifically, this conversation recorder performs a clustering process on all audio features when recording of the conference or the like is completed, determines the number of people participating in the conversation and the representative audio features of each speaker, compares the audio features of each speaker with the recorded data to distinguish speakers, and displays the content of speech from the same speaker in different colors or display positions, thereby enabling a display that allows each speaker to be distinguished. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 10-198393 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with the above technology, after recording the voice data, it is necessary to analyze the recorded voice data in order to identify the speaker, and speaker identification cannot be performed as a real-time process (a process that guarantees that the time (delay time) from the start to the end of the process falls within a certain time period).

[0008] In view of the above, an object of the present invention is to provide a simultaneous interpretation system that is capable of performing automatic speech recognition processing, machine translation processing, and speaker identification processing in real time. [Means for solving the problem]

[0009] A first invention for solving the above problem is a simultaneous interpretation device including a speech recognition processing unit, a segment processing unit, a speaker prediction processing unit, and a machine translation processing unit.

[0010] The speech recognition processing unit performs speech recognition processing on a video stream including time information, an audio signal, and a video signal (this video stream may be a stream including time information and an audio signal), thereby obtaining word string data, which is data on a word string corresponding to the audio signal and includes time information on when each word in the word string was uttered.

[0011] The segment processing unit performs segment processing on the word string data to obtain sentence data, which is segmented word string data, and also obtains time range data that specifies the time range in which the word string included in the sentence data was uttered.

[0012] The speaker prediction processing unit predicts, based on the video stream and the time range data, the speaker who spoke during the period specified by the time range data.

[0013] The machine translation processor executes machine translation processing on the text data to obtain machine translation processing result data corresponding to the text data.

[0014] In this simultaneous interpretation device, the segment processing unit performs high-speed, high-precision segment processing to acquire sentence data and also acquires data on the time range in which a word string contained in the sentence data was spoken, making it possible to perform machine translation processing and speaker identification processing in real time.In other words, in this simultaneous interpretation device, while the machine translation processing unit performs machine translation processing on the sentence data acquired by high-speed, high-precision segment processing, a process is performed to predict the speaker who spoke during the period identified by the time range data based on the input video stream and the time range data, making it possible to perform machine translation processing and speaker identification processing in real time (processing that guarantees that the delay time falls within a predetermined period).

[0015] A second invention is the first invention, in which the speaker prediction processing unit includes a video clip processing unit, a speaker detection processing unit, a voice encoder, a face encoder, and a speaker identification processing unit.

[0016] The video clip processing unit acquires a clip video stream, which is data for a period specified by the time range data, from the video stream.

[0017] The speaker detection processing unit extracts a face image area of ​​the speaker from a frame image formed by the clip video stream.

[0018] The audio encoder performs audio encoding processing on the audio signal included in the clip video stream to obtain audio embedded expression data, which is embedded expression data corresponding to the audio signal.

[0019] The face encoder performs face encoding processing on image data forming the speaker's face image area, thereby obtaining face embedded expression data that is embedded expression data corresponding to the speaker's face image area.

[0020] The speaker identification processing unit identifies the speaker who uttered the voice reproduced in the voice signal included in the clip video stream based on the voice embedded expression data and the face embedded expression data.

[0021] This simultaneous interpretation device can identify the speaker who uttered the voice reproduced in the voice signal included in the clip video stream based on the voice-related embedded expression data and the face-related embedded expression data. In other words, this simultaneous interpretation device performs speaker identification processing using a small amount of embedded expression data, so it can perform speaker identification processing faster (with less calculation volume) and with higher accuracy.

[0022] A third invention is the second invention, further comprising a data storage unit that stores a speaker identifier that identifies a speaker, as well as embedded voice expression data and embedded face expression data that are linked to the speaker identifier.

[0023] The speaker identification processing unit then performs a best matching process on the embedded voice expression data acquired by the voice encoder and the embedded face expression data acquired by the face encoder, and the embedded voice expression data and embedded face expression data stored in the data storage unit, and if the similarity score indicating the degree of similarity between the two data in the best matching process is higher than a predetermined value, it identifies the speaker identified by the speaker identifier corresponding to the embedded voice expression data and embedded face expression data stored in the data storage unit that were the target of the matching process in the best matching process as the speaker who spoke the voice reproduced in the audio signal included in the clip video stream.

[0024] This allows the simultaneous interpretation device to perform speaker identification processing by referring to the data stored in the data storage unit.

[0025] The degree of similarity between two pieces of data in the best matching process is acquired based on, for example, the cosine similarity or distance information (for example, Euclidean distance) between the two pieces of data.

[0026] A fourth invention is a simultaneous interpretation system comprising a simultaneous interpretation device according to any one of the first to third inventions, and a display processing device that receives speaker identification data, which is data acquired by the simultaneous interpretation device for identifying the speaker who spoke the audio reproduced in the audio signal included in the video stream, and machine translation processing result data corresponding to the text data acquired by the machine translation processing unit of the simultaneous interpretation device, and generates display data that displays the speaker identification data and the machine translation processing result data in a specified image area of ​​a screen displayed on a display device.

[0027] This allows the simultaneous interpretation system to display the machine translation results of the original language spoken by the speaker, along with data identifying the speaker, in a specified image area (the same image area on the display screen), allowing the user to easily recognize "who said what."

[0028] A fifth aspect of the present invention is a simultaneous interpretation processing method comprising a speech recognition processing step, a segment processing step, a speaker prediction processing step, and a machine translation processing step.

[0029] The speech recognition processing step performs speech recognition processing on a video stream including time information, an audio signal, and a video signal (this video stream may be a stream including time information and an audio signal), thereby obtaining word string data, which is data on a word string corresponding to the audio signal and includes time information on when each word in the word string was uttered.

[0030] The segment processing step performs segment processing on the word string data to obtain sentence data, which is segmented word string data, and also obtains time range data that specifies the time range in which the word strings included in the sentence data were uttered.

[0031] The speaker prediction processing step predicts the speaker who spoke during the period specified by the time range data, based on the video stream and the time range data.

[0032] The machine translation processing step executes machine translation processing on the text data to obtain machine translation processing result data corresponding to the text data.

[0033] This makes it possible to realize a simultaneous interpretation processing method that has the same effects as the first aspect of the invention.

[0034] A sixth aspect of the present invention is a program for causing a computer to execute the simultaneous interpretation processing method of the fifth aspect of the present invention.

[0035] This makes it possible to realize a program for causing a computer to execute a simultaneous interpretation processing method that has the same effects as the fifth aspect of the invention. [Effects of the Invention]

[0036] According to the present invention, it is possible to realize a simultaneous interpretation system that is capable of performing automatic speech recognition processing, machine translation processing, and speaker identification processing in real time. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of a simultaneous interpretation system 1000 according to a first embodiment. [Figure 2] 1 is a schematic configuration diagram of a speaker prediction processing unit 3 of a simultaneous interpretation apparatus 100 according to a first embodiment. [Figure 3] 1 is a diagram showing a model of a moving image (moving image with audio) formed from a video stream (one example) input to a simultaneous interpretation device 100. FIG. [Figure 4] 10 is a flowchart of a segment process executed in the simultaneous interpretation system 1000. [Figure 5] 10 is a diagram for explaining segment processing executed in the simultaneous interpretation system 1000. FIG. [Figure 6] FIG. 10 is a diagram for explaining a speaker detection process. [Figure 7] 10 is a flowchart of a speaker prediction process executed in the simultaneous interpretation system 1000. [Figure 8] 4 is a diagram schematically showing a screen when the display data obtained by processing the video stream of FIG. 3 by the simultaneous interpretation system 1000 is displayed on a display device. FIG. [Figure 9] 4 is a diagram schematically showing a screen when the display data obtained by processing the video stream of FIG. 3 by the simultaneous interpretation system 1000 is displayed on a display device. FIG. [Figure 10] 4 is a diagram schematically showing a screen when the display data obtained by processing the video stream of FIG. 3 by the simultaneous interpretation system 1000 is displayed on a display device. FIG. [Figure 11] A diagram showing the CPU bus configuration. DETAILED DESCRIPTION OF THE INVENTION

[0038] [First embodiment] The first embodiment will be described below with reference to the drawings.

[0039] <1.1: Simultaneous interpretation system configuration> FIG. 1 is a schematic diagram of a simultaneous interpretation system 1000 according to the first embodiment.

[0040] FIG. 2 is a schematic diagram showing the configuration of the speaker prediction processing unit 3 of the simultaneous interpretation apparatus 100 according to the first embodiment.

[0041] As shown in FIG. 1, the simultaneous interpretation system 1000 includes a video stream acquisition processing device Dev1, a simultaneous interpretation device 100, and a display processing device Dev2.

[0042] The video stream acquisition processing device Dev1 is a device that acquires a video stream (audio and video signals synchronized with AV). The video stream acquisition processing device Dev1 can be connected to, for example, an audio acquisition device (e.g., a microphone) and an imaging device (e.g., a camera), acquires audio and video signals from the audio acquisition device (e.g., a microphone, a sound collection device) and the imaging device (e.g., a camera), performs AV synchronization processing on the acquired audio and video signals, and acquires a video stream (audio and video signals synchronized with AV). The video stream acquisition processing device Dev1 then outputs the acquired video stream (audio and video signals synchronized with AV) to the simultaneous interpretation device 100 as data D_av. The video stream acquisition processing device Dev1 also acquires a video stream (audio and video signals synchronized with AV) from, for example, an external recording device or an external server (e.g., a streaming server or a video distribution server) via an external network (e.g., the Internet), and outputs the acquired video stream as data D_av to the simultaneous interpretation device 100. In addition, if the audio signal and video signal input to the video stream acquisition processing device Dev1 each contain time information (e.g., a timestamp) and are not AV synchronized, the video stream acquisition processing device Dev1 acquires AV-synchronized audio signals and video signals based on the time information (e.g., the timestamp) contained in the audio signal and video signal, and outputs a video stream containing the audio signal and video signal (AV-synchronized video stream) as data D_av to the simultaneous interpretation device 100 and the display processing device Dev2.

[0043] As shown in FIG. 1, the simultaneous interpretation device 100 includes a speech recognition processing unit 1, a segment processing unit 2, a speaker prediction processing unit 3, and a machine translation processing unit 4.

[0044] The speech recognition processing unit 1 receives data D_av (data of a video stream (a video stream synchronized with AV)) output from the video stream acquisition processing device Dev1. The speech recognition processing unit 1 extracts audio data (audio signal) from the data D_av, ​​performs speech recognition processing on the extracted audio data (audio signal), and acquires a word string (word stream) corresponding to the audio data (audio signal) and time information when each word included in the word string was spoken. The speech recognition processing unit 1 then outputs data including the acquired word string and the time information to the segment processing unit 2 as data D_words.

[0045] The segment processing unit 2 receives data D_words output from the speech recognition processing unit 1. The segment processing unit 2 executes segmentation processing on the data D_words and separates the word strings included in the data D_words into sentences to obtain text data. The segment processing unit 2 then outputs the text data obtained by the segmentation processing (word strings separated into sentences (data on word strings that make up one sentence)) to the machine translation processing unit 4 as data Ds_src.

[0046] Furthermore, the segment processing unit 2 acquires information on the period (time range) during which the sentence was uttered from the time information of the word strings that constitute the sentence data when the segment processing is performed on the data D_words to acquire the sentence data. Then, the segment processing unit 2 outputs the acquired data including the information on the period (time range) to the speaker prediction processing unit 3 as data D_t_rng.

[0047] As shown in FIG. 2, the speaker prediction processing unit 3 includes a video clip processing unit 31, a voice encoder 32, a speaker detection processing unit 33, a face encoder 34, a speaker identification processing unit 35, and a data storage unit DB1.

[0048] The video clip processing unit 31 receives data D_av (AV-synchronized video stream data) output from the video stream acquisition processing device Dev1 and data D_t_rng output from the segment processing unit 2. Based on the data D_t_rng, the video clip processing unit 31 performs clip processing on the video stream data included in the data D_av. Specifically, the video clip processing unit 31 acquires period (time range) information from the data D_t_rng, and acquires video stream data (acquired during the period (time range)) corresponding to the information on the period (time range). The video clip processing unit 31 then outputs the acquired video stream data as data D1_av to the speaker detection processing unit 33. The video clip processing unit 31 also extracts only audio stream data from the acquired video stream data, and outputs the extracted audio stream data to the audio encoder 32 as data D1_a.

[0049] The audio encoder 32 receives data D1_a (audio stream data) output from the video clip processor 31, performs encoding on the data D_a, and acquires embedded expression data corresponding to the input data D_a (audio stream (voice stream)).The audio encoder 32 then outputs the acquired embedded expression data to the speaker identification processor 35 as data D_a_emb.

[0050] The speaker detection processing unit 33 inputs data D1_av (video stream data) output from the video clip processing unit 31, performs speaker detection processing on the data D1_av, detects an image area corresponding to a person speaking in the audio-accompanied video formed by the input data D1_av, and acquires speaker icon data based on the detected image area.The speaker detection processing unit 33 then outputs data including the acquired speaker icon data as data Do_face_icon to the display processing device Dev2.

[0051] In addition, the speaker detection processing unit 33 performs speaker detection processing on the input data D1_av, detects an image area corresponding to the face of the person speaking on the video with audio formed by the input data D1_av, and outputs data including the image signal (image data) forming the detected image area to the face encoder 34 as data D_face.

[0052] The face encoder 34 receives the data D_face output from the speaker detection processor 33, performs encoding on the data D_face, and acquires embedded expression data corresponding to the input data D_face. The face encoder 34 then outputs the acquired embedded expression data as data D_face_emb to the speaker identification processor 35.

[0053] The speaker identification processing unit 35 receives the data D_a_emb (embedded expression data of the voice data) output from the voice encoder 32 and the data D_face_emb (embedded expression data of the face image area data) output from the face encoder 34. The speaker identification processing unit 35 can also perform data read processing or data write processing by outputting a data read command or a data write command to the data storage unit DB1.

[0054] The speaker identification processing unit 35 executes speaker identification processing by referring to the data D_a_emb (embedded expression data of voice data) and the data D_face_emb (embedded expression data of face image area data) and the data stored in the data storage unit DB1, thereby identifying the speaker (details will be described later).The speaker identification processing unit 35 then outputs the data of the speaker identified by the above processing (for example, tag data for identifying the speaker) as data Do_spk_tag to the display processing device Dev2.

[0055] The data Do_face_icon output from the speaker detection processing unit 33 to the display processing device Dev2 and the data Do_spk_tag output from the speaker identification processing unit 35 to the display processing device Dev2 are collectively referred to as data Do_spk.

[0056] The data storage unit DB1 is a storage unit that can store and hold data, and is realized by, for example, a database. The data storage unit DB1 reads out the stored data based on a data read command from the speaker identification processing unit 35, and outputs the read data to the speaker identification processing unit 35. Furthermore, the data storage unit DB1 stores the data output from the speaker identification processing unit 35 in a predetermined storage area based on a data write command from the speaker identification processing unit 35. The data storage unit DB1 may be installed outside the simultaneous interpretation apparatus 100.

[0057] The machine translation processing unit 4 receives the text data Ds_src (source language (original language)) output from the segment processing unit 2, and performs machine translation processing on the input source language text data Ds_src to obtain word string data (translation result data) in the translation language (target language) corresponding to the source language text data Ds_src.The machine translation processing unit 4 then outputs the obtained translation result data (word string data in the translation language) as data Do_MT to the display processing device Dev2.

[0058] The display processing device Dev2 receives as input the data D_av output from the video stream acquisition processing device Dev1, and the data Do_MT (machine translation result data) and Do_spk (speaker identification data) output from the simultaneous interpretation device 100. The display processing device Dev2 generates data to be displayed on a display device (not shown) based on the video stream data D_av, ​​the machine translation result data Do_MT, and the speaker identification data Do_spk.

[0059] <1.2: Operation of the simultaneous interpretation system> The operation of the simultaneous interpretation system 1000 configured as above will now be described.

[0060] FIG. 3 is a diagram showing a model of a moving image (moving image with audio) formed from a video stream (one example) input to the simultaneous interpretation device 100. As shown in FIG.

[0061] FIG. 4 is a flowchart of the segment processing executed in the simultaneous interpretation system 1000.

[0062] FIG. 5 is a diagram for explaining the segment processing executed in the simultaneous interpretation system 1000. As shown in FIG.

[0063] FIG. 6 is a diagram for explaining the speaker detection process.

[0064] FIG. 7 is a flowchart of the speaker prediction process executed in the simultaneous interpretation system 1000.

[0065] The video stream acquisition processing device Dev1 acquires a video stream (audio and video signals synchronized with AV). For ease of explanation, it is assumed here that the video stream (audio and video signals synchronized with AV) that forms the moving image (moving image with audio) shown in Fig. 3 is acquired by the video stream acquisition processing device Dev1, and the following describes how the video stream is processed in the simultaneous interpretation system 1000.

[0066] In Fig. 3, for ease of explanation, the content of the spoken voice is displayed as text in a speech bubble, but in an actual video (video with audio), the speech bubble (and the text therein) does not exist on the video (image). Also, Fig. 3 shows some frame images (frame images extracted at a predetermined time) included in the video formed by the video stream in chronological order, but does not show all frame images included in the video formed by the video stream. And in Fig. 3, for ease of explanation, the voice uttered at a predetermined time (period) is displayed as text in a speech bubble in the frame image displayed in Fig. 3.

[0067] Furthermore, the video with audio (video with audio in Figure 3) formed by the video stream acquired by the video stream acquisition processing device Dev1 is a video of a scene in which a man and a woman are having a conversation, and the man and woman are speaking in English (the source language (original language)).

[0068] If the audio and video signals constituting the video stream to be processed are not AV synchronized, the video stream acquisition processing device Dev1 performs AV synchronization processing based on the time information (e.g., timestamp) of the audio signal and the time information (e.g., timestamp) of the video signal to acquire AV-synchronized audio and video signals.The video stream acquisition processing device Dev1 then outputs the acquired video stream (AV-synchronized audio and video signals) (video stream forming the audio-accompanying moving image in FIG. 3) as data D_av to the simultaneous interpretation device 100 and the display processing device Dev2.

[0069] The speech recognition processing unit 1 of the simultaneous interpretation device 100 inputs data D_av (data of a video stream (a video stream synchronized with AV)) output from the video stream acquisition processing device Dev1. The speech recognition processing unit 1 extracts audio data (audio signals) from the input data D_av, ​​performs speech recognition processing on the extracted audio data (audio signals), and acquires a word string (word stream) corresponding to the audio data (audio signal) and time information (timestamps) at which each word included in the word string was spoken.

[0070] The speech recognition processing unit 1 acquires data (data of a word string with a time stamp) in which each word is paired with time information (time stamp) at which the word was uttered, for example, as follows: "I'm" (0.5s) "Smith" (0.9s) "nice" (1.1s) "to" (1.3s) "Meet" (1.4s) ... In the above, the character string in " " is the word data, and the number in () is the time information (timestamp) shown in seconds.

[0071] Then, the speech recognition processing unit 1 outputs data including the acquired word string and the time information (data of the word string with time stamp) to the segment processing unit 2 as data D_words.

[0072] The segmentation processing unit 2 executes segmentation processing on the data D_words output from the speech recognition processing unit 1. Specifically, the segmentation processing unit 2 performs the following processing. The segmentation processing will be described with reference to the flowchart in FIG. 4 and the explanatory diagram in FIG. 5.

[0073] (Step S11): In step S11, the segmentation processing unit 2 sets a variable k indicating the order of the character string to an initial value (k=0).

[0074] (Step S12): In step S12, the segmentation processing unit 2 acquires features for segmentation. Specifically, the segmentation processing unit 2 converts time information (timestamps) attached to words into time t Then, the time t n (n: natural number) feature quantities in t 0,feat t 1,···,feat t n-1 The above feature amounts are data required to execute segment processing and obtain a segment evaluation value (segment score (e.g., a value indicating the probability that a word string is separated (a value indicating the probability that a sentence is separated (end of a sentence)))). In this embodiment, two feature amounts (i.e., n=2) are used, the first feature amount being words (data indicating words), and the second feature amount being the duration of pauses (silence periods) after each word.

[0075] (Step S13): In step S13, the segment processing unit 2 calculates a segment evaluation value (segment score) score based on the feature amount for segment processing. seg Specifically, the segment processing unit 2 executes a process corresponding to the following formula to obtain the segment evaluation value (segment score) of the kth word (a value indicating the possibility that the kth (k: natural number) word is followed by the end of a sentence (a delimiter of a word string)) score seg Obtain (k).

number

number

[0076] (Step S14): In step S14, the segment processing unit 2 calculates the segment evaluation value (segment score) score acquired in step S12. seg A process is performed to compare (k) with a threshold value th1. As a result of the comparison process, score seg If (k)>th1, the segmentation processing unit 2 advances the process to step S16. seg If (k)>th1 is not true, the segment processing unit 2 advances the process to step S15.

[0077] (Step S15): In step S15, the segmentation processing unit 2 increments the value of the variable k by 1, and returns the process to step S12. Then, the processes of steps S12 to S14 are executed in the same manner as above.

[0078] (Steps S16 and S17): In step S16, the segment processing unit 2 acquires text data (data on a string of words that make up a sentence) from the first word when segment processing begins to the word that is determined to be the segment boundary thereafter, and also acquires data on the time range of the time information (timestamp) of the word string of the text data.

[0079] Then, in step S17, the segment processing unit 2 outputs the sentence data acquired as described above to the machine translation processing unit 4 as data Ds_src, and outputs the data for the time range acquired as described above to the speaker prediction processing unit 3 as data D_t_rng.

[0080] A specific example of segment processing will now be described with reference to FIG.

[0081] In Fig. 5, the top diagram is a diagram that schematically shows the input data, processing, and output data in segment processing, and the middle and bottom diagrams in Fig. 5 are diagrams that schematically show the input data, processing, and output data in segment processing when actual data is input. Note that in Fig. 5, the first feature used in segment processing is word data, and the second feature is the duration (in seconds) of a pause (silent period) after the word to be processed.

[0082] In the upper diagram of Figure 5, time k Feature quantity in k 0,···,feat k n-1 are acquired in chronological order by the segment processing unit 2, and the segment evaluation value score seg k (=score seg (k)) is obtained, and the obtained segment evaluation value score seg k is compared with the threshold th1, and the score segIf (k)>th1, the string (word string) sentence consisting of the 0th to kth words and the data of the time range in which the string (word string) consisting of the 0th to kth words is spoken, that is, the start time of the utterance of the string start and the end time of the utterance of the string. end 10A and 10B are diagrams showing how the above information is acquired.

[0083] As shown in the middle diagram of Figure 5, the word "Smith" was entered at time 0.9 seconds (time 1 =0.9s) (time is obtained from the time information (timestamp) attached to the word), the segment evaluation value obtained by segment processing is seg k (=score seg The value of (k), k=1) is "0.65", which is larger than the threshold value th1. It is assumed that the threshold value th1 is set to "0.6". Therefore, the segment processing unit 2 determines that there is a segment boundary (end of sentence) after the first word "Smith" (time: 0.9 s), and acquires the word string from the 0th word "I'm" (time: 0.5 s) to the first word "Smith" (time: 0.9 s) as sentence data (data of the segmented word string) Ds_src.

[0084] The segment processing unit 2 also determines the time range in which the acquired sentence data was spoken as the period from "time: 0.5 s" acquired from the time information (timestamp) of the 0th word "I'm" to "time: 1.1 s" acquired from the time information (timestamp) of the word (the word "nice" in the middle diagram of FIG. 5) that follows the 1st word "Smith" (time: 0.9 s). In other words, the segment processing unit 2 time start =0.5s (time information of the 0th word) time end =1.1s (time information of the word following the first word (second word)) and obtains data including the information as time range data D_t_rng.

[0085] Then, the segment processing unit 2 outputs the data Ds_src acquired as described above to the machine translation processing unit 4, and outputs the data D_t_rng acquired as described above to the speaker prediction processing unit 3.

[0086] Then, the segment processing unit 2 performs the same processing as above on the word string starting from the next word determined to be a segment boundary (in the case of Figure 5, "nice" at time: 1.1s) (performing segment processing with the word "nice" at time: 1.1s as the word k=0), thereby continuing to execute segment processing.

[0087] Data Ds_src including the text data (word strings separated by sentences (data of word strings that make up one sentence)) obtained by the segment processing as described above is output from the segment processing unit 2 to the machine translation processing unit 4. Then, data D_t_rng of the time range which is information on the speech period of the text data (word strings separated by sentences (data of word strings that make up one sentence)) obtained by the segment processing as described above is output from the segment processing unit 2 to the speaker prediction processing unit 3.

[0088] The video clip processing unit 31 of the speaker prediction processing unit 3 receives as input the data D_av (AV-synchronized video stream data) output from the video stream acquisition processing device Dev1 and the data D_t_rng output from the segment processing unit 2. The video clip processing unit 31 performs clip processing on the video stream data included in the data D_av based on the data D_t_rng. Specifically, the video clip processing unit 31 acquires information on the period (time range) from the data D_t_rng, and acquires video stream data (acquired during the period (time range)) corresponding to the information on the period (time range). For example, in the case of FIG. 5, when the data Ds_src is "I'm Smith", the time range data D_t_rng=(time start ,time end)=(0.5 s, 1.1 s), the video clip processing unit 31 acquires data for the period from time 0.5 s to time 1.1 s from the data D_av, ​​and sets the acquired data as data D1_av.

[0089] The video clip processing unit 31 then outputs the acquired video stream data D1_av to the speaker detection processing unit 33. The video clip processing unit 31 also extracts only the audio stream data from the acquired video stream data D1_av, and outputs the extracted audio stream data to the audio encoder 32 as data D1_a.

[0090] The audio encoder 32 inputs data D1_a (audio stream data) output from the video clip processing unit 31, performs encoding processing on the data D_a (processing to obtain embedded expression data corresponding to the audio stream from the audio stream), and obtains embedded expression data corresponding to the input data D_a (audio stream (voice stream)).The audio encoder 32 then outputs the obtained embedded expression data to the speaker identification processing unit 35 as data D_a_emb.

[0091] The speaker detection processor 33 executes speaker detection processing on the data D1_av (video stream data) D1_av output from the video clip processor 31, detects an image area corresponding to a person speaking on the video with audio formed from the input data D1_av, and acquires speaker icon data based on the detected image area. Note that this speaker detection processing can be realized, for example, by the technology disclosed in the following document B. (Reference B): Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, and Caroline Pantofaru, "AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection" 2019. For example, in the case shown in Figure 6, the speaker detection processing unit 33 acquires image areas R1_det_spk, R2_det_spk, and R3_det_spk corresponding to the person speaking on the audio-accompanied video formed by data D1_av, and acquires reduced images of the image areas as speaker icon data.

[0092] Then, the speaker detection processing unit 33 outputs data including the acquired speaker icon data as data Do_face_icon to the display processing device Dev2.

[0093] In addition, the speaker detection processing unit 33 performs speaker detection processing on the input data D1_av, detects an image area corresponding to the face of the person speaking on the video with audio formed by the input data D1_av, and outputs data including the image signal (image data) forming the detected image area to the face encoder 34 as data D_face.

[0094] The face encoder 34 receives the data D_face output from the speaker detection processor 33, performs encoding on the data D_face (processing for acquiring embedded expression data corresponding to image data from image data (image signals) forming an image area corresponding to a face), and acquires embedded expression data corresponding to the input data D_face.The face encoder 34 then outputs the acquired embedded expression data to the speaker identification processor 35 as data D_face_emb.

[0095] The speaker identification processing unit 35 receives the data D_a_emb (embedded expression data of voice data) output from the voice encoder 32 and the data D_face_emb (embedded expression data of face image area data) output from the face encoder 34.

[0096] The speaker identification processing unit 35 executes speaker identification processing by referring to the data D_a_emb (embedded expression data of voice data) and data D_face_emb (embedded expression data of face image area data) and the data stored in the data storage unit DB1, thereby identifying the speaker. It is assumed that the data storage unit DB1 stores the embedded expression data of face image area data and the embedded expression data of voice data for each speaker, and the data is read out by the speaker identification processing unit 35. It is assumed that the embedded expression data of the face image area data of the speaker with ID=x (referred to as "speaker x") is stored as v f x Let the embedded representation data of the speech data of the speaker with ID=x be v a x It is written as follows.

[0097] The specific processing will be described with reference to the flowchart of FIG.

[0098] (Step S21): In step S21, a search process for best matching data is executed. Specifically, the following process is executed.

[0099] The speaker identification processing unit 35 performs processing corresponding to the following formula to identify the ID=x' of the speaker having the ID that is the best matching data from among the data stored in the data storage unit DB1.

number

number

[0100] (Step S24): In step S24, the speaker identification processing unit 35 identifies the person (speaker) speaking in the video with audio formed by the data D1_av to be processed (video stream clipped within a time range) as the speaker with ID=x'.

[0101] (Step S25, Step S26): In step S25, the speaker identification processing unit 35 determines that the data of the person (speaker) speaking in the video with audio formed by the data D1_av to be processed (video stream clipped within a time range) is not the data of a speaker stored in the data storage unit DB1.

[0102] Then, in step S26, the speaker identification processing unit 35 determines that the person (speaker) speaking in the video with audio formed by the data D1_av to be processed (the video stream clipped within the time range) is a new speaker, and sets the ID of the speaker to a new ID not stored in the data storage unit DB1 (for example, if the IDs of the speakers stored in the data storage unit DB1 are 1≦ID≦M, the ID of the speaker is set to "M+1").The speaker identification processing unit 35 then pairs the ID (the ID of the new speaker) with the embedded expression data of the face image region data and the embedded expression data of the audio data of the speaker of the ID (as data linked by the ID), and stores them in the data storage unit DB1.

[0103] (Step S27): In step S27, the speaker identification processing unit 35 acquires tag data of the identified speaker (for example, data of a character string that identifies the speaker), and outputs the tag data as data Do_spk_tag to the display processing device Dev2.

[0104] The machine translation processing unit 4 receives the text data Ds_src (source language (original language)) output from the segment processing unit 2, and performs machine translation processing on the input source language text data Ds_src to obtain word string data (translation result data) in the translation language (target language) corresponding to the source language text data Ds_src.The machine translation processing unit 4 then outputs the obtained translation result data (word string data in the translation language) as data Do_MT to the display processing device Dev2.

[0105] The display processing device Dev2 receives as input the data D_av output from the video stream acquisition processing device Dev1, and the data Do_MT (machine translation result data) and Do_spk (speaker identification data) output from the simultaneous interpretation device 100. The display processing device Dev2 generates data to be displayed on a display device (not shown) based on the video stream data D_av, ​​the machine translation result data Do_MT, and the speaker identification data Do_spk.

[0106] Here, specific examples of display data generated by the display processing device Dev2 will be described with reference to FIGS.

[0107] 8 to 10 are diagrams showing a screen when the display data obtained by processing the video stream of FIG. 3 by the simultaneous interpretation system 1000 is displayed on a display device. In FIGS. 8 to 10, area Disp1 is the entire display area, area Disp11 is an area for displaying audio-accompanying video (audio-accompanying video formed by the video stream input to the simultaneous interpretation apparatus 100 and the display processing device Dev2), and area Disp12 is an area for displaying translation result data (word string data in the translation target language (Japanese in the case of FIGS. 8 to 10)) obtained by the machine translation processing unit 4. Also in FIGS. 8 to 10, area Disp13 is an area for displaying tag data for identifying speakers obtained by the simultaneous interpretation apparatus 100 (data displayed as "spk0" and "spk1" in FIGS. 8 to 10), speaker icon data (icon data consisting of an image of the speaker's face), and translation result data.

[0108] <In the case of Figure 8> FIG. 8 shows a scene in which a man utters the English phrase "I'm Smith." The segment processing unit 2 of the simultaneous interpretation apparatus 100 executes segment processing, acquires English sentence data "I'm Smith" as data Ds_src, and outputs the sentence data to the translation processing unit 4. At the same time, the segment processing unit 2 acquires data D_t_rng covering the time range in which the man uttered the English phrase "I'm Smith," and outputs the data D_t_rng to the speaker prediction processing unit 3. The speaker prediction processing unit 3 then identifies the facial area of ​​the man who spoke, acquires icon data for the man, and acquires tag data for the man (data shown as "spk0" in FIG. 8). At this time, the simultaneous interpretation apparatus 100 executes the following processing. That is, the speaker prediction processing unit 3 of the simultaneous interpretation apparatus 100 acquires audio contained in the video stream for the time range indicated by data D_t_rng during which a man spoke the English phrase "I'm Smith," acquires embedded expression data D_a_emb for the audio, acquires image data D_face of the speaking person's face region from the video stream for the time range, and acquires embedded expression data D_face_emb for the image data.The speaker prediction processing unit 3 then performs a best-matching process between the acquired embedded expression data D_a_emb for the audio and the embedded expression data D_face_emb for the face image region data and the data stored in the data storage unit DB1 (embedded expression data for the audio and embedded expression data for the face image region data), and if the score of the matching data exceeds a predetermined threshold, determines that the speaker having the best-matched speaker ID is the same as the speaker (male) who spoke "I'm Smith," and outputs tag data for the speaker ID to the display processing device Dev2. On the other hand, if the score of the matching data does not exceed a predetermined threshold, the speaker prediction processing unit 3 determines that the data of the speaker (male) who uttered "I'm Smith" does not exist in the data storage unit DB1, sets a new ID for the speaker, and further stores the data of the speaker (embedded expression data of the voice and embedded expression data of the facial image area data) in the data storage unit DB1.

[0109] The tag data acquired as described above is output from the speaker prediction processing unit 3 to the display processing device Dev2, and is displayed in the area Disp13 by the display processing device Dev2 (in the case of FIG. 8, the tag data "spk0" is displayed).

[0110] Furthermore, the icon data (icon of a male face region image) acquired by the speaker prediction processing unit 3 is output to the display processing device Dev2, and the icon data is displayed in the region Disp13 by the display processing device Dev2.

[0111] Furthermore, the machine translation result data Do_MT obtained by the machine translation processing unit 4 is output to the display processing device Dev2, and the display processing device Dev2 displays the machine translation result data (in the case of Figure 8, "I am Smith." (a string of words in the target language (Japanese))) in area Disp13 and area Disp12 (area for displaying subtitles).

[0112] <In the case of Figure 9> FIG. 9 shows a scene in which a man utters the English phrase "Nice to meet you." The segment processing unit 2 of the simultaneous interpretation apparatus 100 executes segment processing, acquires English sentence data "Nice to meet you" as data Ds_src, and outputs the sentence data to the translation processing unit 4 (the segment processing described above determines that the segment boundary is after the word "you"). At the same time, the segment processing unit 2 acquires data D_t_rng covering the time range in which the man uttered the English phrase "Nice to meet you," and outputs the data D_t_rng to the speaker prediction processing unit 3. The speaker prediction processing unit 3 then identifies the facial area of ​​the man who spoke, acquires icon data for the man, and acquires tag data for the man (data shown as "spk0" in FIG. 9). At this time, the simultaneous interpretation apparatus 100 executes the following processing. That is, the speaker prediction processing unit 3 of the simultaneous interpretation apparatus 100 acquires audio included in the video stream during the time range indicated by data D_t_rng during which a man uttered the English phrase "Nice to meet you," acquires embedded expression data D_a_emb for the audio, acquires image data D_face of the speaking person's face region from the video stream during the time range, and acquires embedded expression data D_face_emb for the image data. The speaker prediction processing unit 3 then performs a best-matching process between the acquired embedded expression data D_a_emb for the audio and the embedded expression data D_face_emb for the face image region data and the data stored in the data storage unit DB1 (embedded expression data for the audio and embedded expression data for the face image region data), and if the score of the matching data exceeds a predetermined threshold, determines that the speaker having the best-matched speaker ID is the same as the speaker (male) who uttered "Nice to meet you," and outputs tag data for the speaker ID to the display processing device Dev2. In the case of Figure 9, since the data of the above speaker is stored in the data storage unit DB1, the score of the matching data exceeds a predetermined threshold, and as a result, the speaker prediction processing unit 3 determines that the speaker (male) who uttered "Nice to meet you" is the speaker corresponding to the tag data "spk0".

[0113] The tag data acquired as described above is then output from the speaker prediction processing unit 3 to the display processing device Dev2, and is displayed in the area Disp13 by the display processing device Dev2 (in the case of FIG. 9, the tag data "spk0" is displayed).

[0114] Furthermore, the icon data (icon of a male face region image) acquired by the speaker prediction processing unit 3 is output to the display processing device Dev2, and the icon data is displayed in the region Disp13 by the display processing device Dev2.

[0115] Furthermore, the machine translation result data Do_MT acquired by the machine translation processing unit 4 is output to the display processing device Dev2, and the display processing device Dev2 displays the machine translation result data (in the case of Figure 9, "Nice to meet you." (a string of words in the target language (Japanese))) in area Disp13 and area Disp12 (area for displaying subtitles).

[0116] <In the case of Figure 10> FIG. 10 shows a scene in which a woman utters the English phrase "Nice to meet you too, Mr. Smith." The segment processing unit 2 of the simultaneous interpretation apparatus 100 executes segment processing, acquires English sentence data "Nice to meet you too, Mr. Smith" as data Ds_src, and outputs the sentence data to the translation processing unit 4. At the same time, the segment processing unit 2 acquires data D_t_rng covering the time range in which the woman uttered the English phrase "Nice to meet you too, Mr. Smith," and outputs the data D_t_rng to the speaker prediction processing unit 3. The speaker prediction processing unit 3 then identifies the face area of ​​the woman who spoke, acquires icon data for the woman, and acquires tag data for the woman (data shown as "spk1" in FIG. 10). At this time, the simultaneous interpretation apparatus 100 executes the following processing. That is, the speaker prediction processing unit 3 of the simultaneous interpretation apparatus 100 acquires the audio included in the video stream during the time range indicated by the data D_t_rng during which the woman uttered the English phrase "Nice to meet you too, Mr. Smith," acquires the embedded expression data D_a_emb for the audio, acquires image data D_face of the speaking person's face region from the video stream during the time range, and acquires the embedded expression data D_face_emb for the image data. The speaker prediction processing unit 3 then performs a best-matching process between the acquired embedded expression data D_a_emb for the audio and the embedded expression data D_face_emb for the face image region data and the data stored in the data storage unit DB1 (the embedded expression data for the audio and the embedded expression data for the face image region data). If the score of the matching data exceeds a predetermined threshold, the speaker having the best-matched speaker ID is determined to be the same speaker (female) who uttered "Nice to meet you too, Mr. Smith," and outputs the tag data for the speaker ID to the display processing device Dev2.On the other hand, if the score of the matching data does not exceed a predetermined threshold, the speaker prediction processing unit 3 determines that the data of the speaker (female) who uttered "Nice to meet you too, Mr. Smith" does not exist in the data storage unit DB1, sets a new ID for the speaker, and further stores the speaker's data (embedded expression data of the voice and embedded expression data of the facial image area data) in the data storage unit DB1.

[0117] The tag data acquired as described above is output from the speaker prediction processing unit 3 to the display processing device Dev2, and is displayed in the area Disp13 by the display processing device Dev2 (in the case of FIG. 10, the tag data "spk1" is displayed).

[0118] Furthermore, the icon data (icon of the female face region image) acquired by the speaker prediction processing unit 3 is output to the display processing device Dev2, and the icon data is displayed in the region Disp13 by the display processing device Dev2.

[0119] Furthermore, the machine translation result data Do_MT acquired by the machine translation processing unit 4 is output to the display processing device Dev2, and the display processing device Dev2 displays the machine translation result data (in the case of Figure 10, "Nice to meet you, Mr. Smith." (a string of words in the target language (Japanese))) in area Disp13 and area Disp12 (area for displaying subtitles).

[0120] In this way, the simultaneous interpretation system 1000 can display the results of machine translation of the original language spoken by the speaker in area Disp13, along with tag data and icon data that identify the speaker, so that the user can easily recognize "who said what."

[0121] Furthermore, in the simultaneous interpretation system 1000, the segment processing unit 2 of the simultaneous interpretation device 100 performs high-speed, high-precision segment processing to acquire sentence data and also acquire data on the time range in which a word string included in the sentence data was spoken, thereby enabling machine translation processing and speaker identification processing to be performed in real time. That is, in the simultaneous interpretation system 1000, while the machine translation processing unit 4 performs machine translation processing on the sentence data acquired by high-speed, high-precision segment processing, the speaker prediction processing unit 3 performs speaker identification processing on the input video stream using data (stream) clipped within the time range, making it possible to perform machine translation processing and speaker identification processing in real time (processing guaranteed to fall within a predetermined delay time).

[0122] [Other embodiments] Each functional unit of the simultaneous interpretation system described in the above embodiment may be realized by one device (system) or by multiple devices.

[0123] Furthermore, in the above embodiment, a case has been described in which data D_av (data of a video stream (a video stream synchronized with AV)) output from the video stream acquisition processing device Dev1 is input to the speech recognition processing unit 1 of the simultaneous interpretation apparatus 100, and the speech recognition processing unit 1 extracts audio data (audio signals) from the data D_av and performs speech recognition processing on the extracted audio data (audio signals), but the present invention is not limited to this. For example, audio data (audio signals) to which time information has been added may be input to the speech recognition processing unit 1 of the simultaneous interpretation apparatus 100, and the speech recognition processing unit 1 may perform speech recognition processing on the audio data to obtain a word string (word stream) corresponding to the audio data (audio signal) and time information when each word included in the word string was spoken. In other words, instead of the speech recognition processing unit 1 performing the process of extracting the audio signal and time information from the data D_av (data of the video stream (video stream synchronized with AV)) (data (signal) including time information, video signal, and audio signal) input to the simultaneous interpretation apparatus 100, for example, the audio signal with time information added may be input from the video stream acquisition processing device Dev1 to the speech recognition processing unit 1 of the simultaneous interpretation apparatus 100. Note that even in this case, the data D_av (data of the video stream (video stream synchronized with AV)) (data (signal) including time information, video signal, and audio signal) is input to the speaker prediction processing unit 3 of the simultaneous interpretation apparatus 100.

[0124] In addition, although the above embodiment has been described with reference to the case where the input language is English, the input language is not limited to English and may be other languages. In other words, in the simultaneous interpretation system of the above embodiment, the source language and the target language may be any language.

[0125] In the above embodiment, the speaker identification processing unit 35 uses cosine similarity to execute processing equivalent to (Equation 3) to identify the ID=x' of the speaker having the ID that is the best matching data from the data stored in the data storage unit DB1, but the present invention is not limited to this. For example, the speaker identification processing unit 35 uses distance information (e.g., Euclidean distance) to determine {d(v f , v f x )+d(v f , v f x )} is found as x', and the ID=x' of the speaker having the ID that is the best matching data may be identified from the data stored in the data storage unit DB1. Note that d(v1, v2) is a function that obtains distance information (e.g., Euclidean distance) between data v1 and v2.

[0126] In the simultaneous interpretation system 1000 described in the above embodiment, each block may be individually implemented as a single chip using a semiconductor device such as an LSI, or some or all of the blocks may be integrated into a single chip.

[0127] Although we refer to it as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0128] Furthermore, the method of integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within LSI to be reconfigured, may also be used.

[0129] Furthermore, part or all of the processing of each functional block in each of the above embodiments may be realized by a program. And part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Furthermore, the programs for performing each processing are stored in a storage device such as a hard disk or ROM, and are executed in the ROM or read out to the RAM.

[0130] Each process in the above-described embodiments may be realized by hardware, software (including cases where it is realized together with an OS (operating system), middleware, or a predetermined library), or may be realized by a combination of software and hardware.

[0131] For example, when each functional unit in the above embodiment is realized by software, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 11 (for example, a hardware configuration in which a CPU, GPU, ROM, RAM, input unit, output unit, communication unit, memory unit (for example, a memory unit realized by an HDD, SSD, etc.), an external media drive, etc. are connected via a bus).

[0132] Furthermore, when each functional unit of the above embodiment is realized by software, the software may be realized using a single computer having the hardware configuration shown in Figure 11, or may be realized by distributed processing using multiple computers.

[0133] The execution order of the processing method in the above embodiment is not necessarily limited to that described in the above embodiment, and the execution order can be changed within the scope of the gist of the invention. Furthermore, in the processing method in the above embodiment, some steps may be executed in parallel with other steps within the scope of the gist of the invention.

[0134] The scope of the present invention includes a computer program for causing a computer to execute the above-described method, and a computer-readable recording medium having the program recorded thereon. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

[0135] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0136] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0137] 1000 simultaneous interpretation system 100 Simultaneous interpretation equipment 1. Speech recognition processing section 2 Segment processing section 3. Speaker prediction processing section 31 Video clip processing section 32 Audio Encoder 33 Speaker detection processing unit 34 Facial Encoder 35 Speaker identification processing unit DB1 data store 4 Machine translation processing section Dev2 display processing device

Claims

1. a speech recognition processing unit that performs speech recognition processing on a video stream including time information, an audio signal, and a video signal to obtain word sequence data that is data of a word sequence corresponding to the audio signal, the data including time information when each word in the word sequence was uttered; a segment processing unit that performs segment processing on the word string data to obtain sentence data that is segmented word string data, and that obtains time range data that specifies a time range in which the word string included in the sentence data was uttered; a speaker prediction processing unit that predicts a speaker who has spoken during a period specified by the time range data, based on the video stream and the time range data; a machine translation processing unit that executes a machine translation process on the text data to obtain machine translation process result data corresponding to the text data; A simultaneous interpretation device comprising:

2. The speaker prediction processing unit a video clip processing unit that acquires a clip video stream, which is data for a period specified by the time range data, from the video stream; a speaker detection processing unit that extracts a face image area of ​​a speaker from a frame image formed by the clip video stream; an audio encoder that performs an audio encoding process on an audio signal included in the clip video stream to obtain audio embedded expression data that is embedded expression data corresponding to the audio signal; a face encoder that performs a face encoding process on image data that forms the face image area of ​​the speaker to obtain face embedded expression data that is embedded expression data corresponding to the face image area of ​​the speaker; a speaker identification processing unit that identifies a speaker who uttered a voice reproduced in an audio signal included in the clip video stream based on the voice embedded expression data and the face embedded expression data; 2. The simultaneous interpretation apparatus according to claim 1, comprising:

3. a data storage unit that stores a speaker identifier for identifying a speaker, the embedded voice expression data, and the embedded face expression data associated with the speaker identifier; The speaker identification processing unit a best matching process is performed on the embedded voice expression data acquired by the voice encoder and the embedded face expression data acquired by the face encoder, and the embedded voice expression data and the embedded face expression data stored in the data storage unit, and if a similarity score indicating the degree of similarity between the two data in the best matching process is higher than a predetermined value, a speaker identified by a speaker identifier corresponding to the embedded voice expression data and the embedded face expression data stored in the data storage unit that were the target of the matching process in the best matching process is identified as the speaker who spoke the voice reproduced in the voice signal included in the clip video stream; 3. The simultaneous interpretation apparatus according to claim 2.

4. A simultaneous interpretation apparatus according to any one of claims 1 to 3; a display processing device that receives speaker identification data, which is data for identifying the speaker who spoke the audio reproduced in the audio signal included in the video stream and is acquired by the simultaneous interpretation device, and the machine translation processing result data corresponding to the text data, which is acquired by the machine translation processing unit of the simultaneous interpretation device, and generates display data that displays the speaker identification data and the machine translation processing result data in a predetermined image area on a screen displayed on a display device; A simultaneous interpretation system equipped with:

5. a speech recognition processing step of performing speech recognition processing on a video stream including time information, an audio signal, and a video signal to obtain word sequence data, the word sequence data being data of a word sequence corresponding to the audio signal and including time information when each word in the word sequence was uttered; a segment processing step of performing segment processing on the word string data to obtain sentence data that is segmented word string data, and obtaining time range data that specifies a time range in which the word string included in the sentence data was uttered; a speaker prediction processing step of predicting a speaker who spoke during a period identified by the time range data based on the video stream and the time range data; a machine translation processing step of executing a machine translation process on the text data to obtain machine translation processing result data corresponding to the text data; A simultaneous interpretation processing method comprising:

6. A program for causing a computer to execute the simultaneous interpretation processing method according to claim 5.

Images