Translation display device

The translation display device synchronizes video and subtitle timing by extracting and converting audio data for real-time translation display, addressing the discrepancy in existing technologies.

JP7591087B2Active Publication Date: 2024-11-27NTT DOCOMO INC
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2023066268
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-04-14
Publication Date
2024-11-27
Estimated Expiration
2043-04-14

AI Technical Summary

Technical Problem

Existing technologies display subtitles in video conversations after the actual utterance, causing a discrepancy between the video and the translation timing.

Method used

A translation display device that acquires content data, extracts audio, converts it into character data, and synchronizes the video and translation results for simultaneous display, using a communication device to transmit data to an external translation server for real-time synchronization.

Benefits of technology

Reduces the time lag between video utterance and subtitle display, ensuring synchronized timing of video and translation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007591087000001
    Figure 0007591087000001
  • Figure 0007591087000002
    Figure 0007591087000002
  • Figure 0007591087000003
    Figure 0007591087000003
Patent Text Reader

Abstract

To suppress a difference between the timing of speech in a video and the timing of displaying subtitles showing a translation result of the spoken speech when viewing content including the video and the audio.SOLUTION: A translation display device 10 includes a content acquisition unit 111 that acquires content data CD, an audio extraction unit 112 that extracts audio data SD indicating audio from the content data CD, a communication control unit 114 that causes first character data LD1 into which the audio data SD has been converted to be transmitted to an external translation server 30, a translation result acquisition unit 115 that acquires second character data LD2 indicating the result of the translation of the first character data LD1 by the translation server 30 from the translation server 30, and a display control unit 117 that synchronizes the image indicated by the content data CD and the translation result indicated by the second character data LD2 and displays them on a display device.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a translation display device. [Background technology]

[0002] 2. Description of the Related Art When users of personal computers (PCs), smartphones, and tablets watch video in which conversations are conducted in a foreign language, subtitles with the conversation translated may be displayed.

[0003] For example, Patent Document 1 discloses a technology relating to a desktop conference system using the Internet. In the system according to Patent Document 1, character data corresponding to audio signals indicating the voices of the conference attendees is translated. Then, subtitles including the translation results are displayed on a display together with the video of the conference attendees. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 10-136327 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, when a conference attendee speaks, the display of the subtitles is delayed from the image of the conference attendee speaking because the subtitles are a translation of the spoken voice. As a result, there is a problem that a mismatch occurs between the image of the conference attendee and the subtitles.

[0006] Therefore, an object of the present invention is to reduce the discrepancy between the timing of speech in the video and the timing of subtitles showing the translation result of the spoken speech being displayed when viewing content including video and audio. [Means for solving the problem]

[0007] A translation display device according to a preferred embodiment of the present invention includes a content acquisition unit that acquires content data indicating content including video and audio, a voice extraction unit that extracts audio data indicating the audio from the content data, a conversion unit that converts the audio data extracted by the voice extraction unit into first character data, a communication control unit that causes a communication device to transmit the first character data converted by the conversion unit to an external translation server, a translation result acquisition unit that acquires from the translation server second character data indicating a translation result of the first character data by the translation server, and a display control unit that synchronizes the video and the translation result indicated by the second character data and displays them on a display device. Effect of the Invention

[0008] According to the present invention, when viewing content including video and audio, the discrepancy between the timing of speech in the video and the timing of subtitles showing the translation of the spoken audio being displayed can be reduced more than with conventional technology. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows the overall configuration of a translation display system 1. [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of a translation display device 10. [Diagram 3] FIG. 2 is a diagram showing an example of a composite image SP. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a content server 20. [Diagram 5] FIG. 2 is a block diagram showing an example of the configuration of a translation server 30. [Figure 6] 4 is a flowchart showing an example of the operation of the translation display device 10. [Figure 7] FIG. 1 is a block diagram showing an example of the configuration of a translation display device 10A. [Figure 8] FIG. 13 is a diagram showing an example of a superimposed image PP. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of a translation display device 10B. [Figure 10] 4 is a timing chart showing the operation of the processing device 11B. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1: First embodiment Hereinafter, a configuration example of a translation display system 1 including a translation display device 10 according to a first embodiment of the present invention will be described with reference to FIGS.

[0011] 1-1: Configuration of the first embodiment 1-1-1: Overall composition Fig. 1 is a diagram showing the overall configuration of a translation display system 1. As shown in Fig. 1, the translation display system 1 includes a translation display device 10, a content server 20, and a translation server 30. The translation display device 10, the content server 20, and the translation server 30 are connected to each other so as to be able to communicate with each other via a communication network NET.

[0012] The content server 20 is a server that provides the translation display device 10 with content data indicating content including video and audio. Examples of the content include movies, dramas, live footage, and videos including animation. However, the content is not limited to videos. For example, the content may be content in which a still image is displayed for a predetermined period of time and audio is delivered for the predetermined period of time. The audio included in the content includes conversations between human or animated characters.

[0013] Translation display device 10 is a device that displays a translation of a conversation included in content supplied from content server 20 as subtitles. Translation display device 10 extracts voice data from content data indicating the content. Translation display device 10 also converts the extracted voice data into first character data. Translation display device 10 also transmits the first character data to translation server 30. Translation display device 10 also obtains second character data indicating the translation result of the first character data from translation server 30. Furthermore, translation display device 10 displays the second character data as subtitles.

[0014] The translation server 30 acquires the first character data from the translation display device 10. The translation server 30 then translates the first character data to generate second character data as a translation result. The translation server 30 then transmits the second character data to the translation display device 10.

[0015] 1, each of the translation display device 10, the content server 20, and the translation server 30 may be a PC (Personal Computer) or a mobile terminal. The mobile terminal may be a tablet or a smartphone.

[0016] 1-1-2: Configuration of the translation display device 2 is a block diagram showing an example of the configuration of translation display device 10. Translation display device 10 includes a processing device 11, a first storage device 12, a communication device 13, a display device 14, and an input device 15. Each element of translation display device 10 is connected to each other using a single or multiple buses for communicating information.

[0017] The processing device 11 is a processor that controls the entire translation display device 10. The processing device 11 is configured, for example, using a single or multiple chips. The processing device 11 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Some or all of the functions of the processing device 11 may be realized using hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). The processing device 11 executes various processes in parallel or sequentially.

[0018] The first storage device 12 is a recording medium that can be read from and written to by the processing device 11. The first storage device 12 also stores a plurality of programs including a control program PR1 that the processing device 11 executes.

[0019] The communication device 13 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), and the like.

[0020] The display device 14 is a device that displays images and text information. The display device 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display device 14.

[0021] The input device 15 accepts operations from a user of the translation display device 10. For example, the input device 15 includes a keyboard, a touch pad, a touch panel, or a pointing device such as a mouse. Here, when the input device 15 includes a touch panel, it may also serve as the display device 14.

[0022] The processing device 11 reads out and executes the control program PR1 from the first storage device 12. As a result, the processing device 11 functions as a content acquisition unit 111, a voice extraction unit 112, a conversion unit 113, a communication control unit 114, a translation result acquisition unit 115, a content playback unit 116, and a display control unit 117.

[0023] The content acquisition unit 111 acquires content data indicating content including video and audio from the content server 20. The content data includes video data indicating the video and audio data indicating the audio. As an example, in the content data, the video data and audio data are associated with each other on the same timeline. For example, if the content is a movie, video data indicating a video of an actor uttering a certain conversation and audio data indicating the conversation are linked at the same time on the same timeline.

[0024] The audio extraction unit 112 extracts audio data from the content data acquired by the content acquisition unit 111. As an example, the audio extraction unit 112 may extract audio data from the content data by using a general-purpose audio extraction technique. Alternatively, the audio extraction unit 112 may extract audio data by executing encoding to convert the data format of the content data.

[0025] The conversion unit 113 converts the voice data extracted by the voice extraction unit 112 into first character data by performing voice recognition. As an example, the conversion unit 113 may convert the voice data into the first character data by using a voice recognition technology based on AI (Artificial Intelligence) technology. When the conversion unit 113 uses a voice recognition technology based on AI technology, the conversion unit 113 may use a voice recognition technology provided in the translation display device 10 itself, or may use a voice recognition technology based on AI technology provided in a voice recognition server (not shown). Specifically, the voice recognition server (not shown) may convert the voice data acquired from the translation display device 10 into the first character data, and transmit the converted first character data to the translation server 30.

[0026] The communication control unit 114 causes the translation server 30 to transmit the first character data converted from the voice data by the conversion unit 113 to the communication device 13 .

[0027] The translation result acquisition unit 115 acquires, from the translation server 30, second character data indicating the result of the translation of the first character data by the translation server 30.

[0028] The content playback unit 116 plays back video data representing the video of the content and audio data representing the audio of the content based on the content data. In this embodiment, as described below, a browser is displayed on the display device 14. The content playback unit 116 plays back the video data and audio data using the browser.

[0029] The display control unit 117 synchronizes the video played by the content playback unit 116 using the browser with the translation result indicated by the second character data acquired by the translation result acquisition unit 115, and causes the display device 14 to display them.

[0030] More specifically, the display control unit 117 delays the playback of the video by controlling the content playback unit 116. Furthermore, the display control unit 117 generates a composite image in which the delayed playback of the video is displayed in a browser included in a first area of ​​the display device 14, and the translation result indicated by the second character data is displayed in subtitles included in a second area of ​​the display device 14. Furthermore, the display control unit 117 causes the display device 14 to display the generated composite image.

[0031] 3 is a diagram showing an example of a composite image SP displayed on a liquid crystal panel serving as the display device 14. The composite image SP includes a first region R1 and a second region R2. The first region R1 displays an image B of the browser. The image B of the browser includes an image PI represented by video data played back by the content playback unit 116. The second region R2 includes subtitles TS. In the example shown in FIG. 3, the composite image SP is displayed full-screen on the display device 14. However, the composite image SP may be displayed in a part of the display device 14.

[0032] As described above, the display control unit 117 delays the playback of the video PI from the initial playback timing. Specifically, as described above, the translation display device 10 extracts voice data from the content data. The translation display device 10 also converts the extracted voice data into first character data. Alternatively, the translation display device 10 transmits the extracted voice data to an external voice recognition server, and the voice recognition server acquires the first character data converted from the voice data. The translation display device 10 also transmits the converted first character data to the translation server 30. The translation display device 10 also acquires second character data as a translation result. That is, it takes a certain amount of time for the translation display device 10 to acquire the second character data after extracting voice data from the content data. Therefore, if the content playback unit 116 plays back the video data included in the content data without delay, a time lag occurs from the time of playback of the video data to the time of displaying the translation result of the conversation indicated by the voice data linked to the same time point as the video data on the timeline of the content data. Therefore, as described above, the display control unit 117 controls the content playback unit 116 to delay playback of the video PI using the browser, thereby eliminating the time lag. As a result, in Fig. 3, the video PI displayed in image B of the browser and the translation result displayed as subtitle TS are synchronized. Also, in Fig. 3, the video PI displayed in image B of the browser and the translation result displayed as subtitle TS correspond to the same point in time on the timeline of the content data.

[0033] In the translation display device 10, the reproduction of the audio data by the content reproduction unit 116 may or may not be delayed, like the video data.

[0034] Furthermore, the user of the translation display device 10 can move the position of the first region R1 and the position of the second region R2 in the composite image SP shown in Fig. 3 by operating the input device 15. For example, the user of the translation display device 10 can move the position of the second region R2 by dragging the second region R2 with a mouse.

[0035] 1-1-3: Content server configuration 4 is a block diagram showing an example of the configuration of the content server 20. The content server 20 includes a processing device 21, a storage device 22, a communication device 23, a display device 24, and an input device 25. The elements included in the content server 20 are connected to each other using one or more buses for communicating information.

[0036] The processing device 21 is a processor that controls the entire content server 20. The processing device 21 is configured, for example, using one or more chips. The processing device 21 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Some or all of the functions of the processing device 21 may be realized using hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0037] The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR2 executed by the processing device 21. The storage device 22 also stores a content database CDB. The content database CDB stores a plurality of content data indicating content including video PI and audio.

[0038] The communication device 23 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), and the like.

[0039] The processing device 21 reads out and executes the control program PR2 from the storage device 22. As a result, the processing device 21 functions as a communication control unit 211.

[0040] The communication control unit 211 causes the translation display device 10 to transmit the content data stored in the content database CDB to the communication device 23.

[0041] As an example, the communication control unit 211 causes the communication device 23 to receive, from the translation display device 10, a designation signal that designates the content to be played on the translation display device 10. In addition, the communication control unit 211 causes the communication device 23 to transmit content data corresponding to the content designated by the designation signal to the translation display device 10.

[0042] 1-1-4: Translation server configuration 5 is a block diagram showing an example of the configuration of the translation server 30. The translation server 30 includes a processing device 31, a storage device 32, a communication device 33, a display device 34, and an input device 35. The elements of the translation server 30 are connected to each other using one or more buses for communicating information.

[0043] The processing device 31 is a processor that controls the entire translation server 30. The processing device 31 is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 31 may be realized using hardware such as a DSP, ASIC, PLD, and FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0044] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR3 that the processing device 31 executes.

[0045] The communication device 33 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), and the like.

[0046] The processing device 31 reads out and executes the control program PR3 from the storage device 32. As a result, the processing device 31 functions as a communication control unit 311 and a translation unit 312.

[0047] The communication control unit 311 causes the communication device 33 to receive the first character data from the translation display device 10.

[0048] The translation unit 312 generates second character data by translating the first character data. As an example, the translation unit 312 generates the second character data by translating the first character data by machine translation using AI technology.

[0049] The communication control unit 311 causes the translation display device 10 to transmit the second character data as a translation result to the communication device 33.

[0050] 1-2: Operation of the First Embodiment FIG. 6 is a flowchart showing an example of the operation of the translation display device 10 according to the first embodiment.

[0051] In step S1, the processing device 11 functions as a content acquisition unit 111. The processing device 11 acquires content data CD indicating content including video PI and audio.

[0052] In step S2, the processing device 11 functions as the audio extraction section 112. The processing device 11 extracts audio data SD indicating audio from the content data CD acquired in step S1.

[0053] In step S3, the processing device 11 functions as the conversion unit 113. The processing device 11 converts the voice data SD extracted in step S2 into first character data LD1.

[0054] In step S4, the processing device 11 functions as the communication control unit 114. The processing device 11 causes the communication device 13 to transmit the first character data LD1 converted in step S3 to the external translation server 30.

[0055] In step S5, the processing device 11 functions as the translation result acquisition unit 115. The processing device 11 acquires, from the translation server 30, the second character data LD2 indicating the translation result of the first character data LD1.

[0056] In step S6, the processing device 11 functions as the display control unit 117. The processing device 11 synchronizes the video PI indicated by the video data PD included in the content data CD acquired in step S1 with the translation result indicated by the second character data LD2 acquired in step S5, and displays them on the display device 14. Specifically, the processing device 11 delays the playback of the video PI using the browser. As a result, the video PI indicated by the video data PD included in the content data CD acquired in step S1 is synchronized with the subtitles TS as the translation result indicated by the second character data LD2 acquired in step S5.

[0057] 1-3: Advantages of the First Embodiment The translation display device 10 according to the present embodiment includes a content acquisition unit 111, a voice extraction unit 112, a conversion unit 113, a communication control unit 114, a translation result acquisition unit 115, and a display control unit 117. The content acquisition unit 111 acquires content data CD indicating content including video PI and voice. The voice extraction unit 112 extracts voice data SD indicating voice from the content data CD. The communication control unit 114 causes the communication device 13 to transmit the first character data LD1 into which the voice data SD is converted, to the external translation server 30. The translation result acquisition unit 115 acquires second character data LD2 indicating the translation result of the first character data LD1 by the translation server 30 from the translation server 30. The display control unit 117 synchronizes the video PI indicated by the content data CD acquired by the content acquisition unit 111 with the translation result indicated by the second character data LD2, and causes the display device 14 to display them.

[0058] Because the translation display device 10 has the above-described configuration, when viewing content including video PI and audio, it is possible to reduce the discrepancy between the timing at which speech is spoken in the video and the timing at which subtitles TS, which show the translation result of the spoken speech, are displayed.

[0059] Furthermore, in the translation display device 10, the display control unit 117 delays the display of the image PI, thereby causing the display device 14 to display the image PI and the translation result indicated by the second character data LD2 in synchronization with each other.

[0060] Since the translation display device 10 has the above-mentioned configuration, by delaying the display of the video PI, it is possible to eliminate the time lag between sending the first character data LD1 to the external translation server 30 and receiving the second character data LD2 as the translation result from the translation server 30.

[0061] The translation display device 10 further includes a content playback unit 116. The content playback unit 116 plays video data PD indicating a video PI of the content and audio data SD indicating audio of the content based on the content data CD. The display control unit 117 executes the following: delaying the video PI based on the video data PD, displaying the delayed video PI in the first area R1, generating a composite image SP that displays the translation result indicated by the second character data LD2 in the second area R2, and displaying the generated composite image SP on the display device 14.

[0062] Since the translation display device 10 has the above-described configuration, it is possible to eliminate the time lag between sending the first character data LD1 to the external translation server 30 and receiving the second character data LD2 as the translation result from the translation server 30, thereby synchronizing the video PI and the subtitles TS as the translation result in the composite image SP.

[0063] 2: Second embodiment Hereinafter, a configuration example of a translation display system 1A including a translation display device 10A according to a second embodiment of the present invention will be described with reference to Fig. 7 and Fig. 8. For the sake of simplicity, the following mainly describes the differences between the translation display system 1A and the translation display system 1. Furthermore, among the components of the translation display system 1A, the same components as those of the translation display system 1 will be designated by the same reference numerals, and the description of their functions may be omitted.

[0064] In the translation display device 10 according to the first embodiment, the content playback unit 116 synchronizes the video PI played back using the browser with the subtitle TS as the translation result indicated by the second character data LD2 acquired by the translation result acquisition unit 115. On the other hand, in the translation display device 10A according to the second embodiment, the display control unit 117A described later delays the display of the video PI based on the video data PD, and generates a superimposed image in which the translation result is superimposed on the delayed video PI. Also, in the translation display device 10A according to the second embodiment, the superimposed image is displayed on the execution screen of the application, and within the superimposed image, the video PI and the subtitle TS as the translation result are synchronized.

[0065] 2-1: Configuration of the second embodiment 2-1-1: Configuration of the translation display device The translation display system 1A differs from the translation display system 1 according to the first embodiment in that it includes a translation display device 10A instead of the translation display device 10. In other respects, the translation display system 1A is the same as the translation display system 1 shown in FIG. 1, and therefore the illustration of the translation display system 1A is omitted.

[0066] 2-1-2: Configuration of the translation display device 7 is a block diagram showing a configuration example of a translation display device 10A. The translation display device 10A is different from the translation display device 10 according to the first embodiment in that it includes a processing device 11A instead of the processing device 11 and a first storage device 12A instead of the first storage device 12.

[0067] The first storage device 12A is a recording medium that can be read and written by the processing device 11A. The first storage device 12A also stores a plurality of programs including a control program PR1A and an application program AP1 executed by the processing device 11A. The application program AP1 is a program for the processing device 11A to execute an application that provides video PI and audio. A user of the translation display device 10A can view the video PI and audio included in the content by causing the processing device 11A to execute the application.

[0068] The processing device 11A reads out and executes the control program PR1A from the first storage device 12A. As a result, the processing device 11A functions as the same content acquisition unit 111, speech extraction unit 112, conversion unit 113, communication control unit 114, and translation result acquisition unit 115 as the processing device 11. The processing device 11A also reads out and executes the application program AP1 from the first storage device 12A. As a result, the processing device 11A functions as a content playback unit 116A in place of the content playback unit 116 included in the processing device 11, and as a display control unit 117A in place of the display control unit 117 included in the processing device 11.

[0069] The content reproducing unit 116A reproduces video data PD indicating the video PI of the content and audio data SD indicating the audio of the content based on the content data CD. In this embodiment, as described later, the video PI indicated by the video data and the audio indicated by the audio data SD are reproduced within an execution screen of an application displayed on the display device 14. Also, as described later, the translation result indicated by the second character data LD2 is displayed superimposed on the video PI on the execution screen of the application.

[0070] The display control unit 117A delays the display of the video PI based on the video data PD. As an example, the processing device 11A uses a virtual imaging device to capture the video PI represented by the video data PD reproduced by the content reproduction unit 116A. The display control unit 117A delays the display of the captured video PI, thereby delaying the display of the video PI based on the video data PD. Furthermore, the display control unit 117A generates a superimposed image in which the subtitles TS as the translation result represented by the second character data LD2 are superimposed on the video PI based on the video data PD. Furthermore, the display control unit 117A causes the display device 14 to display the generated superimposed image.

[0071] FIG. 8 is a diagram showing an example of a superimposed image PP displayed on the display device 14. The superimposed image PP is an image in which subtitles TS are superimposed on the video PI as a translation result. The superimposed image PP is included in the execution screen RP of the application. Note that in the example shown in FIG. 8, the execution screen RP of the application is displayed in a part of the display device 14. However, the execution screen RP may be displayed in the entire screen of the display device 14. Also, in the example shown in FIG. 8, the superimposed image PP is displayed in a part of the execution screen RP of the application. However, the superimposed image PP may be displayed in the entire execution screen RP of the application.

[0072] The display control unit 117A delays the playback of the video PI using the application. As a result, in Fig. 8, the video PI displayed on the application execution screen RP and the translation result displayed as subtitle TS are synchronized. Also, in Fig. 8, the video PI displayed on the application execution screen RP and the translation result displayed as subtitle TS correspond to the same time point on the timeline of the content data CD.

[0073] In the translation display device 10A, the reproduction of the audio data SD by the content reproduction unit 116A may or may not be delayed, similar to the reproduction of the video data PD.

[0074] Furthermore, the user of the translation display device 10A can move the position of the subtitles TS in the superimposed image PP shown in Fig. 8 by operating the input device 15. For example, the user of the translation display device 10 can move the position of the subtitles TS by dragging the subtitles TS with a mouse.

[0075] 2-2: Operation of the second embodiment An example of the operation of the translation display device 10A according to the second embodiment is basically similar to the flowchart showing the example of the operation of the translation display device 10 according to the first embodiment shown in Fig. 6, and therefore is not shown in the figure. However, the details of the operation at step S6 are different between the operation of the translation display device 10A and the operation of the translation display device 10.

[0076] In step S6 according to the second embodiment, the processing device 11A functions as a display control unit 117A. The processing device 11A synchronizes the video PI indicated by the video data PD included in the content data CD acquired in step S1 with the translation result indicated by the second character data LD2 acquired in step S5, and displays them on the display device 14. Specifically, the processing device 11A captures the video PI indicated by the video data PD with a virtual imaging device. The processing device 11A also delays the display of the captured video PI, thereby delaying the display of the video PI based on the video data PD. As a result, the video PI indicated by the video data PD included in the content data CD acquired in step S1 is synchronized with the subtitles TS as the translation result indicated by the second character data LD2 acquired in step S5.

[0077] 2-3: Advantages of the Second Embodiment The translation display device 10A according to this embodiment includes a content playback unit 116A. The content playback unit 116A plays video data PD indicating a video PI of the content and audio data SD indicating the audio of the content based on content data CD. The display control unit 117A executes the following: delaying the video PI based on the video data PD, generating a superimposed image PP in which the translation result indicated by the second character data LD2 is superimposed on the video PI based on the video data PD, and displaying the generated superimposed image PP on the display device 14.

[0078] Since the translation display device 10A has the above-described configuration, it is possible to eliminate the time lag between sending the first character data LD1 to the external translation server 30 and receiving the second character data LD2 as the translation result from the translation server 30, thereby synchronizing the video PI and the subtitles TS as the translation result in the superimposed image PP.

[0079] 3: Third embodiment Hereinafter, a configuration example of a translation display system 1B including a translation display device 10B according to a third embodiment of the present invention will be described with reference to Fig. 9 and Fig. 10. For the sake of simplicity, the following mainly describes the differences between the translation display system 1B and the translation display system 1. Furthermore, among the components of the translation display system 1B, the same components as those of the translation display system 1 will be designated by the same reference numerals, and the description of their functions may be omitted.

[0080] In the translation display device 10 according to the first embodiment and the translation display device 10A according to the second embodiment, the content acquisition unit 111 acquires the content data CD, and the voice extraction unit 112 extracts the voice data SD indicating the voice from the content data CD. Meanwhile, in the translation display device 10B according to the third embodiment, the content acquisition unit 111B described below temporarily stores the content data CD consisting of multiple frames in a storage device. In addition, the voice extraction unit 112B described below extracts the voice data SD from the frames included in the content data CD stored in the storage device.

[0081] 3-1: Configuration of the third embodiment 3-1-1: Configuration of the translation display device The translation display system 1B differs from the translation display system 1 according to the first embodiment in that it includes a translation display device 10B instead of the translation display device 10. In other respects, the translation display system 1B is the same as the translation display system 1 shown in FIG. 1, and therefore the illustration of the translation display system 1B is omitted.

[0082] 3-1-2: Configuration of the translation display device 9 is a block diagram showing a configuration example of a translation display device 10B. Unlike the translation display device 10 according to the first embodiment, the translation display device 10B includes a processing device 11B instead of the processing device 11 and a first storage device 12B instead of the first storage device 12. The translation display device 10B further includes a second storage device 16.

[0083] The first storage device 12B is a recording medium that can be read from and written to by the processing device 11B. The first storage device 12B also stores a plurality of programs including a control program PR1B executed by the processing device 11B.

[0084] The second storage device 16 stores content data CD indicating content including video PI and audio. More specifically, the second storage device 16 stores content data CD consisting of a plurality of frames acquired by the content acquisition unit 111B. The second storage device 16 is, for example, a buffer. The second storage device 16 is also an example of a "storage device".

[0085] The processing device 11B reads out and executes the control program PR1B from the first storage device 12B. As a result, the processing device 11B functions as a content acquisition unit 111B, a voice extraction unit 112B, a conversion unit 113, a communication control unit 114, a translation result acquisition unit 115, and a display control unit 117B. Note that the conversion unit 113, the communication control unit 114, and the translation result acquisition unit 115 are the same as the conversion unit 113, the communication control unit 114, and the translation result acquisition unit 115 provided in the processing device 11 according to the first embodiment.

[0086] Fig. 10 is a timing chart showing the operation of the processing device 11B. First, the content acquisition unit 111B stores the content data CD acquired from the content server 20 in the second storage device 16. As described above, the content data CD is composed of a plurality of frames FL. In the example shown in Fig. 10, the content data CD is composed of n frames FL from the first frame FL1 to the nth frame FLn. Here, n is an integer equal to or greater than 2.

[0087] The audio extraction unit 112B extracts audio data SD from a first frame FL1 included in the content data CD stored in the second storage device 16 at a first time point t1.

[0088] Thereafter, similarly to the first embodiment, conversion unit 113 converts the voice data SD extracted from first frame FL1 by voice extraction unit 112B into first character data LD1 by performing voice recognition. Furthermore, communication control unit 114 causes translation server 30 to transmit first character data LD1 to communication device 13. Furthermore, translation result acquisition unit 115 acquires second character data LD2 indicating the translation result of first character data LD1 from translation server 30.

[0089] The display control unit 117B acquires video data PD indicating the video PI from the first frame FL1 stored in the second storage device 16 at a second time point t2 after the translation result acquisition unit 115 acquires the second character data LD2. Furthermore, the display control unit 117B synchronizes the video PI with the subtitles TS as the translation result indicated by the second character data LD2, and causes them to be displayed on the display device 14.

[0090] Note that the translation display device 10B according to the third embodiment may use a browser to display a composite image SP in which the video PI is displayed in a first region R1 and the subtitles TS as the translation result is displayed in a second region R2, similar to the translation display device 10 according to the first embodiment. Alternatively, the translation display device 10B according to the third embodiment may execute an application and display a superimposed image PP in which the subtitles TS as the translation result is superimposed on the video PI, within the execution screen of the application, similar to the translation display device 10A according to the second embodiment.

[0091] 3-2: Operation of the third embodiment An operation example of the translation display device 10B according to the third embodiment is basically similar to the flow chart showing the operation example of the translation display device 10 according to the first embodiment shown in Fig. 6, and therefore is not illustrated. However, the operation of the translation display device 10B and the operation of the translation display device 10 differ in details of the operation in steps S1, S2, and S6.

[0092] In step S1 according to the third embodiment, the processing device 11B functions as a content acquisition unit 111B. The processing device 11B acquires content data CD indicating content including video PI and audio, and stores the acquired content data CD in the second storage device 16.

[0093] In step S2 according to the third embodiment, the processing device 11B functions as an audio extraction unit 112B. The processing device 11B extracts audio data SD from a first frame FL1 included in the content data CD stored in the second storage device 16 at a first time point t1.

[0094] Furthermore, in step S6 according to the third embodiment, the processing device 11B functions as a display control unit 117B. At a second time point t2 after acquiring the second character data LD2, the processing device 11B acquires video data PD indicating the video PI from the first frame FL1 included in the content data CD stored in the second storage device 16, and displays the video PI and the translation result indicated by the second character data LD2 on the display device 14 in synchronization with each other.

[0095] 3-3: Advantages of the Third Embodiment In the translation display device 10B according to the present embodiment, the content acquisition unit 111B stores the acquired content data CD in the second storage device 16 as a storage device. The content data CD is composed of a plurality of frames FL. The voice extraction unit 112B extracts voice data SD from the first frame FL1 included in the content data CD stored in the second storage device 16 at a first time point t1. The display control unit 117B acquires video data PD indicating the video PI from the first frame FL1 included in the content data CD stored in the second storage device 16 at a second time point t2 after the translation result acquisition unit 115 acquires the second character data LD2. The display control unit 117B also synchronizes the video PI with the translation result indicated by the second character data LD2 and displays them on the display device 14.

[0096] Since the translation display device 10B has the above configuration, it can extract only the audio data SD contained in the first frame FL1 while the content data CD indicating the content is temporarily stored in the second storage device 16 as a buffer, and can translate the conversation indicated by the audio data SD in advance before displaying the video PI indicated by the video data PD contained in the first frame FL1. As a result, the translation display device 10B can reduce the lag between the timing of speech in the video PI and the timing of displaying the subtitles TS indicating the translation of the spoken speech, compared to the prior art.

[0097] 4: Variations The present disclosure is not limited to the above-described embodiments. Specific modified embodiments are exemplified below.

[0098] 4-1: Variation 1 The translation display device 10B according to the third embodiment may further include an audio playback unit as a component of the processing device 11B. The audio playback unit may play back the audio data SD included in the first frame FL1 in parallel with the display control unit 117B displaying, on the display device 14, a video PI represented by the video data PD acquired from the first frame FL1 and a subtitle TS as a translation result represented by the second character data LD2 in synchronization with each other at a time point after the second time point t2.

[0099] 4-2: Variation 2 In the first embodiment, the translation display device 10 converts the extracted voice data into the first character data. However, instead of the translation display device 10, a voice recognition server (not shown) may convert the voice data acquired from the translation display device 10 into the first character data and transmit the converted first character data to the translation server 30. The voice recognition server may be a server directly connected to the translation display device 10, or may be a cloud server communicably connected to the translation display device 10 via the communication network NET.

[0100] 4-3: Variation 3 In the first embodiment, translation server 30, which is separate from translation display device 10, acquires first character data from translation display device 10 and transmits second character data as a translation result of the first character data to translation display device 10. However, translation display device 10 and translation server 30 may be integrated in the same housing and realized as a single device.

[0101] 5:Other (1) In the above-described embodiment, ROM and RAM are exemplified as first storage device 12, 12A, and 12B and second storage device 16, but the storage devices may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0102] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0103] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0104] (4) In the above-described embodiments, the determination may be made based on a value represented using one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0105] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0106] (6) Each function illustrated in Figures 1, 2, 4, 5, 7, and 9 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0107] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0108] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0109] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.

[0110] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0111] (10) In the above-described embodiments, the translation display devices 10, 10A, and 10B, the content server 20, and the translation server 30 may be mobile stations (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In the present disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", and the like may be used interchangeably.

[0112] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read with "access". As used in this disclosure, two elements are considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0113] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0114] (13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0115] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.

[0116] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0117] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0118] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to explicit notification, but may be implicit (e.g., not notifying the specific information).

[0119] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is for illustrative purposes only and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0120] 1, 1A, 1B... translation display system, 10, 10A, 10B... translation display device, 11, 11A, 11B... processing device, 12, 12A, 12B... first storage device, 13... communication device, 14... display device, 15... input device, 16... second storage device, 20... content server, 21... processing device, 22... storage device, 23... communication device, 24... display device, 25... input device, 30... translation server, 31... processing device, 32... storage device, 33... communication device, 34... display device, 35... input device, 111, 11 1B...content acquisition unit, 112, 112B...speech extraction unit, 113...conversion unit, 114...communication control unit, 115...translation result acquisition unit, 116, 116A...content playback unit, 117, 117A, 117B...display control unit, 211...communication control unit, 311...communication control unit, 312...translation unit, AP1...application program, LD1...first character data, LD2...second character data, PR1, PR1A, PR1B, PR2, PR3...control program, R1...first region, R2...second region

Claims

1. a content acquisition unit that acquires content data indicating content including video and audio; a voice extraction unit that extracts voice data indicating the voice from the content data; a communication control unit that causes a communication device to transmit the first character data obtained by converting the voice data to an external translation server; a translation result acquisition unit that acquires second character data indicating a translation result of the first character data by the translation server from the translation server; a display control unit that synchronizes a second image, which is obtained by capturing a first image represented by the content data acquired by the content acquisition unit using a virtual imaging device, with a translation result represented by the second character data, and displays the synchronized second image and the translation result on a display device; Equipped with Translation display device.

2. the display control unit delays the display of the image to synchronize the second image with the translation result indicated by the second character data and display them on the display device. The translation display device according to claim 1 .

3. Based on the content data, video data showing a video of the content and the content a content reproducing unit for reproducing audio data representing the audio of the content, The display control unit is delaying the second image based on the image data; generating a composite image in which the delayed second image is displayed in a first area and a translation result represented by the second character data is displayed in a second area; and displaying the generated composite image on the display device. The translation display device according to claim 2 .

4. Based on the content data, video data showing a video of the content and the content a content reproducing unit for reproducing audio data representing the audio of the content, The display control unit is delaying the second image based on the image data; generating a superimposed image in which a translation result indicated by the second character data is superimposed on the second image based on the image data; and displaying the generated superimposed image on the display device. The translation display device according to claim 2 .

Citation Information

Patent Citations

  • Mobile terminal-based audio processing method and device

    CN103226947A

  • Translation method and device for subtitles of media file, and electronic equipment

    CN106303303A

  • Cloud conference subtitle display method, system and device, electronic equipment and storage medium

    CN114125358A

  • Display apparatus and control method thereof

    EP2840797A2

  • Desk top conference system

    JP1998136327A