System, terminal, transmission method, and program

Abstract

To realize natural subtitle display without discomfort in a simple way. [Solution] A video distribution system distributes comments on a video along with the video. The video distribution system includes a viewer terminal 4 that displays comments on the same screen as the video, and a distributor terminal 1 that provides the video. The distributor terminal 1 includes a transmitter 15 that transmits the video, a converter 12 that converts speech included in the video into text, and a comment transmitter 16 that transmits the text as a comment on the video.

Description

[Technical Field]

[0001] The present disclosure relates to a system, a terminal, a transmission method, and a program. [Background technology]

[0002] Even for viewers watching real-time broadcasts in noisy environments or when speakers are unavailable, subtitles can be used to convey the content of the speech, allowing viewers to understand and participate in the broadcast. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-73901 [Patent Document 2] Patent No. 6295494 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 discloses a technology for recognizing speech of performers in a broadcast program and superimposing it as subtitles.

[0005] The streamed video is encoded on the streamer's device into a format compatible with the video streaming site before being transmitted. To superimpose the streamer's speech as subtitles, one possible method is to recognize the speech and encode the streamed video with the speech results superimposed as subtitles. However, since it takes time to obtain the speech recognition results, the speech is superimposed on the streamed video at a later time than the actual speech. As a result, viewers may perceive the subtitles as being delayed compared to the audio, which can lead to a degradation in the quality of the viewing experience.

[0006] The present disclosure has been made in view of the above, and aims to realize a natural subtitle display without any sense of discomfort using a simple method. [Means for solving the problem]

[0007] A system according to an embodiment of the present disclosure includes: and A system for distributing comments on a video to be distributed includes a viewer terminal that displays the comments on the same screen as the video to be distributed, and a distributor terminal that provides the video to be distributed, wherein the distributor terminal: an input unit for inputting speech and video; a first transmission unit that transmits the distribution video; In parallel with the encoding, the speech and a second transmitting unit that transmits the text as a comment on the video to be distributed. [Effects of the Invention]

[0008] According to the present disclosure, natural subtitle display without any sense of incongruity can be achieved using a simple method. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a video distribution system according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a distributor terminal. [Figure 3] FIG. 3 is a flowchart showing an example of the flow of processing performed by the distributor terminal. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on the viewer terminal. [Figure 5] FIG. 5 is a diagram showing an example of a display of the provisional recognition result. DETAILED DESCRIPTION OF THE INVENTION

[0010] [Video distribution system] An example of the configuration of a video distribution system according to this embodiment will be described with reference to FIG. 1. The video distribution system shown in the figure includes a distributor terminal 1, a video distribution server 2, a comment distribution server 3, and a viewer terminal 4. Each device is communicatively connected via a network. While only two viewer terminals 4 are shown in FIG. 1, this is not limiting. In reality, many viewers exist, and many viewer terminals 4 are connected. Furthermore, although only one distributor terminal 1 is shown, in reality, many distributors (also called broadcasters or performers) exist, and many distributor terminals 1 are connected. Viewers can freely select and watch any live broadcast program (also called live streaming content or video).

[0011] The video distribution server 2 distributes the video received from the distributor terminal 1 to the viewer terminal 4 in real time. Distributing video in real time is also called real-time distribution, live distribution, live broadcast distribution, program distribution, stream distribution, or live streaming. The video distribution server 2 may store the video received from the distributor terminal 1 and distribute it to the viewer terminal 4 at any time in response to a request from the viewer terminal 4. Distributing video at any time in response to a viewer request is also called time-shift distribution, on-demand distribution, or archive distribution.

[0012] The comment distribution server 3 receives comments entered by viewers on videos from the viewer terminal 4 and distributes the received comments in real time to the viewer terminal 4 and the distributor terminal 1 that are receiving the same video distribution. The comments received from the viewer terminal 4 include, for example, the content of the comment (character string, character information, stamp, etc.), display attributes, user ID, and comment time. The display attributes are attributes that indicate the display mode of the comment (size, display position on the screen, color, etc.). The user ID is an identifier for the user who posted the comment. It may also include the poster's name or nickname. The comment time is the playback time of the video at the time the user posted the comment (a timestamp based on the start of the video). It may also include information about the actual time when the comment was posted. Similar to the viewer terminal 4, the comment distribution server 3 receives comments from the distributor terminal 1 and distributes them to the viewer terminal 4.

[0013] The comment distribution server 3 manages and stores comments for each video. When the comment distribution server 3 receives a comment distribution request from the viewer terminal 4, it starts sending comments to the viewer terminal 4 and receiving comments from the viewer terminal 4. The comment distribution server 3 distributes comments received from the distributor terminal 1 and the viewer terminal 4 to the distributor terminal 1 and the viewer terminal 4. In the case of time-shift viewing, the comment distribution server 3 distributes comments posted during real-time distribution to the viewer terminal 4. The technology described in Patent Document 2 can be used to distribute comments.

[0014] The viewer terminal 4 is a terminal used by viewers who watch videos, and receives and displays videos from the video distribution server 2, as well as comments from the comment distribution server 3. The viewer terminal 4 also transmits comments entered by viewers to the comment distribution server 3. The viewer terminal 4 may display comments superimposed on the video, or may display the comments in a comment section outside the video display area. Viewers can check past comments in the comment section.

[0015] The distributor terminal 1 is a terminal used by the distributor, which inputs video and audio, encodes the input video and audio, and transmits them to the video distribution server 2. It also converts the distributor's speech into text using voice recognition and transmits it to the comment distribution server 3.

[0016] The broadcaster terminal 1 converts the spoken voice into text and sends it as a comment to the comment distribution server 3, so that the broadcaster's spoken content is displayed as a comment on the viewer terminal 4. This allows viewers to understand the broadcaster's spoken content even when they cannot hear the audio of the video, allowing them to enjoy the video. Also, viewers who start watching a live broadcast program halfway through can check the broadcaster's past spoken content in the comment section.

[0017] By using the existing comment function as subtitles, it is possible to provide a subtitle display function without adding new functions to the video distribution server 2, the comment distribution server 3, and the viewer terminal 4.

[0018] [Distributor's device] An example of the configuration of the broadcaster terminal 1 will be described with reference to Fig. 2. The broadcaster terminal 1 shown in the figure includes an input unit 11, a conversion unit 12, a voice recognition unit 13, an encoder 14, a transmission unit 15, and a comment transmission unit 16. The broadcaster terminal 1 can be a device such as a personal computer (PC), a smartphone, or a tablet terminal.

[0019] The input unit 11 inputs video signals and audio signals (also referred to as content) from a camera or a microphone. For example, the input unit 11 inputs video of the broadcaster captured by a camera or the broadcaster's speech. The camera and microphone may be provided in the broadcaster terminal 1. In addition to video captured by a camera, the input unit 11 may also input video captured in advance, an application screen (game screen, paint software, browser, etc.) running on the broadcaster terminal 1 or another device (game console, personal computer, mobile terminal, etc.), or a still image such as a photograph or illustration. The input unit 11 may input and combine multiple videos. For example, when broadcasting a video of a game being played, the input unit 11 combines video of the broadcaster captured by a camera with a game screen input from a game console.

[0020] The conversion unit 12 transmits the voice signal to the voice recognition unit 13, converts the distributor's speech into text, acquires time information for the speech interval, and transmits the text and time information to the comment transmission unit 16. The time information for the speech interval is preferably near the start of the speech interval.

[0021] The speech recognition unit 13 detects speech intervals from the audio signal, converts the audio of the speech interval into text, and returns it. Specifically, after detecting the start of speech, the speech recognition unit 13 divides the audio signal into short intervals, extracts features, and estimates phonemes and words. The speech recognition unit 13 returns a provisional result before the audio interval is determined as a temporary recognition result, and then returns the recognition result after the audio interval is determined. The timing at which the first provisional recognition result is obtained may be used as the time information of the speech interval. The speech recognition unit 13 may be a speech recognition program included in the distributor terminal 1, or may be a device separate from the distributor terminal 1. The speech recognition unit 13 may be a speech recognition server available via a network.

[0022] The encoder 14 encodes the video signal and audio signal into a format acceptable to the video distribution server 2 to generate a video to be distributed (also called a live stream, distribution stream, streaming data, live streaming data, data for distribution, or live distribution data).

[0023] The transmission unit 15 transmits the video to be distributed to the video distribution server 2 in real time.

[0024] The comment sending unit 16 receives the text and time information from the conversion unit 12, and transmits the speech content comment to the comment distribution server 3, with the text as the content of the comment and the time information as the comment time. The comment distribution server 3 may treat the speech content comment equally with the comment from the viewer terminal 4 (hereinafter also referred to as a normal comment), or may distinguish between the speech content comment and the normal comment. When the speech content comment is displayed on the viewer terminal 4, a display attribute may be assigned to the speech content comment so that it is clear that the speech content comment is the content of the distributor. For example, the speech content comment may be displayed at a fixed position on the screen displaying the video, or may be displayed while moving on the screen like a normal comment. The speech content comment may be displayed in a color or size different from that of a normal comment. The display attribute of the speech content comment may be assigned by the distributor terminal 1 or the comment distribution server 3. The viewer terminal 4 may assign the display attribute to the speech content comment, or the viewer terminal 4 may display the speech content comment in a manner different from that of a normal comment.

[0025] The distributor terminal 1 may transmit distributor comments that the distributor inputs using an input device such as a keyboard to the comment distribution server 3. Distributor comments and speech content comments may be distinguished.

[0026] [Distributor terminal operation] An example of the flow of processing in the distributor terminal 1 will be described with reference to the flowchart in Fig. 3. The following processing is repeated from when the distributor starts distribution until the distributor ends distribution.

[0027] In step S11, a video signal and an audio signal are input to the input unit 11. The audio signal is input to the audio recognition unit 13, and the video signal and audio signal are input to the encoder .

[0028] In step S12, the speech recognition unit 13 performs speech recognition and returns the text to the conversion unit 12. At this time, the conversion unit 12 also acquires time information of the speech section.

[0029] In step S13, the comment sending unit 16 adds time information to the text and sends it to the comment distribution server 3 as a comment.

[0030] Meanwhile, in step S14, the encoder 14 encodes the video signal and audio signal to generate a moving image to be distributed.

[0031] In step S15, the transmission unit 15 transmits the video to be distributed to the video distribution server 2.

[0032] The processing of the speech recognition unit 13 and the processing of the encoder 14 are executed in parallel. That is, speech recognition is not followed by encoding, nor is speech recognition performed after encoding.

[0033] The viewer terminal 4 receives the distributed video from the video distribution server 2 and comments on the distributed video from the comment distribution server 3, and displays the distributed video and the comments on the same screen. In addition to regular comments from viewers, the viewer terminal 4 also displays the content of the distributor's utterances as comments, so that the viewer can understand the content of the distributor's utterances even if the audio of the distributed video is difficult to hear or cannot be heard at all.

[0034] In this embodiment, subtitles are not included in the distributed video, but are sent as comments, and the comment function of the video distribution service is used to superimpose the subtitles on the distributed video on the viewer terminal 4. Since speech recognition processing can be executed in a shorter processing time than encoding processing, it is possible to shorten the difference between the timing of speech in the distributed video displayed on the viewer terminal 4 and the timing of displaying the subtitles.

[0035] Furthermore, since the comment on the content of the speech is provided with time information that matches the timing of the speech, in the case of time-shift viewing, the subtitles are displayed on the viewer terminal 4 in accordance with the timing of the speech.

[0036] [Display example] An example of a screen displayed on the viewer terminal 4 will be described with reference to FIG.

[0037] The screen 100 shown in Fig. 4 shows a video display area 110 that displays the streaming video and a comment field 120 that displays comments. In the video display area 110, in addition to regular comments posted by viewers, utterance content comments 111 are displayed. Regular comments move from the right side to the left side of the video display area 110. The utterance content comments 111 are fixedly displayed at the bottom of the video display area 110. The display mode of the utterance content comments 111 can be specified by display attributes assigned to the comments.

[0038] In addition to normal comments, comment field 120 also displays utterance content comments 121. Posted comments are added from top to bottom in comment field 120. By tracing the comment field 120 upward, past utterance content can be checked.

[0039] [Variations] If there is a large delay in the video being distributed and the timing of the distributor's speech is delayed compared to the timing of displaying the speech content comment, the distributor terminal 1 may delay the timing of sending the speech content comment to the comment distribution server 3. For example, the amount of delay for sending the speech content comment may be set. The amount of delay may be set by the distributor. The comment sending unit 16 sends the speech content comment after the amount of delay has elapsed since receiving the text. The viewer terminal 4 may also delay the timing of displaying the speech content comment. When delaying the timing of sending the speech content comment, the distributor terminal 1 may add the amount of delay to the comment time assigned to the speech content comment to delay the comment time.

[0040] If the text after speech recognition is longer than a predetermined length, the distributor terminal 1 may divide the text into appropriate lengths that are suitable for display and transmit the divided utterance content comments. The viewer terminal 4 may divide the utterance content comments that are longer than a predetermined length and display the divided utterance content comments.

[0041] The distributor terminal 1 may display the provisional recognition result so that the distributor can confirm it. For example, as shown in FIG. 5, the distributor terminal 1 displays the provisional recognition result, indicating that conversion is in progress, and then displays the confirmed speech recognition result. The distributor terminal 1 may transmit the provisional recognition result as a comment to the comment distribution server 3. When the speech recognition result is confirmed, the distributor terminal 1 transmits the confirmed speech recognition result to the comment distribution server 3. The comment distribution server 3 replaces the provisional recognition result with the confirmed speech recognition result. The viewer terminal 4 displays the provisional recognition result, and then, when it receives the confirmed speech recognition result, replaces the provisional recognition result with the confirmed speech recognition result and displays it.

[0042] If the broadcaster terminal 1 has a function to read posted comments aloud using a machine voice, it does not need to read out utterance content comments. Specifically, the broadcaster terminal 1 receives comments from the comment distribution server 3. Since the received comments include utterance content comments, the broadcaster terminal 1 processes regular comments other than utterance content comments using a voice read-out function. The voice read-out function synthesizes the comments. The generated synthesized voice is distributed as part of the video being distributed. Whether a comment is a utterance content comment can be determined by the poster's name attached to the comment, etc.

[0043] The distributor terminal 1 may translate the speech content comment into another language and transmit it to the comment distribution server 3. The distributor terminal 1 may transmit both the speech content comment and the translated speech content comment to the comment distribution server 3. The viewer terminal 4 may have a function to translate and display comments.

[0044] In the case of a video distribution platform that does not have a timeline-compatible comment function, a video may be generated on the server side in which the spoken content is synthesized and displayed in accordance with the timing of the speech, and the video may be distributed.

[0045] The broadcaster terminal 1 may be used in a remote conference system equipped with a text chat function. Some remote conference systems are equipped with a chat function that allows participants to send and receive text messages in parallel with audio and video conversations. By using the broadcaster terminal 1 in this remote conference system, the content of participants' comments will be displayed in the chat box.

[0046] [Actions and Effects] As described above, the distributor terminal 1 of this embodiment includes a transmitter 15 that transmits the video to be distributed, a converter 12 that converts speech included in the video to text, and a comment transmitter 16 that transmits the text as a comment on the video to be distributed. As a result, the content of the distributor's speech is displayed as a comment on the viewer terminal 4. Furthermore, since the content of the distributor's past speech can be checked in the comment section, it is possible to look back on the content of the distributor's past speech without changing the playback position of the video to be distributed. In this way, by using the existing comment function as subtitles, a subtitle display function can be provided without adding new functions to the video distribution server 2, the comment distribution server 3, and the viewer terminal 4.

[0047] According to this embodiment, by adding time information to the utterance content comment at the time when the provisional recognition result of the speech voice contained in the distributed video was obtained, the utterance content comment is added with time information that matches the timing of the utterance, so that in the case of time-shift viewing, the viewer terminal 4 can display subtitles in accordance with the timing of the utterance.

[0048] According to this embodiment, if the timing of a distributor's speech on a distributed video is delayed from the timing of displaying the speech content comment, the distributor terminal 1 delays the timing of sending the speech content comment to the comment distribution server 3. This allows subtitles to be displayed in accordance with the timing of the speech.

[0049] Any part or all of the functional units described in this disclosure may be implemented by a program. The programs described in this disclosure may be non-temporarily recorded on a computer-readable recording medium and distributed, distributed via a communication line (including wireless communication) such as the Internet, or distributed in a state where they are installed on any terminal. While a person skilled in the art may conceive additional effects and various modifications of the present invention based on the above description, the aspects of this disclosure are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention as derived from the content defined in the claims and their equivalents. For example, what is described in this disclosure as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be implemented by multiple devices. Conversely, what is described in this disclosure as multiple devices (including what is depicted as multiple devices in the drawings) may be implemented by a single device. Alternatively, some or all of the means and functions included in one device (e.g., a server) may be included in another device (e.g., a user terminal). Furthermore, a "system" may be made up of one device, or may be made up of two or more devices (for example, a server and a user terminal, or multiple user terminals).

[0050] Furthermore, not all of the features described in this disclosure are essential requirements. In particular, features described in this disclosure but not in the claims can be considered optional additional features.

[0051] Please note that the applicant is only aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this disclosure, and the present disclosure does not necessarily aim to solve the problems of the disclosed inventions. The problem that the present disclosure aims to solve should be determined by taking into consideration the entire disclosure. For example, if the present disclosure states that a specific configuration achieves a certain effect, it can also be said that the present disclosure solves a problem that is the reverse of the certain effect. However, it is not necessarily intended that such a specific configuration be an essential requirement. [Explanation of symbols]

[0052] 1. Streamer terminal 11 Input section 12 Conversion unit 13 Voice Recognition Unit 14 Encoder 15 Transmitter 16 Comment Submission Section 2. Video distribution server 3. Comment distribution server 4. Viewer terminal

Claims

1. A system for distributing videos and comments on the videos, a viewer terminal that displays the comments on the same screen as the distributed video; a distributor terminal that provides the distribution video; The distributor terminal an input unit for inputting speech and video; a first transmission unit that encodes the speech and video and transmits the video to be distributed; a conversion unit that converts the speech into text in parallel with the encoding; a second transmission unit that transmits the text as a subtitle comment for the distribution video; system.

2. A terminal that provides a streaming video in a system that delivers a streaming video and comments on the streaming video to a viewer terminal, comprising: an input unit for inputting speech and video; a first transmission unit that encodes the speech and video and transmits the video to be distributed; a conversion unit that converts the speech into text in parallel with the encoding; a second transmission unit that transmits the text as a subtitle comment for the distribution video; Terminal.

3. A terminal according to claim 2, The viewer terminal displays the distributed video and the comments on the same screen, The comments include the subtitle comments and viewer comments transmitted from the viewer terminal. Terminal.

4. 4. The terminal according to claim 3, assigning a display attribute to the subtitle comment obtained by converting the speech voice, which is different from that of the viewer comment; Terminal.

5. 3. The terminal according to claim 2, adding time information of a speech section of the speech voice to the comment; Terminal.

6. 3. The terminal according to claim 2, adding time information indicating when the provisional recognition result of the uttered voice is obtained to the comment; Terminal.

7. 3. The terminal according to claim 2, delaying the transmission of the comments in accordance with a delay in the distributed video; Terminal.

8. In a system for delivering a streaming video and comments on the streaming video to a viewer terminal, a terminal for providing the streaming video comprises: Input the speech and video, Encoding the speech and video to transmit the video to be distributed; converting the speech into text in parallel with the encoding; Transmitting the text as a subtitle comment for the distribution video; Sending method.

9. In a system for delivering a streaming video and comments on the streaming video to a viewer terminal, a terminal that provides the streaming video includes: A process of inputting speech and video; A process of encoding the speech and video and transmitting the video to be distributed; converting the speech into text in parallel with the encoding; and transmitting the text as a subtitle comment for the video to be distributed. program.

Images