Content generation device, content generation method, program, and recording medium

Abstract

A publisher terminal (1) is provided with: an input unit (11) for inputting content to be published by a publisher; a comment acquisition unit (12) that acquires comments given to the video published by the video publication server (2); a speech synthesis unit (13) that generates speech on the basis of the comments; a video generation unit (14) that generates character content including character data or a character that performs an action corresponding to the voice; and a video synthesis unit (15) that generates a distribution video in which the character content and the content are superimposed.

Description

Technical Field

[0001] The present disclosure relates to a content generation device, a content generation method, a program, and a recording medium. Background Art

[0002] Services that enable comments to be posted on videos are widely used (Patent Document 1). The posted comments are displayed overlapping in the display area of the video, or in a comment column provided outside the display area of the video. In a so-called live program where a live broadcast is posted, the publisher can communicate with the viewers by reading the comments posted by the viewers.

[0003] Techniques that use mechanical voice to read comments instead of the publisher himself / herself reading the comments are also being used (Non-Patent Document 1).

[0004] In Patent Document 2, the following technique is disclosed: an image in which a virtual image object serving as a user's avatar is overlapped with an image captured by a user terminal device is posted.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Patent No. 6295494 Gazette

[0008] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2020-160645

[0009] Non-Patent Documents

[0010] Non-Patent Document 1: "BouyomiChan", Internet <URL: https: / / chi.usamimi.info / Program / Application / BouyomiChan / > Summary of the Invention

[0011] Problems to be Solved by the Invention

[0012] When the publisher himself / herself reads the comments, sometimes the comments are skipped. Viewers whose comments have been skipped may lose the will to post comments and stop watching the program. By using the technique of Non-Patent Document 1 to read the comments with mechanical voice, the skipping of comments is eliminated, but there is a problem that viewers get bored because it is monotonous synthesized voice.

[0013] The present disclosure has been made in view of the above circumstances, and its object is to generate a more attractive video for posting.

[0014] Means for Solving the Problems

[0015] A content generation device according to one embodiment of the present invention is used to generate content to be published by a content publishing server, and the content generation device comprises: an input unit that inputs content; a comment acquisition unit that acquires comments made on the content published by the content publishing server; a speech synthesis unit that generates speech based on the comments; a generation unit that generates character content including a character or character data that performs an action corresponding to the speech; and a synthesis unit that generates content for publication so that the character content overlaps with the content.

[0016] Effects of the Invention

[0017] According to the present disclosure, it is possible to generate a more attractive video for distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a diagram showing an example of the structure of the video distribution system according to the present embodiment.

[0019] Figure 2 This is a diagram showing an example of the structure of a publisher terminal.

[0020] Figure 3 This is a flowchart of an example of the processing flow of the publisher terminal.

[0021] Figure 4 This is a diagram showing an example of a screen generated by a publisher terminal. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described using the drawings.

[0023] [System Structure]

[0024] Figure 1 1 is a diagram showing an example of the structure of a video distribution system according to the present embodiment. The video distribution system shown in the diagram includes a publisher terminal 1, a video distribution server 2, a comment distribution server 3, and a viewer terminal 4. Each device is connected to each other via a network so as to be communicable. Figure 1 In the figure, only two viewer terminals 4 are shown, but the present invention is not limited to this. There are multiple viewers, and multiple viewer terminals 4 are connected. In addition, only one publisher terminal 1 is shown, but in reality there are multiple publishers, and multiple publisher terminals 1 are connected. Viewers can select the program of the publisher they want to watch and watch it.

[0025] The video publishing server 2 publishes the video received from the publisher terminal 1 to the viewer terminal 4 in real time. Publishing a video in real time is also called live broadcast publishing, live broadcast publishing or streaming media publishing. The video publishing server 2 can store the video received from the publisher terminal 1 and publish the video to the viewer terminal 4 at any time according to the publishing request from the viewer terminal 4. Publishing a video at any time is also called time-shift publishing.

[0026] The comment publishing server 3 receives the comments of the viewer on the video input from the viewer terminal 4, and publishes the received comments in real time to the viewer terminal 4 to which the same video is published. The information of the comments received from the viewer terminal 4 includes the content (string) of the comments, the user ID and the time information. The user ID is the identifier of the user who posted the comment. The time information is the timestamp of the program when the user posted the comment. The comment publishing server 3 can publish the comments to the publisher terminal 1. In addition, the comment publishing server 3 receives the comments input by the publisher from the publisher terminal 1 and publishes them to the viewer terminal 4 as the publisher's comments.

[0027] The comment posting server 3 manages and stores comments for each video. When receiving a posting request from the viewer terminal 4, the video posting server 2 notifies the comment posting server 3 of information identifying the viewer terminal 4 and information identifying the requested video. The comment posting server 3 starts to send comments corresponding to the video to the viewer terminal 4 and receive comments from the viewer terminal 4. Regarding posting of comments, the technology described in Patent Document 1 can be used.

[0028] The viewer terminal 4 is a terminal used by viewers who view programs, and receives and displays videos from the video distribution server 2. When the viewer operates the viewer terminal 4 to select a live broadcast program (live broadcast video) that he wants to watch, the viewer terminal 4 sends a video distribution request to the video distribution server 2. After receiving the distribution request, the video distribution server 2 starts to send the requested video to the viewer terminal 4. As the viewer terminal 4, for example, a personal computer (PC), a smartphone, or a tablet terminal can be used.

[0029] The audience can comment on the live broadcast program while watching it. The audience terminal 4 can display the comments posted on the live broadcast program. Specifically, after the audience inputs the comments to the audience terminal 4, the audience terminal 4 sends the input comments to the comment publishing server 3. The audience terminal 4 publishes the published comments to the publisher terminal 1 and the audience terminal 4 respectively.

[0030] The viewer terminal 4 displays the posted comments. The viewer terminal 4 may display the comments superimposed on the video or in a comment field outside the video display area. The viewer can operate the viewer terminal 4 to turn on / off the display of the comments.

[0031] The publisher terminal 1 is a terminal used by the publisher who publishes the program, and sends the video to be published to the video publishing server 2 in real time. For example, the publisher terminal 1 inputs a video shot by a camera connected to the publisher terminal 1, and superimposes the character video described later on the input video and sends it to the video publishing server 2. The publisher terminal 1 may be equipped with a camera, or may input images from an external device such as a game console. As the publisher terminal 1, for example, a PC, a smartphone, or a tablet terminal may be used.

[0032] The publisher terminal 1 receives comments on a live broadcast program from the comment publishing server 3, generates voice corresponding to the comments, and generates a character video including a character performing an action corresponding to the comments. The action corresponding to the comment is, for example, an action of lip syncing in accordance with the voice generated based on the comment.

[0033] [Structure of publisher terminal]

[0034] Next, an example of the configuration of the publisher terminal 1 will be described.

[0035] Figure 2 1 is a diagram showing an example of the structure of the publisher terminal 1. The publisher terminal 1 shown in the figure includes an input unit 11, a comment acquisition unit 12, a speech synthesis unit 13, a video generation unit 14, a video synthesis unit 15, and a sending unit 16. Each part of the publisher terminal 1 can be composed of a computer with a processing device, a storage device, etc., and the processing of each part can be executed by a program. The program is stored in the storage device of the publisher terminal 1, can be recorded in a non-temporary recording medium that can be read by a computer, such as a magnetic disk, an optical disk, a semiconductor memory, etc., or can be provided through a network.

[0036] The input unit 11 inputs the content that the publisher wants to publish. For example, the content input by the input unit 11 is a video of the publisher himself taken with a camera, a real-life video taken in advance, a computer graphic image drawn by a computer, a screen of an application executed by the publisher terminal 1 or other devices (game consoles, personal computers, smart phones, tablet terminals, etc.) (game screen, drawing software, browser, etc.), or a still image such as a photo or illustration. As long as the video publishing server 2 can publish, the content and form are arbitrary. The input unit 11 can also input multiple contents and synthesize them. For example, in the case where the publisher publishes a video of playing a game, the input unit 11 generates a video in which an image of the publisher taken with a camera is synthesized on the game screen input from the game console. Hereinafter, the content input by the input unit 11 and the content synthesized by the input unit 11 are referred to as contents.

[0037] In addition, the input unit 11 also inputs the sound of the content. When the sound is input from multiple sources, the input unit 11 mixes the sounds. For example, when the publisher publishes a video of playing a game, the input unit 11 mixes the sound of the game with the voice of the publisher. The sound of the game is input from the game machine, and the voice of the publisher is input from the microphone connected to the publisher terminal 1.

[0038] The comment acquisition unit 12 acquires comments posted by viewers on the live broadcast program from the comment distribution server 3. Comments include viewer comments posted by viewers, publisher comments input by publishers, and system comments displayed by the video distribution system. Hereinafter, the term "comments" refers to viewer comments.

[0039] The speech synthesis unit 13 synthesizes (generates) speech based on the comments acquired by the comment acquisition unit 12. The speech synthesis unit 13 can use a general speech synthesis technology. For example, the speech synthesis unit 13 can use a speech synthesis technology that synthesizes text into speech using deep learning technology.

[0040] The speech synthesis unit 13 synthesizes speech from the comments in the order in which the comments arrived and outputs the speech. When the speech synthesis unit 13 finishes outputting the speech, it processes the next comment.

[0041] When a large number of comments are posted, the speech synthesis unit 13 may select comments to be read aloud (generate speech) and read aloud only the selected comments. For example, when a large number of comments are posted, the speech synthesis unit 13 extracts the number of comments that can be read aloud in time according to the order in which the comments arrive, and generates speech based on only the extracted comments. Comments that are not extracted are excluded from the objects to be read aloud. Thereafter, when there is a surplus in processing, the speech synthesis unit 13 restarts the reading of newly posted comments.

[0042] For a long comment, for example, a comment with many characters, the speech synthesis unit 13 performs speech synthesis so that the reading time of the comment falls within a predetermined time. In other words, the speech synthesis unit 13 performs speech synthesis so that the long comment is read quickly.

[0043] The video generation unit 14 generates a character video in which the character performs lip sync based on the speech synthesized by the speech synthesis unit 13. For example, the video generation unit 14 generates the action of the character performing lip sync based on the phoneme information of the synthesized speech. The character video is a video in which the background part other than the character is transparent. The character may be a two-dimensional or three-dimensional character drawn by computer graphics, a hand-drawn character, or a person photographed in real life. The character is not only a person, but may also be an anthropomorphic animal or object.

[0044] The video synthesis unit 15 generates a video for publication by overlapping the character video generated by the video generation unit 14 with the content. The publisher can set the position of the character in the video for publication to any position. The publisher specifies the position and size of the character (the position of overlapping the character video) at the beginning of publication. The publisher can change the position and size of the character during publication. In the case where the content is a real-life video shot in real space, the video synthesis unit 15 can also use augmented reality (AR) technology to configure the character based on the coordinate system of the real space.

[0045] The video synthesis unit 15 may display the comments superimposed on the content, or may not display the comments within the content. The video synthesis unit 15 may display the comments superimposed on the character video, or may display the comments superimposed between the content and the character video. By superimposing the comments on the video in the publisher terminal 1, the display of the comments, the voice of the comments, and the actions of the characters can be synchronized. In addition, even if the comments are not superimposed on the content in the publisher terminal 1, the viewer terminal 4 can obtain the comments from the comment publishing server 3 and display the comments superimposed on the published video.

[0046] The video synthesis unit 15 superimposes the character video on the content, and mixes the voice generated by the voice synthesis unit 13 with the sound of the delivery video.

[0047] The sending unit 16 sends the distribution video to the video distribution server 2 .

[0048] [Actions of the publisher terminal]

[0049] Reference Figure 3 , an example of the processing flow of the publisher terminal 1 is described. From the time the publisher starts to distribute the live broadcast program to the time the publisher ends the distribution, the following processing is repeated.

[0050] In step S11, the publisher terminal 1 inputs the content that the publisher wants to publish.

[0051] In step S12 , the poster terminal 1 obtains the comments posted by the viewer from the review posting server 3 .

[0052] In step S13, the publisher terminal 1 generates speech based on the comments acquired in step S12.

[0053] In step S14, the publisher terminal 1 generates a character video according to the voice generated in step S13.

[0054] Furthermore, the process of step S11 and the processes of steps S12 to S14 may be performed in parallel.

[0055] In step S15 , the publisher terminal 1 superimposes the character video generated in step S14 on the content input in step S11 to generate a character video for distribution.

[0056] In step S16 , the publisher terminal 1 transmits the audio generated in step S13 and the distribution video generated in step S15 to the video distribution server 2 .

[0057] The video distribution server 2 distributes the distribution video to each viewer terminal 4. The comment distribution server 3 receives comments posted by viewers from each viewer terminal 4, and distributes the comments to the distribution terminal 1 and the viewer terminal 4, respectively.

[0058] [Example of video for release]

[0059] Reference Figure 4 , an example of a screen for publishing a video is described. Figure 4 This is an example of a screen generated by the publisher's terminal. Figure 4 In the illustrated screen 100 , comments 110 , 111 and a character 120 are superimposed on a video image captured by a camera.

[0060] Comment 110 is a viewer comment posted by a viewer. The viewer comment moves from the right end to the left end of the screen, for example. Comment 111 is a publisher comment input by a publisher. The publisher comment 111 is displayed at the top of the screen. Although not shown, the system comment is displayed at the bottom of the screen 100.

[0061] The character 120 performs lip-syncing actions in accordance with the voice generated from the comments 110 and 111. This allows a live broadcast program in which the character 120 reads the comments. When the publisher responds to the comments of the audience, it looks as if the publisher has responded to the character 120 reading the comments, so that a more attractive two-way communication can be achieved between the publisher and the audience.

[0062] [Modifications]

[0063] Next, several modified examples of the present embodiment will be described.

[0064] The speech synthesis unit 13 may also perform speech synthesis on the comments with different sound qualities according to the types of comments. For example, the speech synthesis unit 13 may perform speech synthesis on the viewer's comments, the publisher's comments, and the system's comments with different sound qualities, or may perform speech synthesis on only the system's comments with other sound qualities. The speech synthesis unit 13 may also be enabled to learn in a manner of performing speech synthesis with the publisher's voice, and perform speech synthesis on the publisher's comments with the publisher's sound quality. The video generation unit 14 may also generate character videos of different characters for each sound quality. For example, the video generation unit 14 may make the character who reads the viewer's comments different from the character who reads the publisher's comments.

[0065] The speech synthesis unit 13 may also perform speech synthesis on the comments with different voice qualities for each user who made the comments. For example, the speech synthesis unit 13 uses a speech synthesis model that can output multiple (for example, dozens of) voice qualities. When the speech synthesis unit 13 performs speech synthesis on the comments, it stores the correspondence between the user ID and the identification number of the voice quality. In the case where the correspondence between the user ID and the identification number of the voice quality is stored, the speech synthesis unit 13 performs speech synthesis on the comments with the corresponding voice quality. In the case where the correspondence between the user ID and the identification number of the voice quality is not stored, that is, in the case of a new user's comment, the speech synthesis unit 13 establishes a correspondence between the identification number of any voice quality and the user ID, and performs speech synthesis on the comments with the voice quality. In the case where the number of users who make comments is greater than the number of voice qualities, the same voice quality may also be established to correspond to multiple users. The video generation unit 14 prepares characters corresponding to the voice qualities in advance, and generates a character video in which the character corresponding to the voice quality of the speech synthesized by the speech synthesis unit 13 performs lip synchronization.

[0066] The viewer may also specify at least one of the character and voice quality of the person reading his / her comment. For example, the viewer may specify the character and voice quality by providing instructions when posting a comment. The speech synthesis unit 13 may also change the voice quality according to the display method (color, size, display position) of the comment. In this case, the viewer can specify the character and voice quality by the display method of the comment.

[0067] For example, when comments are made at the same time or close to each other, the speech synthesis unit 13 does not perform speech synthesis on the comments one by one, but rather synthesizes and outputs the comments in a speech overlapping manner, and the video generation unit 14 displays multiple characters at the same time.

[0068] The video generation unit 14 may also cause the character to perform an action based on the content of the comment. For example, when the content of the comment is "8888" (a string of 8 consisting of two or more consecutive characters, which means clapping in Japan), the video generation unit 14 generates a character video of the character clapping. At this time, the speech synthesis unit 13 may not output the speech corresponding to "8888", but may output the sound of clapping, or may synthesize a speech of "pa pa". When the content of the comment is "www" (a string of w consisting of one or more consecutive characters, which means laughing in Japan), the video generation unit 14 generates a character video of the character laughing. When the character "w" is assigned to the end of the comment, the video generation unit 14 generates a character video of the character laughing after the comment is read aloud.

[0069] The video generation unit 14 may also cause the character to perform an action corresponding to the posting status of comments (e.g., the number of comments). For example, when a large number of comments are received, the video generation unit 14 generates a character video in which the character performs a panic action. When there are few comments, for example, when no comments are received for more than a predetermined time, the video generation unit 14 generates a character video in which the character performs a leisurely action.

[0070] In the case where gifts can be thrown into a live broadcast program, the video generation unit 14 may also generate a video of a character performing an action of thanking the user when a gift is thrown in. The speech synthesis unit 13 may also synthesize a voice reading the name of the user who threw in the gift. In addition, the video generation unit 14 may also generate a video of a character performing an action corresponding to the performance of the thrown gift. For example, in the case of a performance such as an object falling from the top of the screen, the video generation unit 14 generates a video of a character performing an action of catching the falling object.

[0071] The reading of comments may also be paused during the publisher's speech. For example, when the publisher's voice is input into the microphone, the speech synthesis unit 13 pauses the input of comments and does not perform speech synthesis of the comments. If it is detected that the publisher's utterance has ended, the speech synthesis unit 13 may restart reading the paused comments from the position where the reading was paused, or may read the comments from the beginning. Comments obtained during the publisher's speech may be excluded from the reading object. Alternatively, the speech synthesis unit 13 may temporarily keep the comments obtained during the publisher's speech, and speech synthesize the comments one by one after the publisher's speech.

[0072] The publisher terminal 1 may send character data (e.g., motion data, etc.) for generating a character video. Specifically, the video generation unit 14 generates character data based on synthesized voice, the video synthesis unit 15 overlaps the character data with the content, and the sending unit 16 sends the content overlapped with the character data. In this case, the viewer terminal 4 generates a character video based on the character data, and displays the character video overlapped with the content. The video publishing server 2 may generate a character video, overlap the character video with the content, and send the content overlapped with the character video to the viewer terminal 4. The publisher terminal 1 may send the content and the character data separately.

[0073] In addition, in the present embodiment, the character video is generated by the publisher terminal 1, but the character video may be generated by the viewer terminal 4 and displayed superimposed on the published video. Specifically, the viewer terminal 4 synthesizes speech based on the comments obtained from the comment publishing server 3, generates the character video based on the synthesized speech, displays the character video superimposed on the video received from the video publishing server 2, and outputs the synthesized speech. In the case where the character video is generated by the viewer terminal 4, similarly for the video published in time shift, by also performing speech synthesis and character video on the published comments, the character reading the comments can be displayed to view the video.

[0074] As described above, the publisher terminal 1 of this embodiment includes: an input unit 11, which inputs the content that the publisher wants to publish; a comment acquisition unit 12, which acquires comments posted on the video published by the video publishing server 2; a voice synthesis unit 13, which generates voice based on the comments; a video generation unit 14, which generates a character video including a character performing an action corresponding to the voice; and a video synthesis unit 15, which generates a publishing video in which the character video and the content are overlapped. In this way, a video in which the character reads the comment can be published, so that the desire to publish the comment can be generated. The publisher can publish a video in which the publisher and the character have a conversation by responding to the comments of the audience.

[0075] Description of Reference Numerals

[0076] 1…Publisher terminal

[0077] 11…Input section

[0078] 12…Comment acquisition unit

[0079] 13…Speech Synthesis Department

[0080] 14…Video Production Department

[0081] 15…Video Synthesis Department

[0082] 16…Sending department

[0083] 2…Video publishing server

[0084] 3…Comment publishing server

[0085] 4…Viewer terminal.

Claims

1. A content generation device, which is used to generate content published by a content publishing server, characterized in that: The content generating device comprises: The input part, which contains the input contents; a comment acquisition unit that acquires comments posted on the content distributed by the content distribution server; a speech synthesis unit that generates speech based on the comments; A generating unit that generates character contents including a character or character data that performs an action corresponding to the voice; as well as A synthesis unit generates delivery content by overlapping the character content with the content.

2. The content generating device according to claim 1, characterized in that: The speech synthesis unit generates speech with different sound qualities according to the type of the comment or according to the poster of the comment, The generating unit generates the character content including a character corresponding to the voice quality or data of the character.

3. The content generating device according to claim 2, characterized in that: At least either the voice quality or the role is specified by a poster of the review.

4. The content generating device according to claim 1, characterized in that: The generating unit generates the character content including a character or character data that performs an action corresponding to the content of the comment.

5. The content generating device according to claim 4, characterized in that: When the content of the comment includes a character string of a plurality of consecutive numerals 8, the generating unit generates the character content including a character or character data performing a clapping motion.

6. The content generating device according to claim 1, characterized in that: The generating unit generates the character content including a character or character data that performs an action corresponding to a posting status of the review.

7. The content generating device according to claim 1, characterized in that: The speech synthesis unit temporarily stops generating speech during the speaker's speech.

8. The content generating device according to claim 1, characterized in that: The speech synthesis unit generates speech at a speed corresponding to the length of the content of the comment.

9. A content generation method of a content generation device, wherein the content generation device is used to generate content published by a content publishing server, characterized in that: The content generation method performs the following processing: Input content; Obtaining comments posted on the content published by the content publishing server; generating speech based on the comments; generating character content including a character or character data that performs an action corresponding to the speech; as well as A delivery content is generated in which the character content is overlapped with the content.

10. A program for causing a computer to operate as a content generation device for generating content distributed by a content distribution server, characterized in that: The program causes the computer to execute the following processing: Input content; Obtaining comments posted on the content published by the content publishing server; generating speech based on the comments; generating character content including a character or character data that performs an action corresponding to the speech; as well as A delivery content is generated in which the character content is overlapped with the content.

11. A recording medium storing a program for causing a computer to operate as a content generating device for generating content distributed by a content distribution server, characterized in that: The program causes the computer to execute the following processing: Input content; Obtaining comments posted on the content published by the content publishing server; generating speech based on the comments; generating character content including a character or character data that performs an action corresponding to the speech; as well as A delivery content is generated in which the character content is overlapped with the content.

Images