Webinar video generation system, webinar video generation method, and webinar video generation program

Abstract

Webinar video generation systems, etc., create new, highly reusable webinar videos by replacing the presenters with avatars from the original webinar video. [Solution] The system extracts audio data from the original webinar video, converts the audio data into text data, detects portions of the text data that contain personal information, replaces those portions with predetermined information, generates synthesized speech from the formatted text data including the text data after processing by personal information processing, generates an avatar video in which the mouth moves in sync with the content of the synthesized speech, determines the timing for switching pages of the explanatory materials based on the formatted text data and the content of the explanatory materials, and synthesizes the synthesized speech, avatar video, and the videos of the explanatory materials that are switched according to the timing to generate a new webinar video.

Description

Technical Field

[0001] The present invention relates to a webinar video generation system, a webinar video generation method, and a webinar video generation program, and particularly to a webinar video generation system, a webinar video generation method, and a webinar video generation program suitable for reusing existing webinar videos.

Background Art

[0002] In recent years, with the advancement of Internet lines and video distribution technologies, online seminars, so-called webinars, have been widely used as a marketing activity and information dissemination means for companies. By delivering videos of explanations by presenters and explanations using explanatory materials, as well as audio, either in real-time or on-demand, it is possible to efficiently provide information to a large number of viewers, and thus it is being utilized in various applications such as product / service introductions, technical explanations, training, etc. (see, for example, Patent Document 1).

[0003] For example, Patent Document 1 discloses a technique for collecting and analyzing viewers' reactions (such as "likes" or comments) to distributed content and using it for marketing activities and the like. Such a technique enables companies to streamline communication with customers through webinars.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional webinar videos are generally created in a format where a specific human presenter gives explanations, and there are problems with their reusability and maintainability as video content. Specifically, firstly, if an employee who appeared on a presentation leaves the company, it becomes difficult to continue using the video from the perspective of portrait rights and personal information, resulting in the loss of valuable content assets. Secondly, if the information in a video becomes outdated due to changes in laws or service content, it is difficult to correct only a part of the video; the entire video must be reshot, which presents a significant challenge in terms of time and cost. Thirdly, when human speakers include unnecessary words (fillers) such as "um" and "ah," it becomes difficult for viewers to understand, reducing the efficiency of information transmission. Removing these manually through editing requires advanced skills and time.

[0006] The objective of the technology disclosed in Patent Document 1 is to provide more satisfying and effective content distribution, and it did not consider or adequately consider the reusability and maintainability of the video content. Therefore, the present invention aims to provide a webinar video generation system, a webinar video generation method, and a webinar video generation program that enable the long-term effective use of content as an asset by generating new, highly reusable webinar videos from the original webinar video by replacing the presenters with avatars. [Means for solving the problem]

[0007] The webinar video generation system 10 according to the first embodiment comprises an acquisition unit, an audio extraction unit, an audio recognition unit, a personal information processing unit, an audio synthesis unit, an avatar video generation unit, a timing determination unit, and a video synthesis unit. The acquisition unit acquires the original webinar video and the explanatory materials associated with the webinar video. The audio extraction unit extracts audio data from the webinar video. The audio recognition unit converts the audio data into text data. The personal information processing unit detects portions of the text data that correspond to personal information and replaces those portions with predetermined information. The audio synthesis unit generates synthesized speech from the formatted text data, which includes the text data processed by the personal information processing unit. The avatar video generation unit generates an avatar video in which the mouth moves in synchronization with the content of the synthesized speech. The timing determination unit determines the timing for switching pages of the explanatory materials based on the formatted text data and the content of the explanatory materials. The video synthesis unit synthesizes the synthesized speech, the avatar video, and the videos of the explanatory materials that are switched according to the timing to generate a new webinar video.

[0008] A second embodiment may be a webinar video generation system according to the first embodiment, further comprising a section division unit that divides formatted text data into multiple sections, wherein the section division unit divides the formatted text data into sections based on criteria such as the number of pages of the explanatory material or predetermined time units, an avatar video generation unit generates an avatar video corresponding to each of the divided sections, and a video synthesis unit combines the avatar videos corresponding to each section of the explanatory material to generate a new webinar video.

[0009] In a third embodiment, in the webinar video generation system according to the second embodiment, the avatar video generation unit may generate avatar videos corresponding to each divided section by parallel processing.

[0010] A fourth embodiment is a webinar video generation system according to the second embodiment, further comprising a partial modification unit that modifies some sections of the formatted text data, wherein the partial modification unit regenerates only the avatar video corresponding to the section to be modified and reuses the avatar videos corresponding to other sections.

[0011] The fifth embodiment is a webinar video generation system according to the first embodiment, in which the personal information processing unit detects at least one of the speaker's name, department, or job title contained in the text data and replaces the detected information with pre-configured alternative information.

[0012] A sixth embodiment may further include an information update unit in the webinar video generation system according to the first embodiment, which detects timely information contained in formatted text data and updates the information if it differs from the latest information.

[0013] A seventh embodiment of the webinar video generation system according to the first embodiment further comprises an avatar storage unit that stores avatar images having multiple different appearances, and an avatar selection unit that selects an appropriate avatar image from among the avatar images stored by the avatar storage unit based on the attributes of the product being covered by the webinar or the user's selection, wherein the avatar video generation unit generates an avatar video using the avatar image selected by the avatar selection unit.

[0014] The eighth aspect is a webinar video generation system according to the seventh aspect, in which the avatar selection unit may select an avatar image based on the viewer's attribute information obtained when applying to participate in the webinar.

[0015] The ninth aspect may further include an A / B testing unit in the webinar video generation system according to the seventh aspect that generates multiple new webinar videos using different avatar images for the same formatted text data and compares the viewing performance data of each new webinar video.

[0016] A tenth embodiment is a webinar video generation system according to the first embodiment, further comprising a translation unit that translates formatted text data into a specified language, and a speech synthesis unit that generates synthesized speech in the specified language from the translated text data.

[0017] An eleventh embodiment may be a webinar video generation system according to the first embodiment, wherein the avatar video generation unit generates an avatar video having a green screen background, and the video compositing unit performs a chroma key compositing process that replaces the green screen background with a specified background image.

[0018] The twelfth embodiment is a webinar video generation system according to the first embodiment, further comprising a question and answer unit that generates answer text data to questions received from viewers of the webinar based on formatted text data and explanatory materials, a speech synthesis unit that generates synthesized speech from the answer text data, an avatar video generation unit that generates an avatar video as an answer video to the question based on the synthesized speech generated from the answer text data, and a video synthesis unit that inserts and concatenates the answer videos during the playback of a new webinar video.

[0019] A thirteenth embodiment may further include a co-sponsored video generation unit in the webinar video generation system according to the first embodiment, which combines a webinar video by a human presenter with a new webinar video to generate a co-sponsored video in which a human presenter and an avatar appear together.

[0020] A fourteenth embodiment is a webinar video generation system according to the first embodiment, further comprising a filler removal unit that removes filler words from text data, and a speech synthesis unit that generates synthesized speech from formatted text data including the text data processed by the personal information processing unit and the text data processed by the filler removal unit.

[0021] The webinar video generation method according to the 15th embodiment causes a computer to perform an acquisition step, an audio extraction step, an audio recognition step, a personal information processing step, an audio synthesis step, an avatar video generation step, a timing determination step, and a video synthesis step. In the acquisition step, the original webinar video and explanatory materials associated with the webinar video are acquired. In the audio extraction step, audio data is extracted from the webinar video. In the audio recognition step, the audio data is converted into text data. In the personal information processing step, portions corresponding to personal information are detected from the text data and replaced with predetermined information. In the audio synthesis step, synthesized speech is generated from formatted text data, including the text data processed in the personal information processing step. In the avatar video generation step, an avatar video is generated in which the mouth moves in sync with the content of the synthesized speech. In the timing determination step, the timing for switching pages of the explanatory materials is determined based on the formatted text data and the content of the explanatory materials. In the video compositing step, synthesized audio, avatar footage, and videos of explanatory materials that can be switched at the appropriate timing are combined to generate a new webinar video.

[0022] The webinar video generation program according to the 16th aspect causes a computer to realize an acquisition function, a voice extraction function, a voice recognition function, a personal information processing function, a voice synthesis function, an avatar video generation function, a timing determination function, and a video synthesis function. The acquisition function acquires the original webinar video and the explanatory materials associated with the webinar video. The voice extraction function extracts voice data from the webinar video. The voice recognition function converts the voice data into text data. The personal information processing function detects the portion corresponding to personal information from the text data and replaces the portion with predetermined information. The voice synthesis function generates synthesized voice from the formatted text data including the text data processed by the personal information processing function. The avatar video generation function generates, as an avatar video, a video of an avatar image whose mouth moves in synchronization with the content of the synthesized voice. The timing determination function determines the timing for switching the pages of the explanatory materials based on the formatted text data and the content of the explanatory materials. The video synthesis function synthesizes the synthesized voice, the avatar video, and the video of the explanatory materials switched according to the timing to generate a new webinar video.

Effect of the Invention

[0023] The webinar video generation system according to the present invention includes an acquisition unit, an audio extraction unit, an audio recognition unit, a personal information processing unit, an audio synthesis unit, an avatar video generation unit, a timing determination unit, and a video synthesis unit. Therefore, by replacing the presenter with an avatar in the original webinar video to generate a new webinar video with high reusability, the content can be effectively utilized as an asset for a long time. The acquisition unit acquires the original webinar video and the explanatory materials associated with the webinar video. The audio extraction unit extracts audio data from the webinar video. The audio recognition unit converts the audio data into text data. The personal information processing unit detects the parts corresponding to personal information from the text data and replaces those parts with predetermined information. The audio synthesis unit generates synthesized audio from the formatted text data including the text data processed by the personal information processing unit. The avatar video generation unit generates, as an avatar video, a video of an avatar image whose mouth moves in synchronization with the content of the synthesized audio. The timing determination unit determines the timing for switching the pages of the explanatory materials based on the formatted text data and the content of the explanatory materials. The video synthesis unit synthesizes the synthesized audio, the avatar video, and the video of the explanatory materials switched according to the timing to generate a new webinar video.

[0024] The webinar video generation method and the webinar video generation program according to the present invention, similar to the webinar video generation system according to the present invention, generate a new webinar video with high reusability by replacing the presenter with an avatar in the original webinar video, so that the content can be effectively utilized as an asset for a long time.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is a diagram for explaining the outline of the processing of the webinar video generation system according to the present embodiment. [Figure 2] FIG. 2 is a diagram for explaining the division processing of the original webinar video of the webinar video generation system according to the present embodiment. [Figure 3a]Figure 3a is a diagram illustrating the video generation process and video synthesis process for each section of the webinar video generation system according to this embodiment, and is a diagram illustrating the video generation for each section. [Figure 3b] Figure 3b is a diagram illustrating the video generation process and video synthesis process for each section of the webinar video generation system according to this embodiment, and is a diagram illustrating the synthesis of videos for each section. [Figure 4] Figure 4 shows an example of a diagram illustrating the configuration of an information and communication network surrounding the webinar video generation system according to this embodiment. [Figure 5] Figure 5 shows an example of the hardware configuration of the webinar video generation system according to this embodiment. [Figure 6] Figure 6 is a diagram illustrating an example of the basic functional configuration of the webinar video generation system according to this embodiment. [Figure 7] Figure 7 is a diagram illustrating an example of the overall functional configuration of the webinar video generation system according to this embodiment. [Figure 8a] Figure 8a is a diagram illustrating the various processes of the webinar video generation system according to this embodiment, and shows a specific example of the text formatting process. [Figure 8b] Figure 8b is a diagram illustrating the various processes of the webinar video generation system according to this embodiment, and is a diagram illustrating the determination of the timing for switching pages of the explanatory materials. [Figure 8c] Figure 8c is a diagram illustrating the various processes of the webinar video generation system according to this embodiment, and shows a specific example of updating a portion of the original webinar video. [Figure 9] Figure 9 shows an example of the input and output of the webinar video generation system according to this embodiment. [Figure 10] Figure 10 is a diagram illustrating the chroma key compositing process of the webinar video generation system according to this embodiment. [Figure 11]Figure 11 is a diagram illustrating the overview of an A / B test using different avatars in the webinar video generation system according to this embodiment. [Figure 12] Figure 12 is a flowchart illustrating an example of the basic processing flow of the webinar video generation method and webinar video generation program according to this embodiment. [Figure 13] Figure 13 is a flowchart illustrating an example of the overall processing flow of the webinar video generation method and webinar video generation program according to this embodiment. [Modes for carrying out the invention]

[0026] A webinar video generation system 10 according to one embodiment of the present invention will be described with reference to Figures 1 to 13. In these drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant explanations are omitted. Furthermore, all drawings only show selected components necessary to illustrate the present invention, and other components may be omitted from the illustration. Moreover, the present invention is not limited to the embodiments described below. The webinar video generation system 10 according to this embodiment is a so-called computer, and may be, for example, a workstation, server, personal computer (hereinafter referred to as PC), notebook PC, tablet PC, or smartphone. Furthermore, the webinar video generation system 10 may have a distributed configuration in which some functions are executed on a server in the cloud and other functions are executed on an edge computer. In addition, all functions of the webinar video generation system 10 may be executed on the cloud. The webinar video generation system is a computer system that uses the original webinar video 20 as source material to generate a new webinar video by replacing the presenters with avatars and replacing the voices with synthesized voices 36.

[0027] (Overview of the processing of Webinar video generation system 10) First, with reference to Figure 1, an overview of the processing of the webinar video generation system 10 according to this embodiment will be described. Figure 1 is a diagram illustrating the overview of the processing of the webinar video generation system 10 according to this embodiment.

[0028] As shown in Figure 1, the webinar video generation system 10 performs audio extraction 21 on the original webinar video 20 and obtains audio data 22. The webinar video 20 is a seminar-style video content distributed over the internet, consisting of explanatory video and audio by the presenter, and page images of explanatory materials 30. In the webinar video generation system 10, it is sometimes used as a collective term for the original webinar video 20 in which a human presenter appears, and the new webinar video 41 in which an avatar generated after processing appears. Audio extraction 21 may be performed by the audio extraction unit 63 described later. Then, the webinar video generation system 10 performs speech recognition 23 on the obtained audio data 22 and obtains text data 24. Speech recognition 23 may be performed by the speech recognition unit 64. Explanatory materials 30 is a general term for materials used to explain things to viewers in a webinar, and includes presentation materials (slide format), as well as white papers, sales materials, investor relations materials, web pages, and screen display data. Furthermore, the term "page" as used in explanatory material 30 is not limited to physical divisions such as slides or PDF pages. For example, in continuous (scroll-like) materials such as web pages and text files, the display area (viewport) shown on a single screen, or logical sections divided based on semantic unity, are also included in the concept of a page in this embodiment.

[0029] Next, the webinar video generation system 10 performs personal information processing 25 and filler removal 26 on the text data 24 to obtain formatted text data 27. Personal information processing 25 and filler removal 26 may be performed by a personal information processing unit 65 and a filler removal unit 66, respectively. Next, the webinar video generation system 10 performs timing determination 31 on the formatted text data 27 to determine the timing for switching pages of the explanatory materials 30. Timing determination 31 may be performed by a timing determination unit 71, described later. The webinar video generation system 10 also performs speech synthesis 35 on the formatted text data 27 to obtain synthesized speech 36. Speech synthesis 35 may be performed by a speech synthesis unit 68, described later. Next, the webinar video generation system 10 synchronizes the mouth movements of the avatar image 40 with the synthesized speech 36 to create a lip-sync video 37, and combines it with the video of the explanatory materials 30 that are switched according to the determined timing to generate a new webinar video 41. The creation of the lip-sync video 37 and the generation of the new webinar video 41 may be performed by the video compositing unit 72 described later.

[0030] Furthermore, various processes of the webinar video generation system 10 may be performed using generation AI. For example, speech recognition 23 may be performed using an automatic speech recognition (ASR) model. Image generation may use a general-purpose image generation model. Lip-sync video creation 37 may be performed using speech-driven video generation software. Timing determination 31 may be performed using a large language model (LLM). The Automatic Speech Recognition (ASR) model is a general-purpose model based on the Transformer architecture and has the capability to perform multilingual speech recognition and translation even in noisy environments. The image generation model enables image generation and editing, and in addition to the ability to generate high-resolution images at high speed based on text instructions (prompts), it also has the editing capability to naturally change only the pose and background while maintaining consistency in the characteristics of the same character (face, clothing, etc.). Voice-driven video generation software generates video of a virtual character speaking based on input audio data. This software is characterized by its ability to accurately synchronize not only mouth movements but also head movements, body posture, and facial expressions with the audio, and to generate videos of several minutes or longer, which was difficult with conventional technology. Large-scale language models have the ability to understand the context and logical structure of input text and to accurately identify semantic breaks and topic shifts in complex sentences.

[0031] (Regarding the splitting process of the original webinar video 20 by the webinar video generation system 10) Referring to Figure 2, the process of splitting the original webinar video 20 by the webinar video generation system 10 will be explained. Figure 2 is a diagram illustrating the process of splitting the original webinar video 20 by the webinar video generation system 10 according to this embodiment. As shown in Figure 2, the webinar video generation system 10 performs processing such as speech recognition 23 on the original webinar video 20 to obtain formatted text data 27, and also performs character extraction 32 on the explanatory materials 30 associated with the original webinar video 20 to obtain character information 33 from the explanatory materials 30. The section division unit 69, described later, may also perform the character extraction 32. Next, the webinar video generation system 10 divides the original webinar video 20 into pages, such as slides, based on the formatted text data 27 and character information 33. Dividing 45 may also be performed by the section division unit 69 described later. In dividing 45, the formatted text data 27 may be divided into sections without considering the timing of page changes in the original webinar video 20, and appropriate pages, such as slides, may be assigned according to the content. In this way, natural page assignment of slides, etc., can be achieved in dividing 45. A section refers to a division that constitutes the entire webinar video, representing the smallest unit for editing and management. Specifically, a section is a division that corresponds to one or more pages of slides in the explanatory materials 30, or a series of speeches that conclude a particular topic. In the webinar video generation system 10, each section is treated as an independent data structure capable of speech synthesis 35 and video generation, thereby enabling partial updates (by the partial modification section 73 described later) that modify or replace only specific sections without having to rebuild the entire webinar video.

[0032] The original webinar video 20 and explanatory materials 30 will be divided into sections A, B, and C, for example, as shown in Figure 2. The divided sections will be reviewed by both AI (artificial intelligence) and humans. The original webinar video 20 will be divided into original webinar video 20a, original webinar video 20b, and original webinar video 20c. Explanatory materials 30 will be divided into explanatory materials 30a, explanatory materials 30b, and explanatory materials 30c. The formatted text data 27 will be divided into formatted text data 27a, formatted text data 27b, and formatted text data 27c. Section A will include the original webinar video 20a, explanatory materials 30a, and formatted text data 27a. Section B will include the original webinar video 20b, explanatory materials 30b, and formatted text data 27b. Section C shall include the original webinar video 20c, explanatory materials 30c, and formatted text data 27c. In the AI-driven content review 46, for example, it determines that Section A is a standard presentation and can be used as is (46a in Figure 2), while it determines that Section B contains personal information (self-introduction) and therefore needs to be deleted as a whole (46b in Figure 2). For Section C, since it contains timely information (such as campaigns), an alert 43 is issued, and a human decides whether to replace the content if necessary.

[0033] (Regarding video generation and video compositing processes for each section) Referring to Figure 3, the section-by-section video generation process and video synthesis process of the webinar video generation system 10 will be described. Figure 3 is a diagram illustrating the section-by-section video generation process and video synthesis process of the webinar video generation system 10 according to this embodiment, with Figure 3a illustrating the section-by-section video generation and Figure 3b illustrating the section-by-section video synthesis.

[0034] As shown in Figure 3a, the webinar video generation system 10 performs speech extraction 21 and speech recognition 23 on each section of the original webinar video 20 to obtain section-specific text data 24a of the original webinar video 20. The division of the original webinar video 20 into sections may be performed by a section division unit 69 described later. The section division unit 69 may divide the original webinar video 20 into sections based on formatted text data 27 of each section divided on a page-by-page basis from the explanatory materials 30. The webinar video generation system 10 performs text formatting (filler removal 26 and personal information processing 25) on the section-specific text data 24a to obtain section-specific formatted text data 28, and then performs speech synthesis 35 on the section-specific formatted text data 28 to obtain section-specific synthesized speech 36. Next, the webinar video generation system 10 generates section-specific avatar videos 39, which are videos of avatar images 40 whose mouth movements are synchronized with the content of the section-specific synthesized speech 36. The webinar video generation system 10 divides the explanatory materials 30 associated with the original webinar video 20 into pages 34, such as slides, for each section. This division may be performed by the section division unit 69 described later. The webinar video generation system 10 combines the avatar videos 39 for each section and the pages 34, such as slides, for each section to generate new webinar videos 41 for each section. This generation may be performed by the video compositing unit 72.

[0035] As shown in Figure 3b, the webinar video generation system 10 combines the newly generated webinar videos for each section to produce a completed new webinar video 41 53. Specifically, the webinar video generation system 10 combines the new webinar video 50a for the first section 50, the new webinar video 51a for the second section 51, and the new webinar video 52a for the third section 52 in this order to produce a completed new webinar video 41 53. If there are sections 4 and beyond, the webinar video generation system 10 combines the new webinar videos from the fourth section onward in the order of playback of the sections to produce a completed new webinar video 41 53.

[0036] (Configuration of the information and communication network 11 surrounding the webinar video generation system 10) Referring to Figure 4, the configuration of the information and communication network 11 surrounding the webinar video generation system 10 according to this embodiment will be described. Figure 4 is a diagram showing an example of the configuration of the information and communication network 11 surrounding the webinar video generation system 10 according to this embodiment. The information and communication network 11 is a communication infrastructure that includes the Internet and a LAN (Local Area Network). The webinar video generation system 10 connects to the user terminal 12, the database server 13, and the generation AI server 14 via the information and communication network 11. Note that, for convenience, Figure 4 shows one webinar video generation system 10, one user terminal 12, one database server 13, and one generation AI server 14, but this is not limited to this configuration, and multiple units of each may be in operation.

[0037] User terminal 12 refers to an information processing device used by a user of the webinar video generation system 10, and may be a PC, notebook PC, tablet PC, or smartphone. User terminal 12 may be used for inputting and operating data to the webinar video generation system 10, or for receiving new webinar videos 41 generated by the webinar video generation system 10. Database server 13 refers to the collective term for the original webinar video DB (Database) 56, explanatory materials DB 57, and new webinar video DB 58, which will be described later. The database server 13 may consist of a single device or multiple devices. Furthermore, the database server 13 may be provided and operated by the operator of the webinar video generation system 10, or it may utilize a subscription service provided by another operator. Additionally, the webinar video generation system 10 may connect directly to the database server 13 without going through the information and communication network 11, or it may include the database server 13 as part of its own configuration.

[0038] The generation AI server 14 is a general term for a server computer or group of servers (such as a cloud system) that is equipped with a machine learning model such as deep learning and provides various AI processing functions to the webinar video generation system 10. In this embodiment, the generation AI server 14 may provide AI processing functions for tasks such as voice extraction 21, voice recognition 23, personal information processing 25, filler removal 26, section division 29, timing determination 31, character extraction 32, speech synthesis 35, lip-sync video creation 37, avatar video generation 38, AI-based content review 46, and translation. The generation AI server 14 may be operated by a single operator, or it may be provided by servers operated by multiple operators. Furthermore, the webinar video generation system 10 may connect directly to the generation AI server 14 without going through the information and communication network 11, or it may incorporate the functions of the generation AI server 14 as part of its own functions.

[0039] (Regarding the hardware configuration of the webinar video generation system 10) Referring to Figure 5, an example of the hardware configuration of the webinar video generation system 10 according to this embodiment will be described. Figure 5 is a diagram showing an example of the hardware configuration of the webinar video generation system 10 according to this embodiment. The webinar video generation system 10 includes a communication interface 10a, ROM (Read Only Memory) 10b, RAM (Random Access Memory) 10c, storage unit 10d, arithmetic unit 10e, and input / output interface 10f, etc.

[0040] The communication interface 10a has the function of sending and receiving data handled by the webinar video generation system 10 to and from other devices via the information and communication network 11. The communication interface 10a is an interface for communicating with the user terminal 12, the database server 13, and the generation AI server 14, etc.

[0041] The storage unit 10d can be used as a storage device for the webinar video generation system 10 and can be composed of, for example, a hard disk drive, a solid state drive, and flash memory. Furthermore, the storage unit 10d can also be configured using cloud storage. The storage unit 10d stores the webinar video generation program described later, as well as data acquired and generated by the webinar video generation system 10. The storage unit 10d may also be used as a database server 13. Furthermore, the memory unit 10d stores the OS (Operating System), various other applications, and various data used by those applications that are necessary for the webinar video generation system 10 to operate. The OS is a type of basic application built into the webinar video generation system 10 and has the function of controlling the basic functions of the webinar video generation system 10.

[0042] The arithmetic unit 10e may include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc., and may be implemented by logic circuits (hardware) formed on an integrated circuit (IC) chip, an LSI (Large Scale Integration), etc., or by dedicated circuits. The input / output interface 10f transmits and receives data to and from external devices of the webinar video generation system 10. External devices refer to input devices 10g and output devices 10h that input and output data to and from the webinar video generation system 10. Input devices 10g include, for example, a keyboard and mouse, while output devices 10h include, for example, a monitor, printer, and speakers.

[0043] The webinar video generation system 10 stores the webinar video generation program in ROM 10b or storage unit 10d and loads the webinar video generation program into main memory, which is composed of RAM 10c or the like. The arithmetic unit 10e accesses the main memory into which the webinar video generation program has been loaded and executes the webinar video generation program. The arithmetic unit 10e realizes each functional part of the webinar video generation system 10 by executing the program stored in storage unit 10d or ROM 10b.

[0044] (Regarding the functional configuration of the webinar video generation system 10) An example of the functional configuration of the webinar video generation system 10 will be described with reference to Figures 6 and 7. Figure 6 is a diagram illustrating an example of the basic functional configuration of the webinar video generation system 10 according to this embodiment, and Figure 7 is a diagram illustrating an example of the overall functional configuration of the webinar video generation system 10 according to this embodiment. The webinar video generation system 10, through the execution of a webinar video generation program described later by the calculation unit 10e, realizes the basic functional configuration of functional units such as an acquisition unit 62, an audio extraction unit 63, an audio recognition unit 64, a personal information processing unit 65, an audio synthesis unit 68, an avatar video generation unit 70, a timing determination unit 71, and a video synthesis unit 72. Furthermore, the webinar video generation system 10, through the execution of the webinar video generation program by the calculation unit 10e, realizes, as an overall functional configuration, functional units such as an avatar storage unit 60, an avatar selection unit 61, an acquisition unit 62, an audio extraction unit 63, an audio recognition unit 64, a personal information processing unit 65, a filler removal unit 66, a translation unit 67, an audio synthesis unit 68, a section division unit 69, an avatar video generation unit 70, a timing determination unit 71, an audio synthesis unit 72, a partial modification unit 73, an A / B test execution unit 74, a question and answer unit 75, a co-sponsored video generation unit 76, and an information update unit 77.

[0045] (Avatar storage unit 60) The avatar storage unit 60 stores avatar images 40 having multiple different appearances. The avatar storage unit 60 stores multiple avatar images 40 in the memory unit 10d. Here, the stored avatar images 40 are not limited to still image data, but may also include 3D model data, face mesh data, or texture data for generating lip-syncing and facial expression changes. The avatar storage unit 60 may also manage each avatar image 40 by associating it with a corresponding voice model (voice quality, tone) and attribute tags (e.g., sincere, energetic, professional, female, male, etc.) as metadata. This allows the avatar selection unit 61, described later, to easily extract the most suitable avatar that matches the image of the product or target audience.

[0046] (Avatar selection section 61) The avatar selection unit 61 selects an appropriate avatar image 40 from the avatar images 40 stored by the avatar storage unit 60, based on the attributes of the product being offered in the webinar or the user's selection. Specifically, the avatar selection unit 61 refers to a mapping table (not shown) that defines the correspondence between product categories (e.g., finance, beauty, IT, etc.) and the impression that an avatar gives (e.g., sincerity, friendliness, innovation, etc.), and automatically recommends or selects the avatar image 40 that is best suited to the specified product. The avatar selection unit 61 may select an avatar image 40 based on the viewer's attribute information obtained when they apply to participate in the webinar. The avatar selection unit 61 may also select an avatar image 40 that is predicted to be most favorably received by that viewer group, based on past viewing history data, etc., using the viewer's attribute information (age, gender, occupation, etc.) obtained when they apply to participate in the webinar.

[0047] (Acquisition part 62) The acquisition unit 62 acquires the original webinar video 20 and the explanatory materials 30 associated with the webinar video 20. The webinar video generation system 10 includes an original webinar video DB 56 and an explanatory materials DB 57. Users of the webinar video generation system 10 may pre-store the original webinar video 20 in the original webinar video DB 56 and the explanatory materials 30 in the explanatory materials DB 57. The acquisition unit 62 may acquire the original webinar video 20 from the original webinar video DB 56 and the explanatory materials 30 associated with the webinar video 20 from the explanatory materials DB 57. The acquisition unit 62 may acquire the explanatory materials 30 associated with the original webinar video 20 by performing audio and video analysis on the webinar video 20. Even if only the original webinar video 20 exists and there are no files or paper materials related to the presentation associated with the original webinar video 20, the acquisition unit 52 can acquire the explanatory materials 30 by performing audio or video analysis on the original webinar video 20. The process of acquiring explanatory materials is not limited to receiving existing material files from external devices or users, but is a broad concept that also includes processes of generating, extracting, or reconstructing material data within the webinar video generation system 10. Specifically, in the first embodiment, the acquisition unit 62 may acquire existing file data created in the form of presentation software files, electronic document files, image file groups, etc., from the user terminal 12 or explanatory material DB 57 by receiving or downloading it via the information and communication network 11. In the second embodiment, the acquisition unit 62 may analyze the video data of the original webinar video 20, detect page portions such as slides (screen sharing areas, etc.) that are visible in the video using image recognition processing, and generate and acquire new image data for explanatory materials 30 by cutting out (capturing) those areas as still images. In this case, since the original webinar video 20 itself becomes the source of explanatory materials 30, the user does not need to prepare separate material files. In a third embodiment, the acquisition unit 62 may automatically generate new explanatory materials (summary pages or diagram pages) corresponding to the content of the webinar, based on the audio data 22 or text information 33 within the video of the original webinar video 20, using an algorithm such as a generation AI, and acquire them. In a fourth embodiment, instead of acquiring the actual data of the explanatory materials 30, the acquisition unit 62 may acquire access rights or URL information (link information) indicating the storage location on cloud storage, and refer to the data for rendering the video from that location at the necessary time. Thus, the acquisition unit 62 has the function of preparing explanatory materials 30 (or a group of images equivalent thereto) in a format usable by the video synthesis unit 72 by any of the following methods, or a combination thereof: receiving existing files, extracting images from video, generating new materials using a generation AI, or external referencing. Here, "explanatory material 30" is a comprehensive concept referring to all data used to convey visual information to the audience in a webinar. This includes not only slide-format document files (data with a page concept), but also a series of image data, frame images extracted from a video stream, web pages (HTML / CSS / SVG), or visual objects rendered in real time from text data. Furthermore, explanatory material 30 may include not only still images, but also videos, animation effects, 3D models, or AR (augmented reality) objects.

[0048] (Voice extraction unit 63) The audio extraction unit 63 extracts audio data from the webinar video 20. The audio extraction unit 63 separates and extracts the audio track from the original webinar video 20 acquired by the acquisition unit 62, and extracts audio data 22. Specifically, the audio extraction unit 63 uses a media processing library such as FFmpeg (Fast Forward Moving Picture Experts Group) to separate only the audio stream from the video file (MP4, MOV, etc.). At this time, in order to maximize the accuracy of the subsequent speech recognition processing (processing by the speech recognition unit 64), the audio extraction unit 63 performs downsampling processing to convert the sampling rate to a predetermined frequency (e.g., 16kHz), normalization processing to adjust the volume level to a certain range, and noise reduction processing to remove background noise, and then outputs it as audio data 22 in a general-purpose audio format (e.g., MP3, WAV, etc.). FFmpeg is a free and open-source software project for recording, converting, and playing back video and audio.

[0049] (Voice recognition unit 64) The speech recognition unit 64 converts the speech data 22 into text data 24. The speech recognition unit 64 performs speech recognition processing on the audio data 22 extracted by the audio extraction unit 63 and converts it into text data 24. Specifically, the speech recognition unit 64 uses an automatic speech recognition (ASR) model based on the Transformer architecture, etc. The text data 24 output by the speech recognition unit 64 is structured data (e.g., in JSON format) to which timestamp information indicating the start and end times is attached to each spoken word or phrase. The timestamp information functions as essential metadata for synchronizing text and audio, and consequently for determining the timing of page changes, such as slides, as described later.

[0050] (Personal Information Processing Section 65) The personal information processing unit 65 detects portions of the text data 24 that constitute personal information and replaces those portions with predetermined information. The personal information processing unit 65 detects the portion corresponding to personal information from the text data 24 converted by the speech recognition unit 64 and replaces that portion with predetermined information. The personal information processing unit 65 may detect at least one of the speaker's name, department, or job title contained in the text data 24 and replace the detected information with pre-configured alternative information. In detail, the personal information processing unit 65 processes information in conjunction with a large-scale language model that has contextual understanding capabilities. The personal information processing unit 65 identifies names such as "Suzuki" using named entity recognition (NER), and if it determines from the context that it refers to the speaker, it replaces it with a pre-configured avatar name (e.g., "Sato").

[0051] (Filler removal section 66) The filler removal unit 66 removes filler words from the text data. The filler removal unit 66 removes filler words from the text data converted by the speech recognition unit 64. In detail, the filler removal unit 66, like the personal information processing unit 65, processes data in cooperation with a large-scale language model that has contextual understanding capabilities. The filler removal unit 66 not only removes unnecessary words such as "um" and "uh," but also corrects grammatical inconsistencies resulting from the removal (such as the connection of particles) into natural written language using the LLM, generating formatted text data 27. At this time, the LLM is given the constraint that it "does not change the semantic content of the original utterance," thereby ensuring the accuracy of the information. Filler words refer to unnecessary words and hesitations, such as "um" and "uh," that are unconsciously uttered by speakers during their speech and do not have any substantial meaning in the context. In the webinar video generation system 10, the filler removal unit 66 automatically detects these and deletes them or corrects them to natural conjunctions that are in context, thereby enabling fluent speech by the avatar.

[0052] (Translation Section 67) The translation unit 67 translates the formatted text data 27 into the specified language. The speech synthesis unit 68 may generate synthesized speech 36 in the specified language from the translated text data. The translation unit 67 may translate the formatted text data 27 into the specified language using the LLM provided by the generation AI server 14. The translation unit 67 may also have a function to accurately translate specialized terms by referring to a glossary database (not shown) containing industry or company terminology. This enables multilingual deployment that maintains accuracy of content and synchronization with video, even if the original speaker does not speak the language.

[0053] (Speech synthesis unit 68) The speech synthesis unit 68 generates synthesized speech 36 from the formatted text data 27, which includes the text data processed by the personal information processing unit 65. The speech synthesis unit 68 may generate synthesized speech 36 from formatted text data 27 which includes the text data processed by the personal information processing unit 65 and the text data processed by the filler removal unit 66. The speech synthesis unit 68 generates synthesized speech 36 from the formatted text data 27 after processing by the personal information processing unit 65 and the filler removal unit 66. Specifically, the speech synthesis unit 68 uses a text-to-speech (TTS) engine to dynamically control the speaking speed, intonation, and pauses based on punctuation and contextual information in the formatted text data 27. Furthermore, by applying a voice model that matches the avatar's character (gender, age, trustworthiness, etc.), it generates synthesized speech 36 (MP3, etc.) with a natural, human-like prosody that does not cause discomfort to the listener.

[0054] (Section division 69) The section division unit 69 divides the formatted text data 27 into multiple sections. The section division unit 69 divides the formatted text data 27 into multiple sections based on the text data 24 and explanatory materials 30, and further divides the original webinar video 20 into multiple sections. Specifically, the section division unit 69 analyzes the semantic similarity between the character information 33 extracted from the explanatory materials 30 by OCR (Optical Character Recognition) and the speech-recognized text data 24 to identify turning points in the topic. Based on these turning points, the section division unit 69 defines, for example, a page such as a slide, or a length of about 1 to 2 minutes, as one "section," and uses this as the processing unit for video generation. As a result, the webinar video generation system 10 can perform high-load processing such as avatar video generation, described later, in parallel for each section, achieving efficiency that allows even long webinar videos to be generated in a short time.

[0055] (Avatar video generation unit 70) The avatar video generation unit 70 generates an avatar video 39, which is an avatar image 40 whose mouth movements are synchronized with the content of the synthesized speech. The avatar video generation unit 70 generates an avatar video 39 in which the mouth moves in sync with the content of the synthesized voice 36 generated by the voice synthesis unit 68. Specifically, the avatar video generation unit 70 first uses an image generation AI to generate a source image that will become the appearance of the avatar based on the avatar image 40. At this time, the background color is set to a specific color (for example, green screen: RGB0,255,0) to facilitate the subsequent compositing process (chroma key compositing process). Next, the avatar video generation unit 70 uses a voice-driven video generation AI to input the synthesized voice 36 to the source image and generates a lip-sync video synchronized with the voice waveform (lip-sync motion creation 37 in Figure 1). The voice-driven video generation AI does not simply open and close the mouth, but employs "Sparse-frame video dubbing" technology to infer and add head movements (head pose), gaze, blinking, and facial muscle movements according to the nuances of the voice, outputting a lifelike avatar video 39. The section division unit 69 divides the formatted text data 27 of the explanatory materials 30 into sections based on criteria such as page units of slides or predetermined time units, the avatar video generation unit 70 generates avatar videos 39 corresponding to each divided section, and the video synthesis unit 72 combines the avatar videos 39 corresponding to each section of the explanatory materials 30 to generate a new webinar video 41. The avatar video generation unit 70 may generate avatar videos 39 corresponding to each divided section by parallel processing. The avatar video generation unit 70 may generate the avatar video 39 using the avatar image 40 selected by the avatar selection unit 61. The avatar video generation unit 70 generates an avatar video 39 having a green screen background 80, and the video compositing unit 72 may perform a chroma key compositing process 79 to replace the green screen background 80 with a specified background image.

[0056] (Timing determination unit 71) The timing determination unit 71 determines the timing for switching pages of the explanatory document 30 based on the formatted text data 27 and the content of the explanatory document 30. The timing determination unit 71 determines the timing for switching pages of the explanatory materials 30 based on the content of the synthesized speech 36 and the content of the explanatory materials 30, and generates timing information. In detail, the timing determination unit 71 analyzes the semantic correspondence (Semantic Matching) between each sentence of the formatted text data 27 and the text information contained in each page of the explanatory materials 30 using LLM. The LLM determines from the context which slide or page's explanation the currently being read content corresponds to, and calculates the optimal time (timestamp) for switching slides or pages. This makes it possible to create content in which the audio and video materials are perfectly synchronized without requiring manual adjustments.

[0057] (Video compositing section 72) The video synthesis unit 72 synthesizes the speech synthesis 36, the avatar video 39, and the video of the explanatory materials 30, which can be switched according to the timing, to generate a new webinar video 41. The video compositing unit 72 combines the avatar video 39, the explanatory materials 30, and the timing information to generate a video of a substitute presenter (a new webinar video 41). Specifically, the video compositing unit 72 performs chroma key compositing on the avatar video 39, replacing the green screen background 80 with a specified background image (e.g., a virtual office). Then, while switching page images such as slides according to the timing information, it combines the avatar video 39 and the synthesized voice 36 on the timeline. Finally, it concatenates the video fragments of each section, which were divided and generated in parallel by the section division unit 69, in chronological order, encodes them into a single video file (MP4, etc.), and outputs it.

[0058] (Partial modification section 73) The partial modification section 73 modifies a portion of the formatted text data 27. The partial modification unit 73 regenerates only the avatar video 39 corresponding to the section being modified, and reuses the avatar video 39 corresponding to other sections. When a portion of the formatted text data 27 is modified, the partial modification unit 73 regenerates only the avatar video 39 corresponding to that section. Specifically, when it becomes necessary to correct information within the original webinar video 20 (for example, the statement "2024"), the user only modifies the text data. The partial modification unit 73 identifies only the section containing the user's modification (for example, a one-minute segment) and re-executes the speech synthesis 35, lip-sync video creation 37, and video synthesis only for that segment. For other sections that remain unchanged, the existing generated data is reused and concatenated as is, significantly reducing computational resources and time compared to regenerating the entire original webinar video 20 from scratch.

[0059] (AB test execution unit 74) The A / B test execution unit 74 generates multiple new webinar videos 41 using different avatar images 40 with the same formatted text data 27, or using multiple formatted text data 27 with different content (for example, different scenarios with different selling points) with the same avatar image 40, and compares the viewing performance data of each new webinar video 41. In detail, the A / B testing unit 74 compares the viewing performance data for each of the generated variations of the new webinar video 41, such as a "young male avatar" version and a "veteran female avatar" version, or a "script focusing on feature appeals" version and a "script focusing on case studies" version. This makes it possible to quantitatively acquire and analyze marketing data such as which avatar image 40 attributes or script configurations have high viewer retention rates or conversion rates (CVR). Furthermore, the A / B testing unit 74 works in conjunction with the translation unit 67 to translate the formatted text data 27 into multiple languages, generates multiple new webinar videos 41 corresponding to each language, and compares the viewing performance data for each new webinar video 41 to determine which language's new webinar video 41 is effective, thus enabling functions for global expansion. The A / B test execution unit 74 may collect viewing performance data from playback logs (event information such as playback start, pause, exit point, playback completion, and playback speed change) transmitted from the video playback player running on the user terminal 12, or from behavioral logs (conversion information such as clicking the document download button and answering questionnaires) obtained through tracking tags embedded in the webinar distribution page.

[0060] (Q&A Session 75) The Q&A section 75 generates text data of answers to questions received from webinar viewers based on the formatted text data 27 and explanatory materials 30. The speech synthesis unit 68 generates synthesized speech 36 from the answer text data, the avatar video generation unit 70 generates avatar video 39 as an answer video to the question based on the synthesized speech 36 generated from the answer text data, and the video synthesis unit 72 inserts and concatenates the answer video during the playback of the new webinar video 41. In detail, the question and answer unit 75 may generate the answer text data using an LLM that holds the formatted text data 27 and explanatory materials 30 as a knowledge base (RAG: Retrieval-Augmented Generation).

[0061] (Co-sponsored by Video Production Department 76) The co-sponsored video generation unit 76 combines a webinar video featuring a human presenter with a new webinar video 41 to generate a co-sponsored video in which a human presenter and an avatar appear together. The co-sponsored video generation unit 76 combines a video of a human presenter with a video of an avatar appearing on their behalf (new webinar video 41) to generate a co-sponsored video. In detail, the co-sponsored video generation unit 76 can realize a hybrid configuration in which, for example, the avatar handles the moderation part, while the human developer handles the core technical explanation part. Based on timing information and formatted text data 27 that becomes the scenario, the co-sponsored video generation unit 76 can switch the main display on the screen between the avatar and the human, or display them simultaneously in a split screen format, such as in a dialogue format.

[0062] (Information Update Department 77) The information update unit 77 detects current events information contained in the formatted text data 27 and updates it if it differs from the latest information. In detail, the information update unit 77 analyzes timely information contained in the formatted text data 27 (for example, date information for a limited-time campaign or the name of the target product). The information update unit 77 compares the timely information with pre-set information and detects the obsolescence of the timely information if there are any discrepancies. If obsolescence of the timely information is detected, the unit either notifies the user of an alert 43 (see Figure 2) or automatically corrects the text data based on reliable, up-to-date information and prompts the partial modification unit 73 to regenerate it.

[0063] With the above configuration, the webinar video generation system 10 according to this embodiment decomposes (structures) unstructured video data into elements such as text, audio, pages, and avatars, and implements a pipeline process that reconstructs it by linking multiple cutting-edge generative AI models.

[0064] (Regarding the various processes of the webinar video generation system 10) Referring to Figure 8, the various processes of the webinar video generation system 10 according to this embodiment will be explained. Figure 8 is a diagram for explaining the various processes of the webinar video generation system 10 according to this embodiment, with Figure 8a showing a specific example of the text formatting process, Figure 8b showing a diagram for explaining the timing determination 31 for switching pages of the explanatory materials 30, and Figure 8c showing a specific example of partial updating of the original webinar video 20.

[0065] Refer to Figure 8a to explain a specific example of text formatting. As shown in Figure 8a, the initial text data 24 (example of unprocessed text data 24b) generated by speech recognition 23 (speech recognition unit 64) may contain filler words such as "um" and "uh" that are included in the original speaker's speech, as well as personal information such as the speaker's real name (e.g., Yamada). The filler removal unit 66 detects filler words from the unprocessed text data example 24b and deletes them or modifies them into natural conjunctions that fit the context. At the same time, the personal information processing unit 65 analyzes named entities in the unprocessed text data example 24b, identifies personal information (Yamada), and replaces it with predetermined information such as a pre-set alternative name for the avatar (a fictitious name or the company's avatar, e.g., Sato) (example of formatted text data 27d). Through these processes, the formatted text data example 27d is generated as a fluent and anonymized script suitable for avatar speech. This makes it possible to continue using the content as an organizational asset even if the speaker of the original webinar video 20 leaves the company.

[0066] Referring to Figure 8b, the timing determination 31 for switching pages in explanatory material 30 will be explained. As shown in Figure 8b, the timing determination unit 71 determines the break in the waveform data of the synthesized speech 36 between the explanatory speech 36a on page 1 34a and the explanatory speech 36b on page 2 34b of the explanatory material 30 as the timing for switching from the display of page 1 34a to the display of page 2 34b, and generates timing information describing that timing. By using the timing information, the webinar video generation system 10 can generate a new webinar video 41 in which the content of the synthesized speech 36 and the display of each page are perfectly synchronized, without the need for manual editing.

[0067] Figure 8c illustrates a specific example of a partial update to the original webinar video 20. As shown in Figure 8c, consider a scenario where, due to legal revisions or fiscal year updates, it becomes necessary to update some information in the original webinar video 20 (for example, the numerical value "2024" to "2025"). Traditionally, this would require reshooting the entire original webinar video 20, but in the webinar video generation system 10, the partial modification unit 73 comes into play. When the user or the information update unit 77 modifies the relevant portion in the formatted text data 27, the partial modification unit 73 identifies only the specific section (for example, the second section) that contains the modified portion. The webinar video generation system 10 then re-executes the speech synthesis 35, avatar video generation, and lip-sync video creation 37 only for the second section, and regenerates the updated video fragment 54. Finally, the video synthesis unit 72 combines the existing video fragment 53a of the first section, the updated video fragment 54 of the second section, and the existing video fragment 53c of the third section to generate the completed video 53 of the new webinar video 41. This system makes it possible to provide webinar videos that are always up-to-date, while minimizing the computational resources and time required for video generation.

[0068] (Examples of input and output for Webinar video generation system 10) Referring to Figure 9, an example of the input and output of the webinar video generation system 10 according to this embodiment will be described. Figure 9 is a diagram showing an example of the input and output of the webinar video generation system 10 according to this embodiment. As shown in Figure 9, the input to the webinar video generation system 10 is the original webinar video 20 and the explanatory materials 30 used in the video. The webinar video generation system 10 performs a series of processes (pipeline processing) on ​​these input data, including the aforementioned audio extraction 21, speech recognition 23, personal information processing 25, filler removal 26, timing determination 31, speech synthesis 35, lip-sync video creation 37, and video synthesis. As output, it outputs a new webinar video 41 in which the video of the human presenter is replaced with an avatar video 39 and the audio is replaced with synthesized voice 36. At this time, the image data of the input explanatory materials 30 is incorporated into the new webinar video 41 so as to automatically switch in synchronization with the synthesized voice 36 based on the timing information generated by the timing determination unit 71. As a result, the user can obtain a new webinar video 41 (a video with a substitute presenter) simply by inputting the original webinar video 20 and explanatory materials 30.

[0069] (Regarding the chroma key compositing process 79 of the webinar video generation system 10) Referring to Figure 10, the chroma key compositing process 79 of the webinar video generation system 10 will be described. Figure 10 is a diagram illustrating the chroma key compositing process 79 of the webinar video generation system 10 according to this embodiment. As shown in Figure 10, when the avatar video generation unit 70 generates the avatar video 39, it outputs it with a green screen background 80 in which the background area other than the avatar's area is filled with a specific hue (for example, a highly saturated green such as RGB value (0,255,0)). The video compositing unit 72 acquires this avatar video 39 and a replacement background image 81 specified by the user (for example, a still image of a virtual office, a company brand logo, or a video background). The video compositing unit 72 performs chroma key compositing processing 79, making the area corresponding to the green screen background 80 transparent from the avatar video 39 using the color information as a key, and compositing the replacement background image 81 behind it. As a result, the resulting composite video 82 is a natural video that makes it appear as if the avatar is giving a presentation in the specified space. This mechanism allows the webinar video generation system 10 to instantly and inexpensively change only the background, even after generating an avatar video 39, without having to re-execute computationally intensive avatar motion generation (such as lip-syncing), thereby enabling variations that match the intentions of the webinar and the feeling of the season.

[0070] (Overview of A / B testing based on differences in avatars and content) Referring to Figure 11, an overview of the A / B test using different avatars in the webinar video generation system 10 will be explained. Figure 11 is a diagram illustrating the overview of the A / B test using different avatars in the webinar video generation system 10 according to this embodiment. As shown in Figure 11, the avatar storage unit 60 pre-stores multiple avatar images 40 (for example, a first avatar image 40a and a second avatar image 40b) with different attributes such as gender, age, clothing, and atmosphere. When conducting an A / B test, the avatar selection unit 61 selects multiple avatar images to be compared (for example, a first avatar image 40a representing a young male avatar and a second avatar image 40b representing a veteran female avatar) from the avatar storage unit 60 based on user specifications or instructions from the A / B test execution unit 74. The A / B test execution unit 74 applies the different avatar images (40a, 40b) selected by the avatar selection unit 61 to a single formatted text data 27 (script) and automatically generates a first new webinar video 41a and a second new webinar video 41b that speak the same content. Furthermore, the A / B testing unit 74 may generate multiple videos using the same avatar image 40, but in cooperation with the partial modification unit 73, by changing only a part of the script (for example, the opening greeting or closing remarks). The generated videos are distributed to different viewer groups (first viewer 85a, second viewer 85b). Subsequently, the A / B testing unit 74 collects and compares viewing performance data such as viewer retention rate, completion rate, and link click-through rate (CTR) for the first new webinar video 41a and the second new webinar video 41b. This makes it possible for users to optimize the marketing effect of presenter attributes, such as "which gender, age, and atmosphere of presenter is most effective for explaining this product," based on low-cost and quantitative data, which was difficult to verify with conventional live-action filming.

[0071] (Methods for generating webinar videos and the basic processing flow of a webinar video generation program) Referring to Figure 12, an example of the basic processing flow of the webinar video generation method and webinar video generation program according to this embodiment will be described. Figure 12 is a flowchart illustrating an example of the basic processing flow of the webinar video generation method and webinar video generation program according to this embodiment. The webinar video generation method is executed by the calculation unit 10e of the webinar video generation system 10 based on the webinar video generation program.

[0072] The webinar video generation method and webinar video generation program include an acquisition step S62, an audio extraction step S63, an audio recognition step S64, a personal information processing step S65, an audio synthesis step S68, an avatar video generation step S70, a timing determination step S71, and a video synthesis step S72, among others. The webinar video generation program provides the following functions to the calculation unit 10e of the webinar video generation system 10: acquisition function, audio extraction function, speech recognition function, personal information processing function, speech synthesis function, avatar video generation function, timing determination function, and video synthesis function. These functions are executed in the order shown in the flowchart in Figure 12, but the order can be changed as needed. Note that detailed explanations of each function are omitted as they overlap with the descriptions of the various functions of the webinar video generation system 10 mentioned above.

[0073] The acquisition function acquires the original webinar video 20 and the explanatory materials 30 associated with the webinar video 20 (S62: Acquisition step).

[0074] The audio extraction function extracts audio data 22 from the webinar video 20 (S63: Audio Extraction Step).

[0075] The speech recognition function converts the speech data 22 into text data 24 (S64: speech recognition step).

[0076] The personal information processing function detects portions of the text data 24 that correspond to personal information and replaces those portions with predetermined information (S65: Personal Information Processing Step).

[0077] The speech synthesis function generates synthesized speech 36 from formatted text data 27, which includes text data processed by the personal information processing function (S68: speech synthesis step).

[0078] The avatar video generation function generates an avatar video 39, which is an avatar image 40 whose mouth moves in sync with the content of the synthesized voice 36 (S70: avatar video generation step).

[0079] The timing determination function determines the timing for switching pages in explanatory document 30 based on the formatted text data 27 and the content of explanatory document 30 (S71: Timing determination step).

[0080] The video compositing function combines synthesized speech 36, avatar video 39, and video of explanatory materials 30 that can be switched according to the timing to generate a new webinar video 41 (S72: Video Compositing Step).

[0081] (Webinar video generation method and overall processing flow of the webinar video generation program) Referring to Figure 13, an example of the overall processing flow of the webinar video generation method and webinar video generation program according to this embodiment will be explained. Figure 13 is a flowchart illustrating an example of the overall processing flow of the webinar video generation method and webinar video generation program according to this embodiment. The webinar video generation method and webinar video generation program shown in Figure 13 are modified versions of the webinar video generation method and webinar video generation program shown in Figure 12, differing from the webinar video generation method and webinar video generation program shown in Figure 12 in that the following steps are added: avatar storage step S60, avatar selection step S61, filler removal step S66, translation step S67, section splitting step S69, partial modification step S73, A / B test execution step S74, Q&A step S75, co-sponsored video generation step S76, and information update step S77.

[0082] The webinar video generation program shown in Figure 13 will be described below, along with the webinar video generation method shown in Figure 13. The webinar video generation method and webinar video generation program shown in Figure 13 include an avatar storage step S60, an avatar selection step S61, an acquisition step S62, an audio extraction step S63, an audio recognition step S64, a personal information processing step S65, a filler removal step S66, a translation step S67, an audio synthesis step S68, a section division step S69, an avatar video generation step S70, a timing determination step S71, an audio synthesis step S72, a partial modification step S73, an A / B test execution step S74, a question and answer step S75, a co-sponsored video generation step S76, and an information update step S77, among others.

[0083] The webinar video generation method and webinar video generation program shown in Figure 13 implement the following functions on the calculation unit 10e: avatar storage function, avatar selection function, acquisition function, voice extraction function, voice recognition function, personal information processing function, filler removal function, translation function, voice synthesis function, section division function, avatar video generation function, timing determination function, video synthesis function, partial modification function, A / B test execution function, Q&A function, co-sponsored video generation function, and information update function. These functions are executed in the order shown in the flowchart of Figure 13, but the order can be changed as appropriate. Since each function overlaps with the explanation of the webinar video generation method and webinar video generation program shown in Figure 12 above, the overlapping explanations will be omitted. Also, since each function overlaps with the explanation of the various functional parts of the webinar video generation system 10 above, the detailed explanations will be omitted.

[0084] The avatar storage function stores avatar images having multiple different appearances (S60: Avatar storage step).

[0085] The avatar selection function selects an appropriate avatar image 40 from the avatar images 40 stored by the avatar storage function based on the attributes of the product being covered in the webinar or the user's selection (S61: Avatar Selection Step).

[0086] The filler removal function removes filler words from the text data 24 (S66: Filler Removal Step).

[0087] The translation function translates the formatted text data 27 into the specified language (S67: translation step).

[0088] The section splitting function divides the formatted text data 27 into multiple sections (S69: section splitting step).

[0089] The partial modification function modifies a portion of the formatted text data 27 (S73: partial modification step).

[0090] The A / B testing function generates multiple new webinar videos 41 using different avatar images 40 for the same formatted text data 27, and compares the viewing data for each new webinar video 41 (S74: A / B testing step).

[0091] The Q&A function generates text data of answers to questions received from webinar viewers based on formatted text data 27 and explanatory materials 30 (S75: Q&A step).

[0092] The co-hosted video generation function combines a webinar video featuring a human presenter with a new webinar video 41 to generate a co-hosted video in which a human presenter and an avatar appear together (S76: Co-hosted video generation step).

[0093] The information update function detects current events information contained in the formatted text data 27 and updates it if it differs from the latest information (S77: Information update step).

[0094] (Regarding the effects of the webinar video generation system 10 according to this embodiment) According to the webinar video generation system 10 of this embodiment described above, a new webinar video 41 (substitute presenter video) can be automatically generated from the original webinar video 20 by replacing the presenter's video with an avatar video 39 and replacing the audio with synthesized voice 36. This makes it possible to continue using the content of the webinar video (explanatory content and materials) as an asset of the organization, even if the original presenter leaves the company or permission to use their image can no longer be obtained.

[0095] According to the webinar video generation system 10 of this embodiment described above, by including a section division unit 69, the original long webinar video 20 can be divided into semantic units (sections) such as slides or pages, and avatar video generation processing can be executed in parallel for each section. This distributes the processing load of the generation AI, significantly reducing the generation time compared to generating the entire video at once, while maintaining high quality in the generated video and consistency in characters.

[0096] According to the webinar video generation system 10 of this embodiment described above, by including a partial modification unit 73, even if some information in the video (for example, numbers or proper nouns) changes due to legal revisions or annual updates, it is not necessary to reshoot the entire webinar video or regenerate the whole thing. By identifying only the sections that need correction, modifying the text data, and regenerating and combining only the avatar video for those sections, it becomes possible to maintain and operate webinar videos that always contain the latest information at extremely low cost and in a short amount of time.

[0097] According to the webinar video generation system 10 of this embodiment described above, by including a personal information processing unit 65 and a filler removal unit 66, it is possible to automatically detect and delete or replace personal names of speakers and unnecessary audio information such as hesitations contained in the original webinar video 20. This makes it possible to efficiently create high-quality proxy speaker videos (new webinar videos 41) based on a refined script that protects privacy and is easy to listen to.

[0098] According to the webinar video generation system 10 of this embodiment described above, by including a timing determination unit 71, the system can analyze the semantic correspondence between the content of the synthesized speech 36 and the content of the explanatory materials 30, and automatically determine the optimal timing for switching pages such as slides. This completely automates the synchronization work (timing adjustment) between pages such as slides and audio, which was previously done manually by editors, and makes it possible to significantly reduce the man-hours required for video production.

[0099] According to the webinar video generation system 10 of this embodiment described above, by including an avatar selection unit 61, it is possible to set an avatar with the optimal appearance (gender, age, atmosphere, etc.) as the presenter, according to the product or service of the webinar and the target audience. This makes it possible to perform optimal presentations according to the marketing strategy without being bound by the attributes of the original presenter.

[0100] According to the webinar video generation system 10 of this embodiment described above, by including an A / B testing execution unit 74, it is possible to easily generate multiple video variations using the same script (explanatory content) but with only the presenter's avatar changed. This makes it possible to conduct low-cost verification (A / B testing) to determine which character increases viewer engagement, and to optimize webinar marketing based on data.

[0101] According to the webinar video generation system 10 of this embodiment described above, by including a translation unit 67, it is possible to generate TTS audio and lip-sync video in a language that the original presenter does not speak, and to instantly create multilingual webinar videos. This makes it possible to easily disseminate information to a global market, overcoming language barriers.

[0102] According to the webinar video generation system 10 of this embodiment described above, the video compositing unit 72 performs chroma key compositing processing 79, which allows the background of the avatar video 39 to be easily replaced with any background image that matches the company's brand image or the season. This makes it possible to maintain the freshness of the video by changing the visual presentation without incurring the cost of recalculation, even after the avatar video 39 has been generated once.

[0103] According to the webinar video generation system 10 of this embodiment described above, by including a question and answer unit 75, it is possible to dynamically generate and insert videos in which an AI avatar that has learned the content of the webinar answers questions from viewers. This makes it possible to provide viewers with an interactive communication experience that is similar to a live broadcast, even though it is a recorded broadcast.

[0104] According to the webinar video generation system 10 of this embodiment described above, by including a co-sponsored video generation unit 76, a hybrid video structure can be realized in which human presenters and avatars interact. This allows for the division of important parts that should be presented by humans and routine parts that should be left to avatars, reducing the burden on presenters while enabling effective production that keeps viewers engaged.

[0105] According to the webinar video generation system 10 of this embodiment described above, by including an information update unit 77, it can periodically monitor whether the information in the original webinar video 20 has become outdated based on external data and prompt updates as needed. This prevents the risk of a webinar video being left with outdated information once it has been created, and ensures the credibility of the company.

[0106] According to the webinar video generation system 10 of this embodiment described above, existing high-performance AI models can be combined and used as modules in each process such as speech recognition, image generation, and speech synthesis. As a result, even if individual AI technologies evolve, the benefits can be immediately incorporated into the system, making it possible to always maintain the highest level of video generation capability.

[0107] As described above, the webinar video generation system 10 according to this embodiment first decomposes unstructured data, such as video, into structured data such as text and pages, and then reconstructs it after editing and processing. This achieves significantly higher flexibility in modifications and reduced operating costs compared to technologies that directly generate "video from video." As a result, companies can not only standardize and automate webinar operations, which tend to be highly dependent on individual employees, but also transform the created content from "disposable" to "assets" that continue to generate value over the long term.

[0108] Furthermore, the effects achieved through the coordination of each functional component described herein are applicable not only to webinar videos, but also to the creation and operation of all video content involving presentations, such as internal training videos, product manual videos, and recruitment pitch videos.

[0109] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented in various other modifications or applications without departing from the gist of the invention as described in the claims. For example, each functional unit of the webinar video generation system 10 may be implemented in a single housing, distributed across the cloud and the edge, or configured to run on an edge PC.

[0110] Furthermore, the present invention is not limited to the webinar video generation system 10, webinar video generation method, and webinar video generation program according to the above-described embodiment, and can be implemented by various other modifications or applications without departing from the gist of the present invention as described in the claims. Also, although the word "information" is used in the above-described embodiment, the word "information" can be replaced with "data," and the word "data" can be replaced with "information."

[0111] Furthermore, the above-mentioned functional units of the webinar video generation system 10 are merely examples of the functional units of the webinar video generation system 10 and do not limit the functions that the webinar video generation system 10 may have. For example, the webinar video generation system 10 does not need to have all of the above-mentioned functional units, and may have only some of them. Also, the webinar video generation system 10 may have other functions other than those described above. Furthermore, as described above, each of the functional units of the webinar video generation system 10 has been explained as being implemented by software. However, at least one of the functional units may be implemented by hardware.

[0112] Furthermore, any of the above functional units may be implemented by dividing one functional unit into multiple functional units. Alternatively, any two or more of the above functional units may be combined into a single functional unit. Also, the above description represents the functions of the webinar video generation system 10 using functional blocks, and does not indicate, for example, that each functional unit is composed of separate program files or the like.

[0113] Furthermore, the webinar video generation system 10 may be a device implemented in a single enclosure, or a system implemented from multiple devices connected via a network or the like. For example, the webinar video generation system 10 may implement some or all of its functions using a virtual device, such as a cloud service provided by a cloud computing system. In other words, the webinar video generation system 10 may implement at least one of the above-mentioned functional units in other devices. Also, the webinar video generation system 10 may be a general-purpose computer such as a desktop PC, or it may be a dedicated device with limited functionality. [Explanation of Symbols]

[0114] 10 Webinar Video Generation System 10a communication interface 10b Read Only Memory (ROM) 10c Random Access Memory (RAM) 10d storage section 10e Arithmetic Unit 10f Input / Output Interface 10g input device 10h output device 11. Information and Communication Networks 12 User terminals 13 Database Server 14. Generation AI Server 20 original webinar videos 20a Original webinar video 20b Original webinar video 20c Original webinar video 21. Voice Extraction 22 Audio data 23. Voice Recognition 24 Text data 24a Text data for each section 24b Example text data for each section 25 Processing of personal information 26. Filler removal 27. Formatted text data 27a Formatted text data 27b Pre-formatted text data 27c Pre-formatted text data 27d Example of formatted text data Formatted text data for each of the 28 sections 29 Section Divisions 30 Explanatory Materials 30a Explanatory Materials 30b Explanatory Materials 30c Explanatory Materials 31. Timing determination Extract 32 characters 33 characters Page 34 34a Page 1 34b Page 2 35. Speech Synthesis 36. Synthesized voice 36a First synthesized voice 36b Second synthesized voice 37. Create a lip-sync video 38 Avatar Video Generation 39 Avatar video 40 Avatar Images 40a First Avatar Image 40b Second Avatar Image 41 New Webinar Videos 43 Alerts Divided into 45-page sections 46. ​​Content review by AI 46a AI-driven content review 46b Content review by AI 46c AI-driven content review 50 Section 1 50a New webinar video 51 Section 2 51a New webinar video 52 Section 3 52a New webinar video 53 Completed video 53a Video clips from Section 1 53b Video clips from Section 2 53c Video clip from Section 3 54. Video clips from Section 2 (regenerated) 56 Original Webinar Video Database 57 Explanatory Materials Database 58 New Webinar Video Database 60 Avatar storage 61 Avatar Selection Section 62 Acquisition Department 63. Audio Extraction Unit 64. Voice Recognition Unit 65 Personal Information Processing Department 66 Filler removal section 67 Translation Department 68 Speech Synthesis Unit 69 Section division 70 Avatar Video Generation Unit 71 Timing determination unit 72. Video Compositing Section 73 Partially modified section 74 A / B Testing Execution Unit 75 Question and Answer Session 76 Co-sponsored Video Production Department 77 Information update department 79 Chroma key compositing 80 Green screen background 81 Replacement background image 82 Composite Result Video 85 viewers 85a First Viewer 85b Second Viewer

Claims

1. An acquisition unit that acquires the original webinar video and the explanatory materials associated with the webinar video, An audio extraction unit that extracts audio data from the aforementioned webinar video, A speech recognition unit that converts the aforementioned speech data into text data, A personal information processing unit detects portions of the aforementioned text data that constitute personal information and replaces those portions with predetermined information. A speech synthesis unit that generates synthesized speech from formatted text data including the text data processed by the personal information processing unit, An avatar video generation unit generates an avatar image as an avatar video, in which the mouth moves in sync with the content of the synthesized speech. A timing determination unit that determines the timing for switching pages of the explanatory materials based on the formatted text data and the content of the explanatory materials, A video synthesis unit that synthesizes the synthesized voice, the avatar video, and the video of the explanatory materials which are switched according to the timing, to generate a new webinar video, A webinar video generation system equipped with [features / equipment].

2. The webinar video generation system according to claim 1, characterized in that the personal information processing unit detects at least one of the speaker's name, department, or job title contained in the text data, and replaces the detected information with pre-configured alternative information.

3. The webinar video generation system according to claim 1, further comprising an information update unit that detects timely information contained in the formatted text data and updates the information if it differs from the latest information.

4. An avatar storage unit that stores avatar images having multiple different appearances, An avatar selection unit selects an appropriate avatar image from the avatar images stored in the avatar storage unit based on the attributes of the product being targeted at the webinar or the user's selection, Furthermore, The webinar video generation system according to claim 1, characterized in that the avatar video generation unit generates the avatar video using the avatar image selected by the avatar selection unit.

5. The webinar video generation system according to claim 4, characterized in that the avatar selection unit selects the avatar image based on the viewer's attribute information obtained when applying to participate in the webinar.

6. The webinar video generation system according to claim 4, further comprising an A / B testing unit that generates multiple new webinar videos using different avatar images for the same formatted text data, or using multiple formatted text data with different content for the same avatar image, and compares the viewing performance data of each new webinar video.

7. The system further comprises a translation unit that translates the formatted text data into a specified language. The webinar video generation system according to claim 1, characterized in that the speech synthesis unit generates synthesized speech in the specified language from the translated text data.

8. The avatar video generation unit generates the avatar video having a green screen background, The webinar video generation system according to claim 1, characterized in that the video compositing unit performs a chroma key compositing process that replaces the green screen background with a specified background image.

9. The system further includes a question and answer unit that generates text data of answers to questions received from webinar viewers based on the formatted text data and the explanatory materials, The speech synthesis unit generates synthesized speech from the response text data, The avatar video generation unit generates the avatar video as a video response to the question, based on the synthesized speech generated from the response text data. The webinar video generation system according to claim 1, characterized in that the video synthesis unit inserts and concatenates the response videos during the playback of the new webinar video.

10. The webinar video generation system according to claim 1, further comprising a co-hosted video generation unit that combines a webinar video featuring a human presenter with the new webinar video to generate a co-hosted video in which a human presenter and an avatar appear together.

11. The system further includes a filler removal unit that removes filler words from the aforementioned text data. The webinar video generation system according to claim 1, characterized in that the speech synthesis unit generates synthesized speech from formatted text data including the text data processed by the personal information processing unit and the text data processed by the filler removal unit.

12. On the computer, The acquisition step involves obtaining the original webinar video and the explanatory materials associated with that webinar video, An audio extraction step for extracting audio data from the aforementioned webinar video, A speech recognition step that converts the aforementioned audio data into text data, A personal information processing step that detects portions of the aforementioned text data that constitute personal information and replaces those portions with predetermined information, A speech synthesis step that generates synthesized speech from formatted text data including the text data processed by the personal information processing step, An avatar video generation step is to generate an avatar video in which the mouth movements are synchronized with the content of the synthesized speech, A timing determination step to determine the timing for switching pages of the explanatory material based on the formatted text data and the content of the explanatory material, A video synthesis step that generates a new webinar video by combining the synthesized voice, the avatar video, and the video of the explanatory materials which are switched according to the timing, A method for generating webinar videos to execute [this action].

13. On the computer, A function to retrieve the original webinar video and the explanatory materials associated with that webinar video, An audio extraction function that extracts audio data from the aforementioned webinar video, A speech recognition function that converts the aforementioned audio data into text data, A personal information processing function that detects portions of the aforementioned text data that constitute personal information and replaces those portions with predetermined information, A speech synthesis function that generates synthesized speech from formatted text data, including text data processed by the aforementioned personal information processing function, An avatar video generation function that generates an avatar image whose mouth movements are synchronized with the content of the synthesized speech, A timing determination function that determines the timing for switching pages of the aforementioned explanatory materials based on the formatted text data and the content of the explanatory materials, A video synthesis function that generates a new webinar video by combining the synthesized voice, the avatar video, and the video of the explanatory materials which are switched according to the timing, A webinar video generation program that makes this possible.

Images