Video content generation system based on multilingual script and text and method for providing video content thereby

Abstract

The present invention relates to a video content generation system based on a multilingual script and a method for providing video content thereby and, more specifically, to a narration video generation system based on a multilingual script and a method for providing a narration video thereby. For example, the present invention relates to a video content generation system based on a multilingual script and a method for providing video content thereby, in which a language is recognized from an input script, natural language processing is performed thereon, keywords and category information are extracted, video data and music data are selected from a cloud server unit by using the extracted keywords and category information, and subtitles and narrations in a corresponding language are added to produce a narration video.

Description

Video content creation system based on multilingual scripts and texts and method for providing video content using the system

[0001] The present invention relates to a video content generation system based on a multilingual script and a video content providing method therefor, and more particularly, to a narration video generation system based on a multilingual script and a narration video providing method therefor. For example, the present invention relates to a video content generation system based on a multilingual script and a video content providing method therefor, in which languages ​​are recognized from an input script, natural language processing is performed, keyword and category information are extracted, video data and music data are selected in a cloud server unit based on the extracted keyword and category information, and subtitles and narrations in the corresponding languages ​​are added thereto to produce a narration video.

[0002] Recent research shows that over 80% of online traffic is video, and online video content is preferred over text and image content. However, video content is time-consuming and expensive to produce and distribute. Consequently, demand for narrated videos, which combine pre-recorded and produced footage to provide additional commentary, is exploding. Narrated videos reportedly cost $1,000 and require 30 hours of labor per video. Specifically, this involves two hours of planning, 12 hours of source material research, 12 hours of editing, and four hours of subtitles and music insertion.

[0003] Web platforms are increasingly adopting AI technology to produce narrated videos, allowing even non-professionals without filming or editing skills to create videos using AI-powered video generation. However, these platforms only support English or feature videos primarily related to Western cultures, leaving virtually no platforms supporting Asian languages ​​or Asian-related content.

[0004] The present invention is intended to solve the above problems, and provides a narration video production system based on a multilingual script, in which language is recognized from an input script, natural language processing is performed, keyword and category information is extracted, video data and music data are selected in a cloud server unit based on the extracted keyword and category information, and subtitles and narration in the corresponding language are added to produce a narration video, and a narration video providing method thereby.

[0005] In addition, the present invention relates to a system for automatically generating multilingual video content and a method using the same, and more specifically, to provide a system for automatically generating video content using text of a script and a method for automatically generating video content using the same.

[0006] In order to achieve the above object, a video content creation system based on a multilingual script according to one embodiment of the present invention includes: a purchaser terminal unit for allowing a purchaser to input a script for purchasing a narration video and download a narration video produced for the script; a service server unit for searching for video and music matching the script input into the purchaser terminal unit and supplying a narration video produced by combining video, music, subtitles, and narration to the purchaser terminal unit; and a cloud server unit for uploading video data of a video data provider and music data of a music data provider, and supplying the video data and music data to the service server unit.

[0007] The service server unit includes a natural language processing unit that recognizes the language of a script input from the purchaser terminal unit, performs natural language processing, and then generates keywords and category information in the corresponding language; a video selection unit that searches for video data and music data stored in the cloud server unit using the keywords and category information generated by the natural language processing unit and selects at least one video data and music data; a narration generation unit that generates subtitles and narrations in the corresponding language using texts processed in the natural language processing unit; a video production unit that combines the video data and music data selected by the video selection unit and the subtitles and narration generated by the narration generation unit to produce a narration video; a video provision unit that provides the purchaser terminal unit with a narration video with an option for video data, music data, or narration added to the narration video produced by the video production unit; a payment management unit that manages payments to purchasers of the narration video, video data providers, and music data providers; and a watermark addition unit that adds a watermark to the video of the narration video produced by the video production unit.

[0008] The above service server unit further includes a video filtering unit that, if the video data selected by the video selection unit is not related to the nationality of the language of the input script, lists the video data corresponding to the cultural context of the script in order of relevance and selects the video data with the highest relevance first.

[0009] The above image filtering unit includes a person filtering unit, and when multiple people speaking different languages ​​appear in the image data, the person filtering unit selects the image data related to the language of the input script by specifying the nationality of the language spoken by the person with the highest weight by reflecting the weights for the person located at the center of the image, the size of the person in the image, and the number of people.

[0010] The video data and music data uploaded to the cloud server unit are characterized in that the video data and music data are stored in the cloud server unit by combining description keywords and description category information describing the video data and music data, respectively, as meta information.

[0011] In addition, in order to achieve the above-described object, a method for providing a video by a video content creation system based on a multilingual script according to an embodiment of the present invention includes a first step in which a script is input by a buyer into a buyer terminal; a second step in which a narration video with a watermark added is produced by a service server and provided to the buyer terminal; a third step in which the buyer confirms the narration video with the watermark added and makes a payment for a paid service; and a fourth step in which a narration video with the watermark removed by the service server and with options for video data, music data, and narration added is provided to the buyer terminal.

[0012] The second step includes a step 2-1 in which the language of the script input into the purchaser terminal is recognized and subjected to natural language processing to generate keyword and category information, a step 2-2 in which the keyword and category information is compared with description keywords and description category information of video data and music data stored in a cloud server to select video data and music data, a step 2-3 in which subtitles and narration in a language corresponding to the text subjected to natural language processing in the step 2-1 are generated, and a step 2-4 in which the video data and music data selected in the step 2-2 and the subtitles and narration generated in the step 2-3 are combined to produce a narration video, a watermark is added, and the video data and music data are provided to the purchaser terminal.

[0013] In the above step 2-2, the video data is selected by the video filtering unit so that if the selected video data is not related to the nationality of the language of the input script, the video data is listed in order of relevance and pushed down in priority, and further, if multiple people speaking different languages ​​or having different nationalities appear in the video data by the person filtering unit, the language or nationality used by the person with the highest weight is specified by reflecting the weights for the person located at the center of the video, the size of the person in the video, and the number of people, and the selected language or nationality is compared with the video related to the language or nationality of the input script.

[0014] The reflection of the above weights is done by calculating a first weight that reflects a higher score for each person as their distance from the center of the image is shorter when multiple people who speak different languages ​​or have different nationalities appear in the image, a second weight that reflects a higher score for each person as their size is larger, and a third weight that reflects a higher score for each person as the number of people increases, thereby classifying each person into a group corresponding to a cultural context, and for each of the classified groups, the image data corresponding to the cultural context of the group with the highest sum of the first to third weights is arranged in order of relevance, and the image with the highest weight is selected. Here, the cultural context can be divided into races such as Eastern and Western by language or nationality, and can be defined by clothing, behavior, etc. of a specific culture.

[0015] As described above, the present invention can process narration videos for scripts input in multiple languages ​​in a short period of time, and has the effect of increasing customer satisfaction by providing videos that reflect the language and cultural environments of each country.

[0016] Furthermore, the present invention has the effect of increasing the return on investment (ROI) by localizing video content in each country. Figure 1 is a chart showing the relationship between revenue by language and country for mobile games, with the size of the bubbles representing revenue. In other words, providing video content to multiple countries in their respective languages ​​can increase the ROI.

[0017] In addition, according to various embodiments of the present invention, when a user uploads a script text in a file format or inputs text in a natural language format, natural language processing, context analysis, topic derivation, subtitle generation, narration generation, background music generation, background atmosphere adjustment, etc. are performed through an AI model and an AI processor based on the AI ​​model, thereby reducing the user's involvement while generating the best video content that matches the user's intention.

[0018] Figure 1 is a diagram showing the relationship between revenue by number of languages ​​and number of countries in mobile games.

[0019] FIG. 2 is a schematic diagram of a narration video generation system based on a multilingual script according to one embodiment of the present invention.

[0020] FIG. 3 is a conceptual diagram of a narration video generation system based on a multilingual script according to one embodiment of the present invention.

[0021] Figure 4 is a configuration diagram of a service server unit according to one embodiment of the present invention.

[0022] Figure 5 shows a narration video generated by one embodiment of the present invention with a watermark (left) and without a watermark (right).

[0023] Figure 6 shows a case where the language of the subtitle and the image of the video do not match (left) and a case where the language of the subtitle and the image of the video match (right).

[0024] FIG. 7 is a conceptual diagram of a method for providing a narration video by a narration video generation system based on a multilingual script according to one embodiment of the present invention.

[0025] FIG. 8 is a flowchart of a method for providing a narration video by a narration video generation system based on a multilingual script according to one embodiment of the present invention.

[0026] FIG. 9 is a detailed flowchart of the second step of a method for providing a narration video by a narration video generation system based on a multilingual script according to one embodiment of the present invention.

[0027] FIG. 10 is a flowchart exemplarily illustrating the operation process of a method for automatically generating video content according to another embodiment of the present invention.

[0028] FIG. 11(a) is an exemplary drawing showing a method of editing by scene unit during the video cut editing and subtitle editing steps, FIG. 11(b) is an exemplary drawing showing a method of adjusting the length of a scene during the video cut editing and subtitle editing steps, FIG. 11(c) is an exemplary drawing showing a method of changing a background video during the video cut editing and subtitle editing steps, FIG. 11(d) is an exemplary drawing showing a method of changing background music during the video cut editing and subtitle editing steps, FIG. 11(e) is an exemplary drawing showing a method of changing narration settings during the video cut editing and subtitle editing steps, and FIG. 11(f) is an exemplary drawing showing a method of changing subtitle settings during the video cut editing and subtitle editing steps.

[0029] Hereinafter, a narration video generation system based on a multilingual script according to an embodiment of the present invention will be described with reference to drawings. FIG. 2 is a schematic diagram of a narration video generation system based on a multilingual script according to an embodiment of the present invention, and FIG. 3 is a conceptual diagram of a narration video generation system based on a multilingual script according to an embodiment of the present invention. Referring to FIGS. 2 and 3, a narration video generation system based on a multilingual script according to an embodiment of the present invention may be configured to include a purchaser terminal unit (100), a service server unit (200), and a cloud server unit (300). Here, the purchaser terminal unit (100), the service server unit (200), and the cloud server unit (300) are connected through a wired / wireless communication network, and an open network such as the Internet may be used as the wired / wireless communication network. The Internet can refer to a worldwide open computer network structure that provides various services existing at the TCP / IP protocol and its upper layers, namely HyperText Transfer Protocol (HTTP), Telnet, File Transfer Protocol (FTP), Domain Name System (DNS), Simple Mail Transfer Protocol (SMTP), Simple Network Management Protocol (SNMP), Network File Service (NFS), and Network Information Service (NIS).In addition, any one of the technical standards or communication methods for mobile communication for accessing the Internet (for example, GSM (Global System for Mobile communication), LTE (long-term evolution), LTEA (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications), 5G, etc.) may be used, but is not limited thereto. In addition, the multilingual text of the present invention includes any one of 133 languages, such as Korean, English, Chinese, Japanese, Vietnamese, Indonesian, Malaysian, Thai, and Arabic, but the multilingual text is not limited to the above.

[0030] The purchaser terminal (100) allows the purchaser to input a script for purchasing a narration video and download a narration video produced for the script. The purchaser needs a narration video for the script, inputs the script through the purchaser terminal (100), and requests the service server (200) to produce a narration video for the script. Here, the purchaser terminal (100) may be one of a smart phone, a desktop computer, a laptop computer, and a tablet PC. The purchaser must input the script in the language he or she uses or in the language of the country in which the video content is used so that the service server (200) can use the video corresponding to the language.

[0031] The service server unit (200) searches for video data and music data that match the script input into the purchaser terminal unit (100) and supplies video content produced by combining video data, music data, subtitles, and narration to the purchaser terminal unit (100). Fig. 4 is a configuration diagram of the service server unit (200) according to one embodiment of the present invention. Referring to Fig. 4, the service server unit (200) according to one embodiment of the present invention may be configured to include a natural language processing unit (210), a video selection unit (220), an image filtering unit (230), a narration generation unit (240), a video production unit (250), a video provision unit (260), a payment management unit (270), and a watermark addition unit (280). The service server unit (200) may be implemented as an edge computing terminal such as a smart phone, a desktop computer, a laptop computer, or a tablet PC. However, preferably, the service server unit (200) may be implemented in an integrated form such as a cloud and AI server, a GPU (Graphics Processing Unit)-based cloud server, a web server, a database server, or a proxy server. In addition, the service server unit (200) is operated by machine learning or deep learning.

[0032] The natural language processing unit (210) recognizes the language of the script input from the buyer terminal (100), performs natural language processing, and then generates keyword and category information in the corresponding language. Here, the natural language processing can be performed using deep learning technologies such as recurrent neural networks (RNN) and convolutional neural networks (CNN). The natural language processing unit (210) can receive the text of the script from the buyer terminal (100) or load a text file (e.g., ***.txt, ***.hwp, ***.doc, etc.). The natural language processing unit (210) recognizes the language input into the buyer terminal (100). For example, it recognizes whether the language is one of Korean, English, Chinese, Japanese, Vietnamese, Indonesian, Malaysian, Thai, and Arabic, and performs natural language interpretation based on the recognized language. The natural language processing unit (210) can analyze the text of an input script or the contents of a script contained in a text file to predict or generate keywords, topics, categories, etc. for the entire script or for paragraphs or sentences. Specifically, when text containing a specific topic is input, the text is decomposed into paragraphs, sentences, phrases, and words, and grammar such as spelling is checked. Sentences in the same context are separated, and important keywords are extracted from the separated sentences. Major parts of speech (nouns, verbs, etc.) are extracted for the sentences through a morphological analyzer. At this time, the morphological analyzer can use open source modules such as Okt, Komoran, and Kkma, or its own module. In addition, the natural language processing unit (210) can generate at least one script keyword for the topic of the script, the category of the script, and each sentence constituting the script for the input text. In addition, the natural language interpretation unit can generate synonyms or analogues that are contextually similar to the generated script keywords.That is, the natural language processing unit (210) generates, through relevance mapping, the script's category, keywords, and synonyms contextually similar to the keywords in the language in which the script was entered. For example, if a buyer entered a script about the efficacy of vitamins, the category of the script could be health food or medicine, keywords could be health food, vitamins, etc., and synonyms could be health supplements, immunity, active vitamins, etc.

[0033] For example, if a buyer inputs one script into the buyer terminal (100), it is assumed that the script includes one topic (e.g., drug), three paragraphs (e.g., drug introduction, drug administration method, drug efficacy), 12 sentences, and 400 words. The natural language processing unit (210) uses a morphological analyzer to morpheme-analyze the 400 words included in the script. The natural language processing unit (210) reviews the spelling and grammar of each word analyzed by morpheme, and can allow the user to decide whether to correct words, phrases, spacing, etc. that have spelling and grammar errors. The natural language processing unit (210) separates the 12 sentences included in the script, and generates a script keyword for each sentence. That is, the keyword for the first sentence is generated as the first keyword, and the keyword for the second sentence is generated as the second keyword, thereby generating a script keyword for the entire sentence. The natural language processing unit (210) generates synonyms using a pre-trained natural language processing model for multiple keywords. The natural language processing unit (210) generates multiple script keywords and multiple similar keywords. In addition, the natural language processing unit (210) classifies the script as a drug, generates synonyms for the category information (drug) for the script, and maps them based on relevance.

[0034] The video selection unit (220) searches for video data and music data stored in the cloud server unit (300) using keywords and category information generated in the language entered into the purchaser terminal unit (100) by the natural language processing unit (210) to select at least one video data and music data. The video selection unit (220) searches the cloud server unit (300) using categories, keywords, and synonyms generated by the natural language processing unit (210) to select a plurality of related video data and music data. Since the video data and music data uploaded to the cloud server unit (300) are each stored as a meta file with description keywords and description categories, the video selection unit (220) performs a search by comparing the meta file with the categories, keywords, and synonyms generated by the natural language processing unit (210). That is, since the video data uploaded to the cloud server unit (300) is analyzed by the cloud server unit (300) for the image frames of the corresponding video data and a video description text explaining the context of the image frames of the corresponding video data is generated, the video selection unit (220) can search the video data frame by frame. The video selection unit (220) can rank the searched video data and music data in order of proximity to categories, keywords, and synonyms and select the top 2 to 10 video data or music data. When multiple video data and music data are selected, the video production unit (250) can edit and combine multiple video data and music data to produce multiple narration videos, and the video provision unit (260) can optionally provide multiple video data and music data to the purchaser so that the purchaser can select the video data and music data he or she wants.The above video selection unit (220) selects music data having a specific genre and tempo based on the character information and background information of the video data. The character information may be based on the character's gender and age, and the background information may be based on the character's movements, etc. to select music data. For example, before a soccer match between young male members is in progress, a slow-tempo ballad music may be selected, and in the case of video data showing a soccer match, a fast-tempo waltz music may be selected, thereby having the advantage of being able to select music data based on the character and background information shown in the video data.

[0035] The narration generation unit (240) generates subtitles and narration in a corresponding language from text processed by the natural language processing unit (210). The narration generation unit (240) is equipped with a TTS API, which is software that converts text into voice audio. Text processed by the natural language processing unit (210) in sentence units is converted into subtitles by the narration generation unit (240), and a narration voice file corresponding to the subtitle is generated by the TTS API. The narration generation unit (240) generates multiple narration voice files so that a purchaser can select a narrator.

[0036] The video production unit (250) produces a narration video by combining the video data and music data selected by the video selection unit (220) and the subtitles and narrations generated by the narration generation unit (240). The video production unit (250) can produce a plurality of narration videos by combining a plurality of video data and music data selected by the video selection unit (220) and the subtitles and narrations generated by the narration generation unit (240). Therefore, the video production unit (250) edits the video by combining a plurality of video data selected frame by frame with music data, subtitles and narrations to produce a plurality of narration videos.

[0037] The video provider (260) provides the purchaser with a narration video with options for video data, music data, or narration for the video content produced by the video production unit (250) to the purchaser's terminal. The video provider (260) selects one of the multiple narration videos produced by the video production unit (250) as a primary narration video and provides the remaining narration videos as options to the purchaser's terminal (100). The video provider (260) provides the purchaser's terminal (100) with video data options, music data options, and narration options with different voices along with the primary narration video, and the purchaser can change the narration video by changing the narration video to a desired selection among the various options. In addition, the video provider (260) provides the purchaser's terminal (100) with options for the subtitle position, font, font size, font color, and font transparency for the subtitles.

[0038] The payment management unit (270) manages payments to buyers of narrated videos, video data providers, and music data providers. The payment management unit (270) provides a payment platform for buyers to make payments for paid services, and controls the watermarking feature to be disabled for buyers who have paid for the service. Furthermore, if a buyer who has paid for a paid service requests the production of a narrated video, the payment management unit (270) pays the video and music fees used in the narrated video to the video and music providers.

[0039] The watermark adding unit (280) adds a watermark to the video of the narration video produced by the video production unit (250). FIG. 5 illustrates a narration video produced by one embodiment of the present invention with a watermark (left) and without a watermark (right). In the present invention, if a purchaser who has not paid for a paid service wants to check a trial version of the narration video, a watermark can be added to the video of the narration video to protect the copyright of the video data. If the purchaser has paid for the paid service, the narration video with the watermark removed is provided to the purchaser terminal unit (100). The watermark adding unit (280) includes a recognition watermark generating unit that generates a recognition watermark having a size of 1 / 50 to 1 / 100 of the size of the watermark at a specific location in the video data, and the recognition watermark generating unit causes the color of the recognition watermark to change but not be deleted even after payment for the paid service. The size of the watermark can be defined as the length of the horizontal and vertical diagonals of the watermark. Accordingly, it is possible to confirm whether the video data used in a specific narration video was produced by a narration video generation system based on a multilingual script, and it has the advantage of being able to confirm whether the video was used after payment based on the color change of the recognition watermark before and after payment.

[0040] The service server unit (200) further includes a filtering unit (230) that lists video data corresponding to the cultural context of the script in order of relevance and selects video data with a high relevance first, if the video data selected by the video selection unit (220) is not related to the nationality of the language of the input script, and then selects video data corresponding to the cultural context of the script in order of relevance. FIG. 6 illustrates a case where the language of the subtitle and the image of the video do not match (left) and a case where the language of the subtitle and the image of the video match (right). The video selection unit (220) searches for and selects video data from the cloud server unit (300) based on keywords selected for the language of the input script, but there is a possibility that the selected video data is not from a country where the language is used, but from a culture other than that in which the language is used. In the present invention, if the video data searched by the video selection unit (220) is not from a country where the language is used, or is from a culture other than that in which the background is used, the video data is pushed down in priority by the video filtering unit (230). The video filtering unit (230) detects the content of the video data by deep learning image recognition technology. The image filtering unit (230) has sufficient learning to be able to distinguish between people, buildings, landscapes, and other images related to each country and East and West based on the cultural context of the language recognized by the present invention, and recognizes images from the image data selected by the video selection unit (220) to determine whether the selected images are image data unrelated to the country where the language is used, and lists the unrelated image data in order of relevance to push it down in priority. In addition, the image filtering unit (230) further includes a person filtering unit that selects the image data by reflecting the weights for the person located at the center of the image, the size of the person in the image, and the number of people when multiple people speaking different languages ​​appear in the image data, and confirming the language or nationality used by the person with the highest weight.Accordingly, even if multiple people of different races are mixed in the video, if the person is located in the center or is close-up or if there are many people, the weight can be set high so that the image with the high sum of the total weights can be statistically set as the image related to the nationality of the language of the person. In addition, the image filtering unit (230) may be equipped with a background filtering unit that reflects, in addition to the weight of the person filtering unit, based on buildings and scenery. For example, in cases where the size of the person in the video is too small or it is difficult to select an image related to the nationality of the person, the image has the advantage of being able to select an image related to the nationality of the language of the input script based on buildings and scenery. In addition, the image filtering unit (230) may be equipped with a ratio filtering unit that selects an image by selecting the ratio of the person and the background. The filtering ratio of the person and background is selected from 1:0.1 to 10 by the above ratio filtering unit, and when the ratio of the person is set high, filtering is centered on the person, when the ratio of the background is set high, filtering is centered on the background, and when the ratio of the person and background is similar, filtering is performed accordingly, so it has the advantage of filtering by taking the person and background into consideration.

[0041] The above service server unit (200) indexes videos with a high correlation to the input script by mapping the results of the three labels below into multiple languages ​​when converting text into a narration video. First, a video category classification method using deep learning is performed, in which a preprocessing function is performed on video data, and a CNN model is used as an input value to classify the category of video data related to the preprocessed image. Second, a method of tagging keywords on video data is performed, in which categories for video data are automatically classified, keywords are automatically generated based on the classified category information and the video data content, and the generated keywords are provided to the user so that the user can selectively use the keywords to perform keyword tagging on the corresponding video data. Third, a method of outputting multilingual sentences describing the video using a CLIP model is performed, in which a video for which output of multilingual sentences describing the video is required is input, image frames for the video are extracted, categories, keyword tags, and sentence tags for the image are generated and mapped, and a final sentence tag with high similarity and relevance to the image is output.

[0042] The above service server unit (200) further includes an option storage unit that stores information on changed options when the purchaser changes the options of the narration video provided by the video provision unit (260). The option storage unit stores information on options selected by the purchaser from among a plurality of video data, music, and narrations provided as options together with the narration video, as well as changes to subtitles changed by the purchaser. The stored information is reflected in the options selected by the purchaser when the video production unit (250) produces the purchaser's next narration video. By continuously storing the contents of options changed by the purchaser by providing an option storage unit, there is an advantage in that the purchaser's taste can be identified and a narration video tailored to the purchaser's taste can be produced. For example, if the purchaser selects soft music and a female voice narration, the option storage unit stores the option contents so that the narration video is produced with soft music of the same genre and a female voice narration when the next narration video is produced.

[0043] The service server unit (200) may further include a video history investigation unit that investigates the history of narration videos previously produced with the same category and keywords as the narration video produced by the video production unit (250), and if the same video data or music data was used, causes the video selection unit (220) to search for and select the video data and music data again. If many buyers request the production of a narration video, if the category and keywords are the same, the same video data or music data may be used, which may reduce the originality of the narration video and may lead to disputes between buyers. To prevent this, the present invention includes a video history investigation unit that can investigate the history to see if a narration video with the same category and keywords has been produced, and if there is a history of using the same video data or the same music data, a search is performed again through the video selection unit (220) to select the video data and music data again.

[0044] The cloud server unit (300) uploads video data from a video data provider and music data from a music data provider, and supplies the video data and music data to the service server unit (200). The cloud server unit (300) is a cloud-type that anyone can freely access, and video data providers can freely upload video data to the cloud server unit (300), and music data providers can freely upload music data to the cloud server unit (300). The cloud server unit (300) can also upload copyrighted sound sources and allow the use of the sound sources by paying a certain usage fee for the sound sources. The cloud server unit (300) may be implemented as an edge computing terminal such as a smart phone, a desktop computer, a laptop computer, or a tablet PC, but preferably, the cloud server unit (300) may be implemented in an integrated form such as a cloud and AI server, a GPU (Graphics Processing Unit)-based cloud server, a web server, a database server, or a proxy server.

[0045] Video data and music data uploaded to the cloud server unit (300) are stored in the cloud server unit (300) by combining description keywords and description category information that describe the video data and the music data, respectively, as meta information. Since the description keywords and description category information are stored as meta information for the video data and the music data, respectively, the video data and the music data can be searched by the video selection unit (220). The cloud server unit (300) causes the video data provider and the music data provider to input description keywords and description categories when the video data and the music data are uploaded. In addition, the cloud server unit (300) analyzes the image frames of the uploaded video data to generate video description text that describes the context of the image frames of the corresponding video data and stores the generated video description text as meta information, thereby enabling the video selection unit (220) to search the video data frame by frame.

[0046] The above-described purchaser terminal unit (100) includes a purchaser direct selection unit that allows the purchaser to directly search for and select video data and music data through the cloud server unit (300) and to produce a narration video using the video data and music data selected by the purchaser. The purchaser searches for the video data and music data he or she desires through the purchaser direct selection unit in the cloud server unit (300), selects the video data and music data, and transmits them to the service server unit (200). The service server unit (200), which receives the script, video data, and music data from the purchaser terminal unit (100), performs natural language processing on the script through the natural language processing unit (210) to generate subtitles and narration, and produces a narration video by combining the video data, music data, subtitles, and narration in the video production unit (250).

[0047]

[0048] Hereinafter, a method for providing a narration video by a narration video generation system based on a multilingual script according to an embodiment of the present invention will be described with reference to the drawings. Fig. 7 is a conceptual diagram of a method for providing a narration video by a narration video generation system based on a multilingual script according to an embodiment of the present invention, and Fig. 8 is a flowchart of a method for providing a narration video by a narration video generation system based on a multilingual script according to an embodiment of the present invention. Referring to Fig. 8, a method for providing a narration video by a narration video generation system based on a multilingual script according to an embodiment of the present invention can be composed of the following four steps.

[0049] Step 1 (S10) is where the buyer inputs a script into the buyer terminal (100). The buyer inputs the script, which requires the production of a narrated video, into the buyer terminal (100) in the language they use. The buyer can use a dedicated program, access a specific website, and upload or directly input the script file.

[0050] Step 2 (S20) is a step in which a watermarked narration video is produced by the service server (200) and provided to the purchaser terminal (100). Fig. 9 is a detailed flowchart of the second step of a method for providing a narration video using a narration video production system based on a multilingual script according to one embodiment of the present invention. Referring to Fig. 9, Step 2 (S20) may be comprised of the following four steps.

[0051] Step 2-1 (S21) is the step where the language of the script input into the purchaser terminal (100) is recognized, natural language processed, and keyword and category information is generated. The script input into the purchaser terminal (100) is processed by the natural language processing unit (210) in the service server unit (200) connected to a wired / wireless communication network. The natural language processing unit (210) recognizes the language of the script, and image data search is performed based on the recognized language. The script is processed into natural language to generate keywords, category information, and synonyms for the keywords.

[0052] Step 2-2 (S22) is a step in which the video data and music data are selected by comparing the keyword and category information with the description keywords and description category information of the video data and music data stored in the cloud server unit (300). The video selection unit (220) searches for the video data and music data stored in the cloud server unit (300) based on the keywords and category information generated by the natural language processing unit (210). Since the video data and music data stored in the cloud server unit (300) each have their description keywords and description category information stored as meta information, the video data and music data are selected in multiple pieces based on the closest keywords and category information by comparing them.

[0053] In the above step 2-2 (S22), the video data is selected by the video filtering unit so that if the selected video data is not related to the nationality of the language of the input script, the video data is pushed down in priority. In addition, if multiple people speaking different languages ​​appear in the video data by the person filtering unit, the nationality of the language used by the person with the highest weight is specified by reflecting the respective weights of the person located at the center of the video, the size of the person in the video, and the number of people, and the nationality of the language used by the person with the highest weight is listed in order of relevance compared to the video related to the nationality of the language of the input script and pushed down in priority. Therefore, if multiple people exist in the video data, the weight is set to 3 if the person is located at the center of the video, and to 1 if the person is located at the periphery of the video, and the weight is set to 1 or 3 depending on the size of the person in the video, and a weight of 1 is added according to the number of people, and it is determined whether the video data is related to the nationality of the language of the race with the highest weight and the nationality of the language of the input script.

[0054] Step 2-3 (S23) is the step where subtitles and narration in the corresponding language are generated from the text processed in Step 2-1 (S21). The text of the script, which is separated into sentences through natural language processing in the natural language processing unit (210), is subtitled in the corresponding language in which the script was written, and a narration voice file corresponding to the subtitle is generated by the TTS API.

[0055] Step 2-4 (S24) is a step in which the video data and music data selected in Step 2-2 (S22) and the subtitles and narrations generated in Step 2-3 (S23) are combined to produce a narration video, a watermark is added, and the video is provided to the purchaser terminal (100). The video production unit (250) produces multiple narration videos by combining video data, music data, subtitles, and narrations. Since the purchaser has not yet made payment for the paid service, the narration videos are provided as a trial version. The watermark addition unit (280) adds a watermark to the video for the narration video provided as a trial version to protect the copyright of the video data. In addition, since the trial narration video does not provide video data, music data, or narration options, the video provision unit (260) provides a single narration video with a watermark added to the purchaser terminal (100).

[0056] The second step (S20) may further include a step of relegating the image data to a lower priority by listing the image data in order of relevance if the image data searched by the video selection unit (220) is not from a country that uses the language in question or is from a background of a different culture, for the image data selected in the second-second step (S22). Since the present invention uses image data that reflects the culture of a country that uses the language in question for a multilingual script, if the image data searched and selected by the video selection unit (220) is not related to the language of the multilingual script or the country that uses the language in question, the image data is listed in order of relevance and listed to a lower priority. As illustrated in FIG. 5, in order to prevent a case where the language of the multilingual script and the image data do not match, the image filtering unit (230) relegates the image to a lower priority if the image is not related to the language of the script, even if sufficient learning has been performed on the culture and related images of the country in which the language of the script is used.

[0057] The second step (S20) may further include a step of investigating the history of narration videos produced for the same category and keywords as the narration video produced in the second-fourth step (S24) and, if the same video data or music data was used, searching the video data and music data again. If many buyers request the production of a narration video, if the category and keywords are the same, the same video data or music data may be used, which may reduce the originality of the narration video and may cause disputes between buyers. To prevent this, the present invention provides a video history investigation unit to investigate the history of whether a narration video with the same category and keywords has been produced, and if there is a history of using the same video data or the same music data, the same video data or the same music data is excluded and a search is performed again through the video selection unit (220) to reselect the video data and music data.

[0058] Step 3 (S30) is where the buyer reviews the watermarked narration video and proceeds with payment for the paid service. Upon reviewing the watermarked narration video, the buyer can decide whether to purchase the paid service. The buyer makes payment for the paid service through the payment management unit (270). Upon payment, a fee for the video data and music data used in the narration video purchased by the buyer is paid to the video data provider and music data provider through the payment management unit (270).

[0059] Step 4 (S40) is a step in which the watermark of the narration video is removed by the service server unit (200) and the narration video with added options for video data, music data, and narration is provided to the purchaser terminal unit (100). If the purchaser has paid for the paid service, the video providing unit (260) provides the narration video with the watermark removed and with added options for video data, music data, and narration to the purchaser terminal unit (100). The purchaser can change the options for the video data, music data, and narration of the narration video through the purchaser terminal unit (100), and can also change the position of the subtitles, font, font size, font color, font transparency, etc.

[0060] A method for providing a narration video by a narration video generation system based on a multilingual script according to one embodiment of the present invention further includes a step of, when the purchaser selects an option for the video data, music data, and narration provided in the fourth step (S40), storing the contents of the option selected by the purchaser and reflecting the selected option when producing the purchaser's next narration video. The option storage unit stores information on the option selected by the purchaser among a plurality of video data, music, and narration options provided as options together with the narration video and changes to subtitles made by the purchaser, and the stored contents are reflected in the video production unit (250) when producing the purchaser's next narration video. The present invention has an advantage in that the purchaser's taste can be identified by continuously storing the contents of options changed by the purchaser by providing the option storage unit, so that a narration video tailored to the purchaser's taste can be produced.

[0061] FIG. 10 is a flowchart exemplarily illustrating the operation process of a method for automatically generating video content according to another embodiment of the present invention. Referring to FIG. 10, the method for automatically generating video content in the natural language processing unit performs a step of receiving basic information for video production by directly inputting text or loading a text file (S610). The natural language processing unit generates a topic and multiple script keywords using the input text. The video caption generation unit generates video caption information using a video file. The video generation unit combines the language caption information and the video caption information (S630). The video generation unit searches for videos using the language caption information and the video caption information, selects videos with high similarity, and combines subtitles, narration, and music to generate primary video content (S650). The video editing unit provides the user with a user interface for editing video cuts and subtitles for the primary video content, and when the user performs the editing process, the video content is finally generated. Below, the process of editing the primary video content by the user will be described with reference to a separate drawing. The video generation unit of the present invention further includes a video duplication checking unit and a video duplication restriction unit, wherein the video duplication checking unit checks whether a video to be used for video content creation has been used in previous video content creation. In addition, the video duplication restriction unit excludes a video from video content creation if the video has been used in video content creation exceeding a certain number of times in the video duplication checking unit. Accordingly, a video with a high degree of similarity is searched to determine whether the video has been used previously, and if the video has been used exceeding a certain number of times, the use of the video is excluded from candidates, thereby enabling video content creation using a video with high originality.In addition, the video generation unit of the present invention further includes a video duplication control unit, and the video duplication control unit changes the criterion of a specific number of times in the video duplication limit unit according to the number of videos related to similarity when searching for video information in a pre-stored video database using a script keyword and a video description keyword. Accordingly, the similarity of videos recommended as candidates according to the script keyword and the video description keyword is determined, and when the number of video candidates with high similarity is small, the criterion of the specific number of times is lowered, and when the number of video candidates with high similarity is large, the criterion of the specific number of times is lowered, thereby enabling a flexible determination of duplication.

[0062] Fig. 11(a) is a diagram exemplarily illustrating a method of editing by scene unit among the video cut editing and subtitle editing steps illustrated in Fig. 10. Referring to Fig. 11(a), it can be exemplarily confirmed that a user interface capable of editing primary video content by image frame unit is provided on the display of the user terminal. Image frames can be edited by sentence unit based on sentence. For example, the image frame corresponding to sentence N is the previous CUT #N of CUT #N+1, and the user can intuitively edit up to the corresponding cut to match sentence N. In the case of sentence N+1, it can be edited to match from CUT #N+1.

[0063] FIG. 11(b) is a diagram exemplarily illustrating a method for adjusting the length of a scene during the video cut editing and subtitle editing steps illustrated in FIG. 6. Referring to FIG. 11(b), the synchronization between cuts and sentences can be adjusted by adjusting the length of sentence N. The video editing unit may include a sentence length control unit that adjusts the length of sentences for video cut editing and subtitle editing to synchronize the number of image frame cuts with the sentences. For example, by pulling the length of sentence N in the direction of the arrow, the length is reduced, and accordingly, the corresponding image frame cut also decreases. On the other hand, the image frame cut synchronized with sentence N+1 increases. That is, if the sentences are defined as a first sentence and a second sentence having different lengths, a first image frame cut corresponding to the first sentence, and a second image frame cut corresponding to the second sentence, the number of first image frame cuts and the number of second image frame cuts can be generated proportionally according to the ratio of the length of the first sentence and the length of the second sentence. Accordingly, as the length of a sentence increases, the related video information also increases, and the image frame cut is managed so that the user can appropriately select the length of the sentence and manage the image frame cut.

[0064] FIG. 11(c) is a drawing exemplarily illustrating a method for changing a background image during the video cut editing and subtitle editing steps illustrated in FIG. 6. Referring to FIG. 11(c), the background image can be changed on a cut-by-cut basis for a plurality of image frames. The video editing unit may include an image frame modification unit. When a user performs an operation on an image frame to be modified, the image frame modification unit displays a user selection item for adding or deleting an image frame, and the user can change the background image of the image frame by performing an operation for deleting or adding the image frame. In addition, the video editing unit may include an image frame recommendation unit. The image frame recommendation unit can search for and recommend a new image frame for a frame deleted or added by the user. Therefore, a new image frame can be recommended according to the modification of the image frame, making it convenient for modifying an image.

[0065] FIG. 11(d) is a diagram exemplarily illustrating a method for changing background music during the video cut editing and subtitle editing steps illustrated in FIG. 6. Referring to FIG. 11(d), background music can be manipulated to change from #N to #M for each background image frame. If background music is changed from #N to #M in the Nth image frame, background music #M can be applied to the Nth image frame and thereafter. The video editing unit may include a background music changing unit. The background music changing unit may select background music according to the conditions of the background video, and background music having a specific tempo is selected according to the movement of a person or object in the background video. For example, in a case where there is little change in the background video, such as in nature, background music with a slow tempo may be recommended, and in a case where there is a lot of change, such as in a sports game, background music with a fast tempo may be recommended.

[0066] Figure 11(e) is a diagram exemplifying a method for changing narration settings during the video cut editing and subtitle editing steps illustrated in Figure 6. Referring to Figure 11(e), narration settings can be changed for each background image frame. For example, the voice of the narrator in the Nth image frame can be changed to male or female. The narration speed can be increased or decreased. Additionally, the narration tone can be raised or lowered.

[0067] Figure 11(f) is a diagram exemplifying a method for changing subtitle settings during the video cut editing and subtitle editing steps illustrated in Figure 6. Referring to Figure 11(f), settings can be changed for subtitles displayed in video content. For example, when selecting a subtitle portion in video content, settings such as the subtitle font, size, subtitle color, subtitle type, transparency, and outline can be changed.

Claims

1. A purchaser terminal where a purchaser inputs a script to purchase a narration video and downloads a narration video produced for the script; A service server unit that searches for video data and music data that match the script entered in the purchaser terminal unit and supplies a narration video produced by combining video data, music data, subtitles, and narration to the purchaser terminal unit; and A video content creation system based on a multilingual script, comprising a cloud server unit that uploads video data from a video data provider and music data from a music data provider, and supplies video data and music data to the service server unit.

2. In claim 1, The above service server unit, A natural language processing unit that recognizes the language of the script entered in the above buyer terminal, performs natural language processing, and then generates keyword and category information in the corresponding language. A video selection unit that searches for video data and music data stored in the cloud server unit using keyword and category information generated by the natural language processing unit and selects at least one video data and music data; A narration generation unit that generates subtitles and narration in the corresponding language using text processed in the natural language processing unit above; A video production unit that produces a narration video by combining the video data and music data selected by the video selection unit and the subtitles and narration generated by the narration generation unit. A video provision unit that provides a narration video with video data, music data, or narration options added to the narration video produced by the above video production unit to the purchaser's terminal. Payment management department that manages payments to buyers of narrated videos, video data providers and music data providers; A video content creation system based on a multilingual script, including a watermark addition unit that adds a watermark to the video of a narration video produced by the above video production unit.

3. In claim 2, A video content creation system based on a multilingual script, characterized in that the video data and music data uploaded to the cloud server unit are stored in the cloud server unit by combining description keywords and description category information describing the video data and music data respectively as meta information.

4. In claim 3, A system for creating video content based on a multilingual script, wherein the service server unit further includes a video filtering unit that, if the video data selected by the video selection unit is not related to the nationality of the language of the input script, lists video data corresponding to the cultural context of the script in order of relevance and selects video data with a high relevance first.

5. A method for providing a narrated video by a video content creation system based on a multilingual script of any one of claims 1 to 4, Step 1: The script is entered into the buyer terminal by the buyer; A second step in which a narration video with a watermark added is produced by the service server and provided to the purchaser terminal; A third step in which the purchaser checks the narration video with the watermark added and proceeds with payment for the paid service; and A fourth step in which a narration video with a watermark removed by the service server and options for image data, music data, and narration are added is provided to the purchaser terminal; The second step above is, Step 2-1, where the language of the script entered into the above buyer terminal is recognized and processed as natural language, and keyword and category information is generated. Step 2-2, where the video data and music data are selected by comparing the above keyword and category information with the description keyword and description category information of the video data and music data stored in the cloud server. Step 2-3, in which subtitles and narration in the corresponding language are generated from the text processed in natural language in the above step 2-1, and Step 2-4 includes a step in which the video data and music data selected in step 2-2 are combined with the subtitles and narration generated in step 2-3 to produce a narration video, add a watermark, and provide the video to the purchaser terminal. In the above step 2-2, the video data is selected by the video filtering unit so that if the selected video data is not related to the nationality of the language of the input script, the video data is sorted in order of relevance and pushed down in priority, and further, if multiple people speaking different languages ​​appear in the video data by the person filtering unit, the nationality of the language used by the person with the highest weight is specified by reflecting the respective weights for the person located at the center of the video, the size of the person in the video, and the number of people, and the narration video providing method by the video content creation system based on a multilingual script is characterized in that the nationality of the language used is compared with the video related to the nationality of the language of the input script.

Images