Interactive natural language processing based video generation method

Abstract

The present invention generally relates to a technique for automatically generating a video by performing natural language processing on a scenario provided by a user. In particular, the present invention relates to an interactive natural language processing-based video generation technique that divides a user's scenario into multiple division sections based on natural language processing, with at least one template image stored for multiple unit stories, matches and sets a template image for each division section according to the user's selection, and then synthesizes a user's character generated from a user's photo with each template image, and generates a video by linking the template images.

Description

[Technical field]

[0001] The present invention generally relates to a technique for automatically generating a video by performing natural language processing on a scenario provided by a user. In particular, the present invention relates to an interactive natural language processing-based video generation technique that divides a user's scenario into multiple division sections based on natural language processing, with at least one template image stored for multiple unit stories, matches and sets a template image for each division section according to the user's selection, and then synthesizes a user's character generated from a user's photo with each template image, and generates a video by linking the template images. [Background technology]

[0002] With the spread of smartphones and faster networks, media-based information has become more widely used in recent years. In particular, as the number of users of social networking services such as Instagram (registered trademark), Facebook (registered trademark), Kakao Page (registered trademark), and video services such as YouTube (registered trademark) increases, videos are being actively used for self-expression and product promotion.

[0003] However, it is not easy for ordinary people to create videos with the content they want. Although it is possible to use a smartphone camera to record personal daily life and events, it is difficult to directly create videos with a certain level of content.

[0004] To produce such a video, it is necessary to find a location, cast the performers, shoot the footage from various angles, and then edit it and add special effects. This work requires skilled techniques and production experience, as well as various hardware and software, making it impossible for the average person to do. Those who absolutely need such a video to promote their company, publicize an event, or advertise a product will have to spend a considerable amount of money to hire a professional company.

[0005] However, the trend of the times is clear: the distribution of information based on media on the Internet. Therefore, technology is needed that allows ordinary people to more easily create videos with a certain level of story.

[0006] Patent Document 1: Korean Patent No. 10-1038903 (May 30, 2011) "Video content generation system based on public data provided by multiple community member systems" Patent Document 2: Korean Patent Registered No. 10-2262702 (June 3, 2021) "Video Generation System Using Shot Change Patterns" Patent Document 3: Korean Patent No. 10-2069897 (January 17, 2020) "User video generation method and device therefor" Patent Document 4: Korean Patent Publication No. 10-2021-0040882 (April 14, 2021) "Method and Apparatus for Generating Video" [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent No. 10-1038903 [Patent Document 2] Korean Patent No. 10-2262702 [Patent Document 3] Korean Patent No. 10-2069897 [Patent Document 4] Korean Patent Publication No. 10-2021-0040882 Summary of the Invention [Problem to be solved by the invention]

[0008] A general object of the present invention is to provide a technique for automatically generating a video by performing natural language processing on a scenario provided by a user. In particular, an object of the present invention is to provide an interactive natural language processing-based moving image generation technology that divides a user's scenario into a plurality of division sections based on natural language processing, with at least one template image stored for a plurality of unit stories, matches and sets a template image for each division section in accordance with the user's selection, and then synthesizes a user's character generated from the user's photograph with each template image, and connects these template images to generate a moving image. [Means for solving the problem]

[0009] In order to achieve the above object, the interactive natural language processing-based moving image generation device according to the present invention includes a template database 101 storing at least one template image for each of a plurality of unit stories, a natural language processing unit 120 for performing natural language processing, a scenario input unit 130 for receiving a user's scenario, a scenario division unit 140 for setting a plurality of divided sections corresponding to the unit stories by analyzing the user's scenario through the natural language processing of the natural language processing unit 120, a divided section matching unit 150 for matching and setting at least one template image stored in the template database 101 for each of the plurality of divided sections according to the unit stories, an interactive selection unit 160 for presenting a user with selection options for the divided sections to which the plurality of template images have been matched and performing natural language processing on the user's answers to those options, thereby matching and setting a template image corresponding to the user's selection for the corresponding divided section, and a moving image combination generation unit 180 for generating a user's moving image for the user's scenario by linking the plurality of template images that have been matched for each of the plurality of divided sections.

[0010] The video generating device according to the present invention may further include a video personalization unit 170 that cooperates with the interactive selection unit 160 to change the template image matched to the divided section according to a user's request and perform personalization processing.

[0011] The video generating device according to the present invention may further include a character generating and combining unit 110 that generates a character of the user by using a user's photo provided by the user and combines the user's character with a template image.

[0012] Also, the template database 101 may be configured to store a plurality of compound elements in which a combination of a plurality of branch information and a template image is matched with respect to a unit story. In this case, when a compound element corresponds to a division section, the interactive selection unit 160 may be configured to present a plurality of branch information included in the compound element to a user via the user terminal 200, and to match and set a template image linked to the branch information selected by the user to the corresponding division section.

[0013] The video generation device according to the present invention may further include a video database 102 for storing and managing user videos generated by the video combination generation unit 180, and a video external linkage unit 190 for linking and providing multiple user videos stored in the video database 102 to an external service server 300 or a user terminal 200.

[0014] In addition, to achieve the above object, the interactive natural language processing-based moving image generation method according to the present invention may include a first step in which the template database 101 stores at least one template image for each of a plurality of unit stories; a second step in which the scenario input unit 130 accepts a user's scenario; a third step in which the natural language processing unit 120 performs natural language processing on the user's scenario; a fourth step in which the scenario division unit 140 sets a plurality of divided sections corresponding to the unit stories by analyzing the user's scenario using natural language processing; a fifth step in which the divided section matching unit 150 matches and sets at least one template image stored in the template database 101 for each of the plurality of divided sections according to the unit stories; a sixth step in which the interactive selection unit 160 presents a selection option to the user for the divided sections to which the plurality of template images have been matched, and performs natural language processing on the user's response to the option, thereby matching and setting a template image corresponding to the user's selection to the corresponding divided section; and a seventh step in which the moving image combination generation unit 180 generates a user's moving image for the user's scenario by concatenating the plurality of template images that have been matched and set to the plurality of divided sections, respectively.

[0015] The video generation method according to the present invention may further include a step in which the video personalization unit 170 cooperates with the interactive selection unit 160 to change the template image matched to the divided section according to a user request and perform personalization processing.

[0016] The video generating method according to the present invention may further include a step of the character generating and combining unit 110 generating a user character using a user's photo provided by the user, and a step of the character generating and combining unit 110 synthesizing the user's character with a template image set to match a plurality of divided sections.

[0017] Also, the first step may include a step in which the template database 101 stores a plurality of compound elements in which a combination of a plurality of branch information and a template image is matched with a unit story. In this case, the sixth step may include a step in which, when a compound element corresponds to a division section, the interactive selection unit 160 presents a plurality of branch information included in the compound element to a user via the user terminal 200, the interactive selection unit 160 identifies branch information selected by the user from the presented plurality of branch information, and the interactive selection unit 160 matches and sets the template image linked to the selected branch information to the corresponding division section.

[0018] Meanwhile, a computer program according to the present invention is stored in a storage medium to cause a computer to execute the above-mentioned interactive natural language processing-based moving image generation method. Effect of the Invention

[0019] The present invention has an advantage that ordinary people can easily create videos with a certain number of stories. In addition, the present invention has the advantage of helping ordinary people to create videos that include stories that express themselves and upload them to Instagram (registered trademark), Facebook (registered trademark), Kakao Page (registered trademark), YouTube (registered trademark), etc.

[0020] In addition, the present invention has an advantage in that it can promote flexibility and diversity in video production by performing natural language processing on a scenario provided by a user in an interactive manner and reflecting the result in video generation. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a concept of automatically generating a moving image according to the present invention. [Diagram 2]1 is a block diagram showing the overall configuration of an interactive natural language processing-based moving image generation device according to the present invention. [Diagram 3] 1 is a flow chart showing the overall process of an interactive natural language processing-based video generation method according to the present invention; [Figure 4] FIG. 2 is a diagram showing the concept of storing a template image for each unit story in a template database in the present invention. [Diagram 5] FIG. 2 is a diagram showing a concept of dividing and processing a scenario in the present invention. [Figure 6] FIG. 2 is a diagram showing the concept of interactive processing in the present invention. [Figure 7] FIG. 2 is a diagram showing the concept of the configuration of a moving image generated by dividing a scenario in the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] In the following, the invention will be explained in detail with reference to the drawings. FIG. 1 is a diagram showing the concept of automatic generation of moving images according to the present invention. The video creation device 100 of the present invention is connected to multiple user terminals 200 via the Internet. As a result, a scenario and a user's photograph are provided by the user, and a number of selection inputs are accepted, whereby a video is automatically created based on the scenario. The video creation device 100 creates a video using materials stored in a template database 101, and stores the created video in a video database 102.

[0023] The video generation device 100 generally provides the generated video to the user who provided the scenario for download. Depending on the implementation example, the video generation device 100 may provide the generated video to an external service server 300 in response to a user request. In that case, the video may be utilized in a social network service (SNS), a video service, a metaverse service, or the like.

[0024] FIG. 2 is a block diagram showing the overall configuration of an interactive natural language processing-based moving image generating device 100 according to the present invention. Referring to FIG. 2, the interactive natural language processing based video generation device 100 according to the present invention includes a template database 101, a natural language processing unit 120, a scenario input unit 130, a scenario division unit 140, a division section matching unit 150, an interactive selection unit 160, a video personalization unit 170, a video combination generation unit 180, a video database 102, and a video external linkage unit 190.

[0025] First, the template database 101 is a component that stores at least one template image for each of a plurality of unit stories. A "unit story" is a wording that is a basic unit for generating a video screen, and may be set in various ways depending on the implementation of the invention. For example, "clouds are floating," "walking down the road," "driving a car," etc. may be unit stories.

[0026] FIG. 4 is a diagram showing the concept of storing a template image for each unit story in template database 101 in the present invention. One or more template images are associated with each unit story in the template database 101. Referring to Fig. 4, one template image 010 is associated with the unit story 01, and three template images 021, 022, and 023 are associated with the unit story 02.

[0027] In this specification, an item in which one template image is matched one-to-one with a unit story is called a simple element. In Fig. 4, the items related to unit stories 01 and 04 correspond to simple elements. An item in which multiple combinations of branch information and template images are matched with a unit story is called a composite element. In Fig. 4, the items related to unit stories 02, 03, and 99 correspond to composite elements.

[0028] In the composite element, branch information is matched for each template image. This branch information corresponds to variants for the corresponding unit story. For example, if the unit story 02 is "clouds floating", the branch information 021 may be "black clouds", the branch information 022 may be "cirrus clouds", and the branch information 023 may be "cumulus clouds". In this case, the template image 021 is an image of black clouds floating, the template image 022 is an image of cirrus clouds floating, and the template image 023 is an image of cumulus clouds floating. The template database 101 preferably stores a plurality of such combinations of unit stories and template images. In particular, in order to generate various types of moving images, it is preferable that the template database 101 is provided with composite elements.

[0029] These template images may be real-life images that have been actually captured, or may be three-dimensional computer graphic images that have been rendered using a Unity Engine, Unreal Engine, or the like.

[0030] The character generation and combination unit 110 is a component that generates a user character using a user's photo provided by the user and synthesizes the user character with a template image. For example, when a user takes a picture of his or her face with a smartphone and uploads it, a character for the user is generated. By synthesizing the user's character with a video, a story video that can effectively express the user can be generated.

[0031] The user's character is generated using computer software, such as the Unity Engine or Unreal Engine, based on actual photographed images. On the other hand, if a means for setting a different main character of the video is provided, the character generation and combination unit 110 may not be required. However, if the character of the user's video can be set using the user's own photo (image) via the character generation and combination unit 110, the user's satisfaction can be greatly improved.

[0032] The natural language processing unit 120 is a component for performing natural language processing (NLP). Natural language processing technology is a well-known technology, and the present invention does not relate to the natural language processing technology itself, so a detailed description thereof will be omitted.

[0033] The scenario input unit 130 is a component that accepts a user's scenario. A user's scenario is a text description of the contents of a video that the user desires, and may be uploaded from the user terminal 200 or transmitted via another server device.

[0034] The scenario division unit 140 is a component that divides a user's scenario according to its contents by natural language processing of the natural language processing unit 120 for the user's scenario. That is, when an appropriate user's scenario is input for a 10-minute video, the scenario division unit 140 divides the user's scenario into 50 fragments. As an embodiment for automatic processing by computer means, the scenario division unit 140 analyzes the user's scenario and sets a plurality of division sections corresponding to unit stories.

[0035] In this case, the scenario created by a person is not in a predetermined form, but in various forms that are difficult to predict, so it is difficult to expect that it will exactly match the unit story prepared in advance. Rather, a formal comparison based on the text is not suitable for the present invention, and the user's scenario and the unit story must be compared based on the content. Therefore, natural language processing is required in the present invention.

[0036] Fig. 5 is a diagram showing the concept of dividing a scenario in the present invention. It shows how the entire user's scenario is subjected to natural language analysis, and based on the keywords contained therein, a scenario corresponding to a unit story stored in advance is searched for and divided into n pieces.

[0037] The divided section matching unit 150 analyzes the contents of the divided sections by natural language processing, and searches for a corresponding unit story in the template database 101. As a result, the divided section matching unit 150 is a component that sets a matching template image stored in the template database 101 for each of the divided sections based on the unit story.

[0038] 5, divided section 01 has content corresponding to unit story 04, and template image 040 is set for matching. Also, divided section 02 has content corresponding to unit story 03, and template images 031 and 032 are set for matching. In this way, at least one template image is set for matching with respect to each divided section based on the unit story.

[0039] The interactive selection unit 160 is a component that presents the user with selection options for a division section to which multiple template images are matched, and performs natural language processing on the user's response to the options, thereby matching the template image corresponding to the user's selection to the corresponding division section.

[0040] 6 is a diagram showing the concept of interactive processing in the present invention. For example, when considering the divided section 03, three template images 021, 022, and 023 are set to match the divided section 03. That is, according to the above content regarding the template database 101, this corresponds to the case where a composite element corresponds to the divided section.

[0041] In this case, a plurality of branch information items included in the composite element are presented to the user via the user terminal 200, and the user's selection is accepted. For example, the branch information items 021, "black clouds", 022, "cirrus clouds", and 023, "cumulus clouds", are displayed on the user terminal 200, and the user's selection for these branch information items is input. In this case, when the user selects any one of the branch information items, for example, the branch information item 023, "cumulus clouds", the interactive selection unit 160 sets the template image 023 linked to the selected branch information item, i.e., "an image of floating cumulus clouds", to be matched to the divided section 03.

[0042] The video personalization unit 170 cooperates with the interactive selection unit 160 to change the template image matched to the divided section according to a user's request and perform personalization processing. The template images pre-stored in the template database 101 may not match the user's preferences. They may not fully reflect the detailed wordings written in the user's scenario, and further detailed expressions may need to be added during the user's interaction process. By resolving such issues, it is possible to increase the satisfaction level of the video quality.

[0043] Thereby, the video personalization unit 170 performs natural language processing on the user's scenario or the user's answer, and changes the template image to reflect the result of the natural language processing. As an example, consider a case where the unit story 03 of the divided section 03 is "purple-tinted thin clouds." The three template images 021, 022, and 023 matched to the divided section 03 are images corresponding to "black clouds," "cirrus clouds," and "cumulus clouds." In this case, the video personalization unit 170 may change the template images of those template images 021, 022, and 023 so that they correspond to "purple-tinted" and "thin."

[0044] As another example, consider a case where a template image 023, i.e., "an image with floating cumulus clouds", is matched and set by the interactive selection unit 160 for the divided section 03. There are cases where the user wishes to modify this matched template image 023. In this case, in response to a question from the video generating device 100 about whether or not the image can be modified, the user can reply, for example, in text, with the content of the modification, for example, "purple-tinted clouds" and "thin clouds". The video personalization unit 170 may change this template image 023 so that it corresponds to "purple-tinted" and "thin".

[0045] For this purpose, the video personalization unit 170 may be configured to perform natural language processing on the user's scenario or the user's answer, compare the natural language processing result with branch information of the corresponding template image, extract items that are not included in the branch information from the natural language processing result, and then change the template image according to the extracted items. In one embodiment, this process may be performed by software using Unity Engine, Unreal Engine, etc. In this case, a plurality of material images (e.g., purple clouds) may be stored in the template database unit 101 and then used for personalization processing.

[0046] On the other hand, personalization processing may be performed on one or more template images matched to each division section by the division section matching unit 150, and personalization processing may be performed on the final template image set to be matched to each division section by the interactive selection unit 160.

[0047] The video combining and generating unit 180 is a component that generates a user's video for a user's scenario by combining multiple template images that are set to match multiple divided sections. Fig. 7 is a diagram showing the concept of the configuration of a video generated by dividing a scenario in the present invention. A single video is generated by combining multiple template images.

[0048] The moving image database 102 is a component that temporarily or long-term stores and manages the user's moving images generated by the moving image combination generation unit 180 . The video external linking unit 190 is a component that links and provides videos of multiple users stored in the video database 102 to an external service server 300 or a user terminal 200. At this time, downloads may be provided, or only access (URL) may be provided. Such video external linking makes it possible to utilize user videos generated by the present invention in services such as SNS, video services, music video services, corporate advertising, personal advertising, and metaverse. Depending on the implementation example, advertising content stored in the video generating device 100 itself may be linked to user videos and provided.

[0049] FIG. 3 is a flow chart showing the overall process of the interactive natural language processing-based video generation method according to the present invention. Since the technical configuration of the present invention has been described in detail in relation to the animation generating device, the processing steps will be briefly discussed below.

[0050] Step S100: First, the template database 101, as shown in FIG. 4, prepares and stores at least one template image for each of a plurality of unit stories. Step S110: The character generation and combination unit 110 generates a character for the user using a Unity engine, Unreal engine, or the like, by using a photo (image) of the user.

[0051] Step S120: The scenario input unit 130 accepts a scenario from the user. Steps S130 and S140: The natural language processing unit 120 performs natural language processing on the user's scenario.

[0052] The scenario division unit 140 analyzes the user's scenario through the natural language processing of the natural language processing unit 120 for the user's scenario, and sets a plurality of division sections for the user's scenario corresponding to the unit stories, as shown in FIG.

[0053] Step S150: As shown in FIG. 5, the divided section matching unit 150 sets at least one template image stored in the template database 101 for each of the divided sections according to the unit story for matching.

[0054] Step S160: For a divided section to which multiple template images have been matched, as shown in FIG. 6, the interactive selection unit 160 presents selection options to the user, and matches the template image corresponding to the user's selection to the corresponding divided section through natural language processing of the user's response to the options.

[0055] To consider this process in detail, when a composite element corresponds to a division section, multiple branch information included in the composite element is presented to the user via the user terminal 200, the branch information selected by the user from the multiple branch information presented is identified, and a template image linked to the selected branch information is matched and set to the corresponding division section.

[0056] Step S170: The video personalization unit 170 performs personalization processing by changing the template image matched to the divided section according to a user request while cooperating with the interactive selection unit 160. That is, the video personalization unit 170 performs natural language processing on the user's scenario or the user's answer, and changes the template image to reflect the natural language processing result.

[0057] At that time, the personalization process may be performed on one or more template images matched to each divided section by the divided section matching unit 150, or on a final template image set to be matched to each divided section by the interactive selection unit 160.

[0058] Step S180: The character generating and combining unit 110 combines the user's character with the template image that is set to match the multiple divided sections. Step S190: The video combining unit 180 combines a plurality of template images that are set to match a plurality of divided sections, respectively, as shown in FIG. 7, to generate a user's video for the user's scenario.

[0059] Steps S200 and S210: The video database 102 temporarily or long-term stores and manages the generated user's video. The video external linking unit 190 links and provides the user's video in the video database 102 to an external service server 300 or a user terminal 200, thereby making it possible to utilize the user's video generated by the present invention for services such as SNS, video services, music video services, corporate advertising, personal advertising, and the metaverse.

[0060] The present invention may be realized in the form of computer-readable code stored in a computer-readable non-volatile recording medium. As such a non-volatile recording medium, there are various types of storage devices, such as a hard disk, SSD, CD-ROM, NAS, magnetic tape, web disk, cloud disk, etc., and a form in which the code is distributed and stored in a plurality of storage devices connected via a network and executed may also be realized. The present invention may also be realized in the form of a computer program stored in a medium to execute a specific procedure in combination with hardware.

Claims

1. A first step in which a template database (101) stores at least one template image for each of a plurality of unit stories; a second step in which a scenario input unit (130) accepts a user's scenario; a third step of a natural language processing unit (120) performing natural language processing on the user's scenario; a fourth step of setting a plurality of divided sections corresponding to unit stories by analyzing the user's scenario through the natural language processing by a scenario division unit (140); A fifth step in which a division section matching unit (150) matches and sets at least one template image stored in the template database (101) for each of the plurality of division sections according to a unit story; A sixth step in which an interactive selection unit (160) presents a selection option to a user for the divided section to which a plurality of template images are matched, and performs natural language processing on the user's response to the selection option, thereby matching and setting a template image corresponding to the user's selection to the corresponding divided section; a video personalization unit (170) cooperating with the interactive selection unit (160) to change the template image matched to the divided section according to a user's request and perform personalization processing; a seventh step in which a video combining and generating unit (180) combines a plurality of template images that are set to match the plurality of divided sections, respectively, to generate a user video for the user scenario, The first step includes: The template database (101) is configured to include a step of storing a plurality of composite elements in which a combination of a plurality of branch information and a template image is matched with the same unit story, branch information is matched for each of the template images, and the branch information corresponds to a mutation for the corresponding unit story, The personalization step includes the step of: personalizing the answer of the user by the video personalization unit (170) performing the natural language processing on the answer of the user and changing the template image by reflecting the result of the natural language processing; An interactive natural language processing-based video generation method, wherein the user's video generated in the seventh step is provided in conjunction with an external server.

2. A character generation and combination unit (110) generates a character of a user by using a user's photo provided by the user; The interactive natural language processing-based video generation method according to claim 1, further comprising a step in which the character generation and combination unit (110) synthesizes the user's character with a template image that is set to match the multiple divided sections.

3. The sixth step comprises: When the composite element corresponds to the divided section, the interactive selection unit (160) presents a plurality of branch information included in the composite element to a user via a user terminal (200); The interactive selection unit (160) identifies branch information selected by the user from the presented plurality of branch information; The interactive natural language processing based video generation method according to claim 1, further comprising a step of: the interactive selection unit (160) matching and setting a template image linked to the selected branch information to the corresponding division section.

4. A computer program stored on a storage medium for causing a computer to execute the interactive natural language processing-based video generation method according to claim 1.

Images