Three-dimensional video generation method

By decomposing the generated instructions and combining the three-dimensional material library, planning the storyboard content and matching character models, the problem of difficulty in generating three-dimensional videos in the existing technology is solved, and 360-degree free viewing angle and highly realistic three-dimensional video generation is achieved, improving the user experience and professionalism of the content.

CN119071564APending Publication Date: 2024-12-03SUZHOU BOTTLER ROBOT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410979462.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

It is difficult to generate three-dimensional videos in the prior art, and the existing Wensheng video technology can only generate two-dimensional videos based on neural networks, making it difficult to ensure the consistency of the front and back frame images, and does not support the output of long videos and three-dimensional content.

Method used

By providing instructions to generate three-dimensional videos, decompose them into structured descriptions, combining the three-dimensional material library to generate initial story scenes, planning storyboard content, matching character model and 3-dimensional material library, generating audio of character model, performing three-dimensional rendering and iterating processing, we obtain 3D video.

Benefits of technology

It realizes 3D video generation with a 360-degree free viewing angle, allowing users to modify during the viewing process, reducing manual modeling costs, improving the professionalism and fidelity of video content, and avoiding the single stereotype effect of character models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071564A_ABST
    Figure CN119071564A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a three-dimensional video generation method. The method comprises the following steps: providing an instruction for generating a three-dimensional video; decomposing the instruction into a structured description, and combining the structured description with a three-dimensional material library to generate an initial story scene; planning the split content according to the initial story scene and the structured description, and deconstructing the split content into a multi-round message; when the role model exists in the initial story scene, matching the role model with a three-dimensional material library; matching timbres in a voice model library according to the role model, and generating an audio of the role model and an audio of the paraphrasing; and performing three-dimensional rendering on the initial story scene, driving the role model, and performing iterative processing according to multiple rounds of messages to obtain a three-dimensional video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of video generation, and in particular to a three-dimensional video generation method. Background Art

[0002] In existing technologies, generative AI can automatically help users generate desired content by taking prompt information such as text and images as input. This type of method has high application value in the film, television, advertising and communications industries because it can improve production efficiency, reduce costs and enhance user experience.

[0003] Existing Wensheng video technology mainly draws images or creates videos by running diffusion models through high-performance graphics processing units. This type of method has two stages, namely the training stage and the deployment stage. Among them, the training stage usually requires a large number of deep learning servers to enable the model to learn a large number of images or videos; in the deployment stage, the user uses a photo or text as input to generate images or videos through the trained model. Based on the technical solutions of this type of method, it is generally necessary to wait for a period of time to generate the final content. Moreover, its essence is to generate images based on neural networks, so it can only be used to generate two-dimensional videos. It is difficult to ensure the consistency of the previous and next frame images to generate long videos of more than 1 minute, and it does not support the output of three-dimensional content.

[0004] In the field of computer graphics (CG), vertices, triangles, and textures are usually used to represent the three-dimensional information of a model when rendering a three-dimensional model. This method provides clear boundaries and surface details, which can be directly used for lighting calculations, texture mapping, and animation. However, this method requires a lot of computing resources to create and process high-quality models, and modeling may become difficult for irregular shapes or shapes with holes.

[0005] To address the above problems, no effective solution has been proposed yet.

[0006] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present invention and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology section of the present invention. Summary of the invention

[0007] The purpose of this application is to provide a three-dimensional video generation method to solve the above problems.

[0008] This application provides a three-dimensional video generation method, comprising the following steps:

[0009] Providing instructions for generating three-dimensional video;

[0010] Decomposing the instructions into structured descriptions, and combining the structured descriptions with a three-dimensional material library to generate an initial story scene;

[0011] Planning storyboard content according to the initial story scene and the structured description, and deconstructing the storyboard content into multiple rounds of messages;

[0012] When a character model exists in the initial story scene, matching the character model with the three-dimensional material library;

[0013] Matching the timbre in the voice model library according to the role model, and generating the audio of the role model and the audio of the narration;

[0014] The initial story scene is rendered in three dimensions, and the character model is driven to perform iterative processing according to the multiple rounds of messages to obtain a three-dimensional video.

[0015] Preferably, the instructions include generating a subject and a title for the three-dimensional video.

[0016] Preferably, the structured description includes generating an outline of the video content and generating a text description of the video scene, the birth position of the character model, the moving position of the character model, the action of the character model, the expression of the character model, and the language content of the character model.

[0017] Preferably, the multiple rounds of messages include main actions and movement information of multiple role models.

[0018] Preferably, in the step of "matching the timbre in the speech model library according to the role model", the timbre is matched according to the gender and age information of the role model.

[0019] Preferably, in the step of "generating the audio of the character model and the audio of the narration", it is set to generate them through the T9TS algorithm.

[0020] Preferably, in the step of "driving the character model", it is configured to drive the skeleton, mouth shape and expression of the character model.

[0021] Preferably, in the step of "generating an initial story scene", the initial block of the scene is first randomly initialized, and the WFC algorithm is used to generate other scene blocks, and then it is determined whether a rule conflict occurs; if a rule conflict occurs, the initial block of the scene is reinitialized and the WFC algorithm is used to generate other scene blocks; if there is no rule conflict, the three-dimensional model of each block is placed using the line graph algorithm.

[0022] Preferably, the multi-round messages include role model information, audio information, and location information, and the role model information includes role model expression information and role model action information.

[0023] Preferably, the modified 3D video is automatically regenerated according to the story content modified during the playing of the 3D video.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention can enable users to watch dynamic stories with a 360-degree free perspective by generating three-dimensional videos, and can allow users to make a large number of modifications during the viewing process, thereby improving the user experience. It has good controllability in the process of intelligent generation, so that users can generate three-dimensional videos more conveniently.

[0026] 2. The present invention automatically generates scene layouts through the three-dimensional material library setting, which can quickly generate diverse random scene layouts, thereby effectively reducing the cost of manual modeling.

[0027] 3. The present invention decomposes the input instructions by setting a large language model, and can then accurately generate a more professional video content description, thereby stably generating story scenes and role models.

[0028] 4. The present invention can plan in advance and develop autonomously in a logical direction by setting the behavior and interaction of the role model, thereby avoiding the role model display effect being single and stereotyped. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 It is a flow chart of a three-dimensional video generation method provided in an embodiment of this application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be described clearly and completely below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0032] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "upper", "middle", "lower", "inner", "outer", "front", "back", etc. are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. The terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The following describes an implementation method of the present invention based on its overall structure.

[0033] Reference Figure 1 As shown, the embodiment of the present application provides a three-dimensional video generation method, comprising the following steps:

[0034] Providing instructions for generating three-dimensional video;

[0035] Decomposing the instructions into structured descriptions, and combining the structured descriptions with the three-dimensional material library to generate an initial story scene;

[0036] Plan the storyboard content based on the initial story scene and structured description, and deconstruct the storyboard content into multiple rounds of messages;

[0037] When a character model exists in the initial story scene, the character model is matched with the three-dimensional material library;

[0038] Matching the timbre in the voice model library according to the role model, and generating the audio of the role model and the audio of the narration;

[0039] The initial story scene is rendered in three dimensions, and the character model is driven to iterate based on multiple rounds of messages to obtain a three-dimensional video.

[0040] In existing video generation methods or products, images are generated based on neural networks, and therefore can only be used to generate two-dimensional videos. It is difficult to ensure the consistency of previous and subsequent frame images to generate long videos of more than 1 minute.

[0041] In the present application, those skilled in the art can use the above method to generate a three-dimensional video. That is, the above fully automatic video generation method can be used to produce a three-dimensional video.

[0042] Specifically, those skilled in the art may provide instructions for generating a three-dimensional video. Preferably, the instructions include the subject and title of the generated three-dimensional video.

[0043] Further, the subject and title are decomposed. In this embodiment, the subject and title can be decomposed into a structured description. Preferably, the structured description can include an outline of the generated video content and a text description of the generated video scene.

[0044] In one embodiment, the instruction may also include a role model. It is understood that in the aforementioned case, the structured description may also include precise information such as the birth location of the role model, the moving location of the role model, the actions of the role model, the expressions of the role model, and the language content of the role model.

[0045] According to the above process, after receiving the instruction to generate a three-dimensional video, the instruction is decomposed into an outline of the video content and a text description of the video scene, so that the information of the instruction can be completely processed.

[0046] After the decomposition instruction is structured, it can be combined with the three-dimensional material library to generate the initial story scene. It is understandable that in this application, an AI workstation can be set up. By setting up a large language model, a three-dimensional material library and a language model library in the AI ​​workstation, the three-dimensional video can be fully automatically generated.

[0047] With the above settings, the instructions can be decomposed through the large language model in the AI ​​workstation, and the outline of the video content and the text description of the video scene after the instruction decomposition can be automatically obtained.

[0048] The three-dimensional material library in the AI ​​workstation can also be matched and synthesized with the decomposed structured description to obtain an initial story scene, wherein the generated initial story scene can include all the content and specific details.

[0049] In the above process, the initial story scene can be automatically generated through the AI ​​workstation. Furthermore, the storyboard content can be planned based on the initial story scene and the structured description.

[0050] It will be appreciated that, in the absence of a character model in the scene, only the initial story scene may be generated and / or the audio of the narration may be generated.

[0051] In the process of processing the storyboard content, the storyboard content can be further deconstructed into multiple rounds of messages. Preferably, the multiple rounds of messages include main actions and motion information of multiple role models.

[0052] In this embodiment, those skilled in the art can set the multi-round message to be multiple messages. For example, assuming that the character in the instruction needs to go from A from (6, 5) to (7, 5), the message is:

[0053] [{"target":"A","position":"X=6,Y=5,Z=0",}],

[0054] [{"target":"a","position":"X=7,Y=5,Z=0",}].

[0055] In the process of generating a 3D video, when there is a character model in the initial story scene, the character model is matched with the 3D material library. Specifically, the 3D material library stores the 3D model. In the process of matching the character model with the 3D model in the 3D material library, the character model can be determined and generated.

[0056] It is understandable that in this application, the three-dimensional model in the three-dimensional material library can use a Gaussian model, which can optimize the quality of the model generated by this application.

[0057] According to the above process, the initial story scene and role model can be automatically generated. In the process of continuing to generate, the role model needs to be further improved.

[0058] Specifically, the timbre of the character model can be matched, and the audio and narration audio can be generated for the character model. Those skilled in the art can match the timbre according to the gender and age information of the character model. Further, the audio of the character model can be generated by the T9TS algorithm. The audio of the narration can also be further generated by the T9TS algorithm.

[0059] Based on the above process, those skilled in the art can generate an initial story scene and a role model, as well as audio of the role model and audio of the narration.

[0060] In the process of continuing to generate, it is necessary to drive the character model, and then generate the video. Preferably, it can be set to drive the skeleton, mouth shape and expression of the character model.

[0061] For the initial story scene, those skilled in the art may also set it to be rendered in three dimensions. That is, the specific story content generated by the large language model may be processed by a three-dimensional rendering engine to render the initial story scene from multiple aspects. The rendered initial story scene is then combined with the driven character model to generate a preliminary video.

[0062] In this embodiment, those skilled in the art can combine multiple rounds of messages to iteratively process the initial video, and then obtain the final rendered three-dimensional video.

[0063] It is understandable that the character model can be processed during the iterative processing. Specifically, for example, if the character model Xiao Ming walks from point A to point B, stops for a while, and then walks to point C, then the instruction content will be decomposed into three messages. First, the first data is parsed, and Xiao Ming is controlled to go from point A to point B; after reaching point B, the second message is iteratively parsed to control Xiao Ming to stay; the third message is iteratively parsed, and after staying, Xiao Ming is controlled to go from point B to point C. And so on to complete the whole story. In addition to mobile control, there are also language control, action control, etc., with similar principles.

[0064] In the above process, the initial story scene can be automatically generated by the algorithm. Preferably, in the "generating initial story scene" step, the initial block of the scene is first randomly initialized, and the WFC algorithm is used to generate other scene blocks, and then it is determined whether a rule conflict occurs; if a rule conflict occurs, the initial block of the scene is reinitialized and the WFC algorithm is used to generate other scene blocks; if there is no rule conflict, the three-dimensional model of each block is placed through the line graph algorithm.

[0065] Through the above detailed steps, the initial story scene without rule conflicts can be stably output. It can be understood that in this embodiment, those skilled in the art can stably generate scene blocks by using the WFC algorithm. Then, multiple blocks can be accurately placed by using the line graph algorithm.

[0066] In the above process of generating 3D video, those skilled in the art can deconstruct the storyboard content into multiple rounds of messages. Preferably, the multiple rounds of messages include role model information, audio information, and position information, and the role model information includes role model expression information and role model action information.

[0067] The specific process is to receive multiple messages and then analyze each message in turn. In the process of analyzing the messages, character information, audio information, and location information in the messages can be obtained from the analyzed messages. In addition, character model information is also included, wherein the character model information includes character model expression information and character model action information.

[0068] It is worth noting that those skilled in the art can generate the lip shape of the character model according to the audio information. Specifically, the lip shape of the character model can be controlled, thereby making the character model more realistic and natural in the three-dimensional video.

[0069] In one embodiment, preferably, the modified 3D video is automatically regenerated according to the story content modified during the 3D video playback. That is, the user can modify the story content while watching the 3D video, and then automatically regenerate a new 3D video.

[0070] Specifically, the modified story content can be input during the process of watching the 3D video, and the existing 3D video can be modified according to the modified story content read according to the above method steps for generating the 3D video. Thus, the 3D video can be automatically modified and generated at any time and in real time.

[0071] Although different specific embodiments are mentioned in the content of this application, this application is not limited to the situations described in the industry standards or embodiments, etc. Some industry standards or slightly modified implementation plans based on the implementation of the custom methods or embodiments can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above embodiments. The embodiments of data acquisition, processing, output, judgment methods, etc. after these modifications or deformations can still fall within the scope of the optional implementation plans of this application.

[0072] Although the present application has been described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present application without departing from the spirit of the present application. It is intended that the attached embodiments include these modifications and changes without departing from the present application.

Claims

1. A three-dimensional video generation method, It is characterized in that The following steps are involved: Providing instructions for generating three-dimensional video; Decomposing the instructions into structured descriptions, and combining the structured descriptions with a three-dimensional material library to generate an initial story scene; Planning storyboard content according to the initial story scene and the structured description, and deconstructing the storyboard content into multiple rounds of messages; When a character model exists in the initial story scene, matching the character model with the three-dimensional material library; Matching the timbre in the voice model library according to the role model, and generating the audio of the role model and the audio of the narration; The initial story scene is rendered in three dimensions, and the character model is driven to perform iterative processing according to the multiple rounds of messages to obtain a three-dimensional video.

2. The three-dimensional video generation method according to claim 1, It is characterized in that The instructions include generating a subject and a title for the 3D video.

3. The three-dimensional video generation method according to claim 1, It is characterized in that The structured description includes generating an outline of the video content and generating a text description of the video scene, the birth position of the character model, the moving position of the character model, the action of the character model, the expression of the character model, and the language content of the character model.

4. The three-dimensional video generation method according to claim 1, It is characterized in that The multiple rounds of messages include main actions and movement information of multiple role models.

5. The three-dimensional video generation method according to claim 1, It is characterized in that In the step of "matching the timbre in the voice model library according to the role model", the timbre is matched according to the gender and age information of the role model.

6. The three-dimensional video generation method according to claim 1, It is characterized in that In the "Generate audio for character model and narration" step, set the audio to be generated using the T9TS algorithm.

7. The three-dimensional video generation method according to claim 1, It is characterized in that In the "driving the character model" step, it is set to drive the skeleton, mouth shape and expression of the character model.

8. The three-dimensional video generation method according to claim 1, It is characterized in that In the "Generate Initial Story Scene" step, the initial scene block is first randomly initialized, the WFC algorithm is used to generate other scene blocks, and then it is determined whether there is a rule conflict; If there is a rule conflict, the initial block of the scene is reinitialized and the WFC algorithm is used to generate other scene blocks; If there is no rule conflict, the three-dimensional model of each block is placed using the row graph algorithm.

9. The three-dimensional video generation method according to claim 1, It is characterized in that The multi-round messages include role model information, audio information, and location information, and the role model information includes role model expression information and role model action information.

10. The three-dimensional video generation method according to claim 1, It is characterized in that The modified 3D video is automatically regenerated according to the story content modified during the playing of the 3D video.