Method and system for generating video based on text information by means of three-dimensional rendering

By generating format scripts through large language models and combining 3D models, the rendering weight is determined for differentiated rendering, solving the problems of uncontrollability and low resolution of video generation in the prior art, and achieving efficient and deterministic video generation.

CN118803387BActive Publication Date: 2024-11-22HUNAN SANYUE SUWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411281800.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-11-22
Estimated Expiration
2044-09-13

AI Technical Summary

Technical Problem

The existing diffusion model based on Transformer architecture generates videos with low content resolution, poor consistency and uncontrollability, and the three-dimensional rendering process takes a long time, affecting the video production speed.

Method used

Generate format scripts through large language models, combine 3D models and camera perspective data, determine the rendering weight of the 3D models in each frame of image, perform differentiated rendering, and generate videos using the rendering engine to ensure high resolution and consistency of the video.

Benefits of technology

It improves the certainty and quality of video generation, reduces uncontrollable factors, shortens rendering time, and improves video production speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118803387B_ABST
    Figure CN118803387B_ABST
Patent Text Reader

Abstract

The present application relates to the field of image processing technology, and specifically to a method and system for generating videos based on text information with the help of three-dimensional rendering. The method includes: converting user text into a formatted script through a large language model script generator, generating a motion 3D scene based on the formatted script combined with a 3D model driven by a large model, obtaining each 3D model and camera perspective data in the 3D scene described by the formatted script, and obtaining an unrendered video; obtaining the nouns in the formatted script corresponding to each 3D model, determining the nouns corresponding to each 3D model; determining all mapping data pairs of each 3D model in each frame of the image; determining the rendering weight and shortest side length of each 3D model in each frame of the image, reconstructing all 3D models, rendering the unrendered video by means of a rendering engine, and finally obtaining a video. The present application can perform differentiated rendering of different 3D models, speed up video rendering, and solve the problem of long rendering time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for generating a video based on text information by means of three-dimensional rendering. Background Art

[0002] With the rapid development of artificial intelligence technology, text-to-video generation has become a hot research field. By using AI big models, it is possible to automatically generate corresponding video content from natural language text. The principle of traditional text-to-video technology is to use a large model composed of large-scale deep neural networks to achieve the conversion of natural language text to target results. During this period, noise data is diffused to imitate the relationship between training sample system text and video.

[0003] In the prior art, the synthesized frame video based on the diffusion model of the Transformer architecture may have defects such as low resolution, poor consistency, and uncontrollable generated content. For example, the synthesized frame video based on the diffusion model of the Transformer architecture may have aliasing defects, and the frame video may show behaviors or actions that are contrary to human common sense. Using a large model to generate a script text in natural language format, and then using a three-dimensional engine to parse the format script to render the text video, can effectively reduce the uncontrollability of the large model synthesized video and avoid consistency errors in the text video.

[0004] In the process of generating formatted text from the model and then generating video through three-dimensional rendering, the existing technology needs to spend a lot of time to render the three-dimensional model in the script scene to improve the realism of the image video. There are problems such as randomness and uncontrollability of the output content and low video resolution. At the same time, the existing technology rendering will spend a lot of time on deep rendering of unnecessary model details, resulting in too long video rendering time, which greatly affects the video production speed. Summary of the invention

[0005] In order to solve the above technical problems, a method and system for generating a video based on text information by means of three-dimensional rendering are provided to solve the existing problems.

[0006] The solution to the technical problem of the present application is to provide a method and system for generating a video based on text information by means of three-dimensional rendering, including the following steps:

[0007] In a first aspect, an embodiment of the present application provides a method for generating a video based on text information by means of three-dimensional rendering, the method comprising the following steps:

[0008] Obtain user text and use a large language model script generator to obtain a formatted script, obtain an unrendered video based on the 3D models and camera view data in the 3D scene described in the formatted script; obtain nouns in the formatted script corresponding to each 3D model based on the formatted script and the 3D models of all images in the unrendered video;

[0009] According to the matching of the nouns in the format script corresponding to each 3D model and the nouns in the user text, the nouns corresponding to each 3D model are determined; according to the number of pixels in each 3D model, the distance between each 3D model and the camera view data, and the mapping relationship between each noun and each frame of image, all mapping data pairs of each 3D model in each frame of image are determined;

[0010] Based on the mapping data pairs, analyzing the position distribution of each noun corresponding to each 3D model in each frame of image in the user text, and determining the rendering weight of each 3D model in each frame of image;

[0011] Based on the rendering weight, the shortest side length of each 3D model in each frame image is determined, all 3D models are reconstructed, and the unrendered video is rendered by means of a rendering engine to finally obtain a raw video.

[0012] Preferably, the method for obtaining the unrendered video is:

[0013] Each 3D model and camera focus position is input into a three-dimensional model image software to obtain an unrendered video, and the 3D model to which each pixel point of each frame image in the unrendered video belongs is marked.

[0014] Preferably, the step of obtaining the noun corresponding to each 3D model in the format script includes:

[0015] A noun extraction algorithm is used on the format script to obtain all nouns in the format script;

[0016] Each noun is labeled with the corresponding 3D model in all images in the unrendered video to obtain the noun corresponding to each 3D model in the format script.

[0017] Preferably, determining the nouns corresponding to each 3D model includes:

[0018] A noun extraction algorithm is used to obtain all nouns in the user text. A string fuzzy matching algorithm is used to match the nouns corresponding to each 3D model in the format text with the nouns in the user text, and the 3D models corresponding to the nouns matching the format text and the user text are used as the nouns corresponding to each 3D model.

[0019] Preferably, determining all mapping data pairs of each 3D model in each frame of image includes:

[0020] Calculate the distance between the geometric center of each 3D model in each frame image and the focal position in the camera view data, and record it as the position weight;

[0021] Calculate the product of the number of pixel points corresponding to the mark of each 3D model in each frame of image and the position weight as the presentation weight of each 3D model in each frame of image;

[0022] Arrange the presentation weights of each 3D model in all images in the order of the image frame numbers to form a presentation weight sequence, and obtain all maximum value points in the presentation weight sequence of each 3D model;

[0023] Analyze the position relationship of each noun corresponding to each 3D model in the user text, and determine the relative text position of each noun corresponding to each 3D model;

[0024] Analyze the difference between the relative text position of each noun and the image frame number corresponding to the maximum point, and determine the mapping discrimination value between each noun and each maximum point corresponding to each 3D model in combination with the presentation weight corresponding to the maximum point;

[0025] For each noun and the mapping discrimination value of all maximum points, the maximum point and image frame number corresponding to the maximum value of the mapping discrimination value of each noun are selected, and each noun and the corresponding image frame number are combined into a two-dimensional array to obtain all mapping data pairs of each 3D model in each frame image.

[0026] Preferably, determining the relative text position of each noun corresponding to each 3D model includes:

[0027] For each noun corresponding to each 3D model, determine the character position of the first character of each noun in the user text;

[0028] The ratio between the character position of each noun and the number of all characters in the user text is used as the relative text position of each noun corresponding to each 3D model.

[0029] Preferably, the step of determining the mapping discriminant value between each noun and each maximum point corresponding to each 3D model includes: ,in, is the mapping discriminant value between the rth noun and the kth maximum point corresponding to the mth 3D model, is the relative text position of the rth noun corresponding to the mth 3D model, is the number of all images in the unrendered video, is the image frame number corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the presentation weight corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the ceiling function, The default value is greater than zero.

[0030] Preferably, determining the rendering weight of each 3D model in each frame of image includes:

[0031] Calculate the average of the relative text positions of the nouns corresponding to all the mapping data pairs of all the 3D models in each frame image as the video text position of each frame image;

[0032] Recording the difference between the relative text position of each noun corresponding to each 3D model in each frame of image and the video text position as a first difference;

[0033] The sum of the first differences of all nouns corresponding to each 3D model in each frame of image is used as the rendering weight of each 3D model in each frame of image.

[0034] Preferably, determining the shortest side length of each 3D model in each frame of image and reconstructing all 3D models includes:

[0035] Get the maximum value of the rendering weights of all 3D models in each frame image;

[0036] Recording the ratio of the rendering weight of each 3D model in each frame of image to the maximum value as a first ratio;

[0037] The sum of the first ratio and a preset edge length threshold is used as the shortest edge length of each 3D model in each frame of the image;

[0038] Each 3D model and the corresponding shortest side length are input into a triangulation algorithm to obtain a surface reconstructed 3D model; and a rendering engine is used to render the reconstructed 3D model.

[0039] In a second aspect, an embodiment of the present application also provides a system for generating a video based on text information with the aid of three-dimensional rendering, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the system implements any one of the steps of the method for generating a video based on text information with the aid of three-dimensional rendering as described above.

[0040] This application has at least the following beneficial effects:

[0041] The present application converts user text into a formatted script through a large language model, and obtains an unrendered video based on the formatted script and a 3D database. The beneficial effect is that a script of a specified format can be obtained through a large language model, which reduces the possibility of defects in video frames and reduces the occurrence of uncontrollable factors. The present application analyzes the weight of each 3D model in the image through the presentation weight of each 3D model in each frame image and the mapping data pair to reflect the importance of each 3D model presented in the corresponding image, and considers the mapping relationship between the appearance of the text and the corresponding image to reflect the correspondence between the position of the text and the moment of image presentation; further, the correlation strength between each 3D model and the corresponding image is analyzed through the rendering weight of each 3D model in each frame image, so as to analyze the different correlation strengths in the image. 3D models are rendered to different degrees; based on the rendering weight, the shortest side length of each 3D model in each frame image is determined, all 3D models are reconstructed, and differentiated rendering of different 3D models is completed, thereby speeding up video rendering and solving the problem of long rendering time; compared with the diffusion model of the traditional Transformer architecture, the present application converts user text into a formatted script, combines it with a 3D model, and renders it in the form of a rendering engine to obtain a video, so that the generated video content has a high degree of certainty and uniqueness, reduces the uncertainty in the video generation process, ensures the quality and expected effect of the final video, improves the quality of the rendered video, reduces the rendering power consumption, and at the same time can adjust the rendering accuracy according to needs and adjust the video resolution at any time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The method of generating video based on text information with the aid of three-dimensional rendering of the present application is further described in detail below in conjunction with the accompanying drawings.

[0043] Figure 1 A flowchart of the steps of a method for generating a video based on text information by means of three-dimensional rendering provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of the steps of a method for obtaining the rendering weights of each 3D model in each frame of an image provided in an embodiment of the present application;

[0045] Figure 3 A flowchart of the shortest side length of each 3D model in each frame of image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clear, the following is a further detailed description of the method and system for generating a video based on text information by means of three-dimensional rendering proposed in the present application in combination with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0048] See also Figure 1 , which shows a flowchart of a method for generating a video based on text information by means of three-dimensional rendering provided by an embodiment of the present application, the method comprising the following steps:

[0049] Step 1, obtain user text, and use the large language model script generator to obtain a formatted script; obtain an unrendered video based on the 3D models and camera perspective data in the 3D scene described in the formatted script; based on the formatted script and the 3D models of all images in the unrendered video, obtain the nouns in the formatted script corresponding to each 3D model.

[0050] The traditional video generation technology based on diffusion model directly generates continuous video frames through text, and then connects the video frames to generate video. When generating continuous video frames, the large diffusion-based model has limited logical capabilities or insufficient understanding of the physical world. At this time, uncontrollable consistency errors are prone to occur in the process of video generation by the diffusion model. For example, during the movement of continuous frames of the video, the target object's performance characteristics change due to lighting, occlusion, blur and perspective changes. At this time, the content synthesized by the large diffusion-based model has uncontrollable factors. The content of the synthesized video is inconsistent with the real physical rules.

[0051] Therefore, in this embodiment, a script is generated by using text, and a 3D model is driven based on the script and a large model to form a motion scene, and the motion scene is shot and rendered to obtain a live video, which can effectively avoid consistency errors. Specifically:

[0052] The user text is obtained as the input of the large language model script generator, and the output is a formatted script in a standard format, including time, sound, weather effects, environmental effects, lenses, scene models, character models, prop models, expressions, actions, etc.

[0053] Among them, the large language model script generator is a large language model obtained after fine-tuning the existing large language model. In this embodiment, GPT, BERT and other models are used as large language models for fine-tuning. The fine-tuning method of the large language model is to use Adam optimization to adjust the parameters of the large language model based on the input scenario requirements and the generated target format script as training data. The specific process of large language model fine-tuning (Fine-tuning) is a well-known technology commonly used in the neural network training process, which will not be repeated here.

[0054] It should be noted that the big model is represented by a large artificial intelligence model, especially a deep learning model used for text generation, 3D animation generation and rendering. In this embodiment, the big model used is a large language model, which converts the user text into a formatted script. The script text can make the video content to be generated clearer, so that the generated content can be unique and avoid defects or actions that go against human common sense.

[0055] A 3D model database is obtained, and a motion 3D scene is generated according to the format script and the 3D model database. In the present embodiment, the format script and the 3D model database are input into the Narrator model to obtain data of various 3D models and camera perspective in the 3D scene described in the script text, wherein the camera perspective data includes data at multiple moments, and the camera perspective at moment t includes the position of the camera in the 3D scene and the focus position of the camera. It should be noted that the 3D model database and the Narrator model are well-known technologies in the field of artificial intelligence and will not be described in detail here.

[0056] Each 3D model and camera focus position is input into a 3D model image software for processing to obtain an unrendered video, and the 3D model to which each pixel point of each frame image in the unrendered video belongs is marked by the 3D model image software.

[0057] Preferably, in this embodiment, Blender is used to obtain unrendered video, wherein the use of 3D model image software is a well-known technology in the field of video rendering and will not be repeated here. As other implementation methods, the implementer may use other image software of the prior art, such as Autodesk Maya, ZBrush, etc., and this embodiment does not limit this.

[0058] A noun extraction algorithm is used on the format script to obtain all nouns in the format script.

[0059] Preferably, in this embodiment, the spaCy library is used to extract noun phrases in text data, wherein the spaCy library is a well-known technology and will not be described in detail here. As other implementation methods, the implementer may adopt other methods of the prior art, such as the NLTK library, etc. This embodiment does not impose any special restrictions on this.

[0060] Each noun in the format script is labeled with the corresponding 3D model in all images in the unrendered video to obtain each noun in the format script corresponding to each 3D model; it should be noted that the 3D model without a corresponding noun is recorded as a non-primary 3D model.

[0061] Preferably, the nouns are mapped to the 3D models by manual labeling, thereby completing the labeling.

[0062] In this embodiment, the specific scene of the format script is that it is a sunny afternoon with a clear blue sky. Zhang is strolling in the downtown with Aunt Li. Suddenly, the weather changes drastically, and the sky is overcast and it looks like it is going to rain. Zhang and Aunt Li say they want to go home immediately to avoid getting wet in the rain, so they take a car home. Before Zhang goes home and enters the room, he is actually caught in a few drops of rain. Then Zhang goes to the dressing table in his bedroom, removes his makeup, takes off his earrings and puts them in the earring box. When he takes off his earrings, there is a violent lightning in the sky. The strong light of the lightning shines into the room through the window, and there is a strong indoor ambient light flashing. The thunder is a little late, which scares Zhang so much that he turns pale. Then the continuous lightning ends and Zhang's expression slowly calms down.

[0063] Time point: afternoon, sound: thunder, weather: sunny and clear sky, sudden change of weather with dark clouds and rain, environment: lightning, flashing light from the window into the bedroom, scene model: downtown scene model, residence, bedroom (with windows and glass), role models: Zhang (young woman in any clothes), Aunt Li (middle-aged woman in any clothes), pedestrians in the downtown (models of men, women and children of all ages, in any clothes), prop models: dressing table, earrings, earring box, vehicle: any car model (can carry people and start), expression: pale, expression slowly calms down, role actions: strolling (walking), opening the car door, getting in the car, closing the car door, getting out of the car, sitting down, taking off the earrings, putting on the earrings.

[0064] In this embodiment, the process of generating a formatted script based on the above-mentioned script scene is as follows: the above-mentioned described scene is input into the trained large language model script generator, and the corresponding formatted script is output, which describes the entire scene process, related storylines, required model assets, and various special effects and other elements. At this point, the 3D models and camera focus positions in the scene described by the formatted script are obtained, and the nouns corresponding to the unrendered video and each 3D model in the formatted script are obtained.

[0065] Step 2: Determine the nouns corresponding to each 3D model based on the matching of the nouns in the format script corresponding to each 3D model and the nouns in the user text; determine all mapping data pairs of each 3D model in each frame of image based on the number of pixels in each 3D model, the distance between each 3D model and the camera perspective data, and the mapping relationship between each corresponding noun and each frame of image.

[0066] The important 3D models in the scene of each frame of video image are determined by text, and different 3D models in each frame of video image are rendered at different depths, with the 3D models with large correlation with text being rendered as the focus, so as to speed up the rendering speed. Therefore, it is necessary to correspond each frame of video image with text, so as to determine the important 3D models in the frame of video image through text and render them more carefully.

[0067] For a 3D model in the scene, when the number of pixels it occupies in different frames of the unrendered video gradually increases, it means that the performance of the 3D scene may have progressed to the noun corresponding to the 3D model in the text. When the number of pixels occupied by the 3D model gradually decreases or even disappears, it means that the 3D scene has completed the text performance at the corresponding noun. Therefore, the changes in the 3D model in different frames of the unrendered video can be used to map the video frame to the text position.

[0068] Furthermore, when a 3D model is closer to the focus of the camera, or a 3D model occupies more pixels in the unrendered video, the text representing the scene performance is closer to the noun corresponding to the 3D model; based on the above analysis, the greater the weight of each 3D model in a frame of the unrendered video, the more likely the 3D model should appear in the frame, and a presentation weight sequence is constructed; specifically:

[0069] The distance between the geometric center of each 3D model in each frame image and the focal position in the camera view data is calculated, and recorded as the position weight; the product of the number of marked pixel points corresponding to each 3D model in each frame image and the position weight is calculated as the presentation weight of each 3D model in each frame image.

[0070] In this embodiment, the presentation weight of each 3D model in each frame of image is: ,in is the rendering weight of the mth 3D model in the nth frame image, is the position weight of the mth 3D model in the nth frame image, is the number of pixels corresponding to the mark of the mth 3D model in the nth frame image.

[0071] It should be noted that the position weight and the number of pixels The larger the value, the more important performance moments of the m-th 3D model appear in the n-th frame image, and the corresponding presentation weight The larger the value is, the closer the performance of the text in the 3D scene of the nth frame image is to the text position corresponding to the mth 3D model.

[0072] Furthermore, the video frame image is associated with the text position according to the presentation weight, which indicates the position of the text at which the content performed in different frame images in the unrendered video is performed. This helps to perform differentiated rendering of the 3D model at different times through the text content and improve rendering efficiency.

[0073] First, the format text includes scene models, role models, and prop models. The nouns in the format text correspond to specific 3D models. Based on the correspondence between each 3D model and the format text, the correspondence between the nouns in the format text and the nouns in the user text is analyzed to obtain the mapping between each 3D model and the user text, specifically:

[0074] A string fuzzy matching algorithm is used to match the nouns corresponding to each 3D model in the format script with the nouns in the user text, and the 3D models corresponding to the nouns matching the format text and the user text are used as the nouns corresponding to each 3D model.

[0075] Preferably, in this embodiment, the TF-IDF algorithm is used to match each noun in the format text corresponding to each 3D model with the user text to obtain the noun corresponding to each 3D model, wherein the TF-IDF algorithm is a well-known technology and will not be described in detail here. As other implementation methods, the implementer may adopt other methods of the prior art, such as Levenshtein distance, Jaccard similarity, etc., and this embodiment does not impose any special restrictions on this.

[0076] Furthermore, the positional relationship between the nouns corresponding to the 3D model and the nouns in the user text is analyzed to determine the relative text position, specifically:

[0077] For each noun corresponding to each 3D model, determine the character position of the first character in each noun in the user text; and use the ratio between the character position of each noun and the number of all characters in the user text as the relative text position of each noun corresponding to each 3D model.

[0078] In this embodiment, the mth 3D model corresponds to R nouns, and the first character of the rth noun is the first character of the user text. characters, and the user text has a total of G characters, then the relative text position of the rth noun is: ,in, is the relative text position of the rth noun corresponding to the mth 3D model.

[0079] Furthermore, the mapping relationship between the video frame and the text position is analyzed, and a mapping discriminant value is determined to reflect the mapping relationship between the noun and the frame image, specifically:

[0080] Arrange the presentation weights of each 3D model in all images in the order of the image frame numbers to form a presentation weight sequence, and obtain all the maximum value points in the presentation weight sequence of each 3D model; the mapping discriminant value is: ,in, is the mapping discriminant value between the rth noun and the kth maximum point corresponding to the mth 3D model, is the relative text position of the rth noun corresponding to the mth 3D model, is the number of all images in the unrendered video, is the image frame number corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the presentation weight corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the ceiling function, The default value is greater than 0.

[0081] It should be noted that the greater the presentation weight corresponding to the maximum point, the more likely the corresponding 3D model should appear in the frame image of the corresponding frame number; secondly, It shows that the timeline and text of the 3D scene performance are performed synchronously. When the r-th noun is performed, the corresponding image frame, the smaller the interval between the image frame corresponding to the r-th noun and the image frame corresponding to the k-th maximum point, the larger the mapping discrimination value, which means that the frame image corresponding to the k-th maximum point is more likely to be an important appearance moment of the r-th noun.

[0082] For each noun and the mapping discrimination value of all maximum points, the maximum point and image frame number corresponding to the maximum value of the mapping discrimination value of each noun are selected, and each noun and the corresponding image frame number are combined into a two-dimensional array to obtain all mapping data pairs of each 3D model in each frame image.

[0083] In this embodiment, if the rth noun corresponding to the mth 3D model is The mapping discriminant value of the maximum point is the largest, then The image frame number corresponding to the maximum point of is , associate the rth noun with the image frame number To form a two-dimensional array, recorded as mapping data pairs, then the A mapping data pair of the mth 3D model of the frame image is , indicating that the important occurrence moment of the rth noun is in the unrendered video In the frame image, the above method is used to obtain all mapping data pairs of the mth 3D model of the kth frame image according to all nouns corresponding to the mth 3D model.

[0084] At this point, all mapping data pairs of each 3D model in each frame of image are obtained.

[0085] Step 3: Based on the mapping data pair, analyze the position distribution of the nouns corresponding to each 3D model in each frame image in the user text, and determine the rendering weight of each 3D model in each frame image.

[0086] Furthermore, according to the text position corresponding to each frame image, the text information is analyzed to filter out the 3D model closely related to the text in each frame image, and the rendering weight is determined, which is specifically:

[0087] The average of the relative text positions of the nouns corresponding to all the mapping data pairs of all the 3D models in each frame of image is calculated as the video text position of each frame of image.

[0088] In this embodiment, it is assumed that there are Q mapping data pairs of all 3D models of the nth frame image, and the Q mapping data pairs correspond to Q nouns. The relative text position of each noun is calculated, and the average of the relative text positions of the Q nouns is taken as the video text position of the nth frame image.

[0089] It should be noted that, assuming that all mapping data pairs in which the second element is equal to n are obtained as the mapping data pairs of the nth frame image, then the first element of all mapping data pairs of the nth frame image is all nouns corresponding to the nth frame image. If there is no mapping data pair for a certain frame image, then the mapping data pair of the n-1 frame image and the n+1 frame image is used as the mapping data pair of the nth frame image. If there is no corresponding mapping data pair between the n-1 frame image and the n+1 frame image, then the mapping data pair of the n-2 frame image and the n+2 frame image is used as the mapping data pair of the nth frame image, and so on, to obtain all mapping data pairs for each frame image.

[0090] The difference between the relative text position of each noun corresponding to each 3D model in each frame image and the video text position is recorded as the first difference; the sum of the first differences of all nouns corresponding to each 3D model in each frame image is used as the rendering weight of each 3D model in each frame image.

[0091] In this embodiment, the rendering weight of each 3D model in each frame image is: ,in is the rendering weight of the mth 3D model in the nth frame image, is the relative text position of the rth noun corresponding to the mth 3D model in the nth frame image, is the video text position of the nth frame image, R is the number of all nouns corresponding to the mth 3D model in the nth frame image, The first difference.

[0092] It should be noted that the smaller the difference between the relative text position and the video text position is, the stronger the relationship between the mth 3D model in the text and the nth frame image is, the smaller the rendering weight of the mth 3D model in the nth frame is, and more attention should be paid to rendering the mth 3D model when rendering the nth frame image.

[0093] Further, the step flow chart of the method for obtaining the rendering weight of each 3D model in each frame image provided by the embodiment of the present application is as follows: Figure 2 shown.

[0094] At this point, the rendering weight of each 3D model in each frame image is obtained.

[0095] Step 4, based on the rendering weight, determine the shortest side length of each 3D model in each frame image, reconstruct all 3D models, render the unrendered video by means of a rendering engine, and finally obtain the Wensheng video.

[0096] Furthermore, based on the rendering weight, the surface of each 3D model in each frame image is reconstructed to determine the shortest side length of the triangulation algorithm (Delaunay algorithm) during the rendering process, specifically:

[0097] Obtain the maximum value of the rendering weights of all 3D models in each frame image; record the ratio of the rendering weight of each 3D model in each frame image to the maximum value as a first ratio; and take the sum of the first ratio and a preset edge length threshold as the shortest edge length of each 3D model in each frame image.

[0098] It should be noted that, if there is a non-main 3D model in each frame image, the shortest side length of the non-main 3D model is the preset side length threshold;

[0099] Preferably, in this embodiment, the shortest side length of each 3D model in each frame image is: ,in, is the shortest side length of the mth 3D model in the nth frame image, is the rendering weight of the mth 3D model in the nth frame image, is the maximum value of the rendering weights of all 3D models in the nth frame image, L is the preset side length threshold, and its value is 2 mm. It should be noted that, as other implementation modes, the implementer can set it according to the actual situation. If the mth 3D model is a non-primary 3D model, the shortest side length of the mth 3D model is L.

[0100] Further, the flowchart of the shortest side length of each 3D model in each frame image provided by the embodiment of the present application is as follows: Figure 3 shown.

[0101] Furthermore, based on the shortest side length of each 3D model in each frame image, all 3D models are reconstructed and the unrendered video is rendered, specifically:

[0102] Each 3D model and the corresponding shortest side length are input into the triangulation algorithm to obtain a surface reconstructed 3D model, wherein the triangulation algorithm is a well-known technology and will not be described in detail here; the reconstructed 3D model is rendered using a rendering engine to obtain a visual video.

[0103] Preferably, in this embodiment, the i3dAct real-time three-dimensional engine is used to render the reconstructed 3D model. It should be noted that the rendering engine refers to the process of using a computer graphics engine to generate images or animations. The three-dimensional rendering is combined with the 3D model to finally present the video, so that the generated video content is highly deterministic and unique, reducing the uncertainty in the video generation process and ensuring the quality of the final video and the expected effect.

[0104] Based on the same inventive concept as the above method, an embodiment of the present application also provides a system for generating a video based on text information with the aid of three-dimensional rendering, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of any one of the methods for generating a video based on text information with the aid of three-dimensional rendering are implemented.

[0105] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the present application. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application without departing from the content of the technical solution of the present application, shall fall within the protection scope of the technical solution of the present application.

Claims

1. A method for generating a video based on text information by means of three-dimensional rendering, characterized in that: The method comprises the following steps: Obtain user text and use a large language model script generator to obtain a formatted script, obtain an unrendered video based on the 3D models and camera view data in the 3D scene described in the formatted script; obtain nouns in the formatted script corresponding to each 3D model based on the formatted script and the 3D models of all images in the unrendered video; According to the matching of the nouns in the format script corresponding to each 3D model and the nouns in the user text, the nouns corresponding to each 3D model are determined; according to the number of pixels in each 3D model, the distance between each 3D model and the camera view data, and the mapping relationship between each noun and each frame of image, all mapping data pairs of each 3D model in each frame of image are determined; Based on the mapping data pairs, analyzing the position distribution of each noun corresponding to each 3D model in each frame of image in the user text, and determining the rendering weight of each 3D model in each frame of image; Based on the rendering weight, the shortest side length of each 3D model in each frame image is determined, all 3D models are reconstructed, and the unrendered video is rendered by means of a rendering engine to finally obtain a raw video.

2. The method for generating a video based on text information by means of three-dimensional rendering according to claim 1, characterized in that: The method for obtaining the unrendered video is: Each 3D model and camera focus position is input into a three-dimensional model image software to obtain an unrendered video, and the 3D model to which each pixel point of each frame image in the unrendered video belongs is marked.

3. The method for generating a video based on text information by means of three-dimensional rendering according to claim 1, characterized in that: The step of obtaining the nouns in the format script corresponding to each 3D model includes: A noun extraction algorithm is used on the format script to obtain all nouns in the format script; Each noun is labeled with the corresponding 3D model in all images in the unrendered video to obtain the noun in the format script corresponding to each 3D model.

4. The method for generating a video based on text information by means of three-dimensional rendering according to claim 1, characterized in that: The step of determining the nouns corresponding to the 3D models includes: A noun extraction algorithm is used to obtain all nouns in the user text. A string fuzzy matching algorithm is used to match the nouns corresponding to each 3D model in the format text with the nouns in the user text, and the 3D models corresponding to the nouns matching the format text and the user text are used as the nouns corresponding to each 3D model.

5. The method for generating a video based on text information by means of three-dimensional rendering according to claim 2, characterized in that: The step of determining all mapping data pairs of each 3D model in each frame of image includes: Calculate the distance between the geometric center of each 3D model in each frame image and the focal position in the camera view data, and record it as the position weight; Calculate the product of the number of pixel points corresponding to the mark of each 3D model in each frame of image and the position weight as the presentation weight of each 3D model in each frame of image; Arrange the presentation weights of each 3D model in all images in the order of the image frame numbers to form a presentation weight sequence, and obtain all maximum value points in the presentation weight sequence of each 3D model; Analyze the position relationship of each noun corresponding to each 3D model in the user text, and determine the relative text position of each noun corresponding to each 3D model; Analyze the difference between the relative text position of each noun and the image frame number corresponding to the maximum point, and determine the mapping discrimination value between each noun and each maximum point corresponding to each 3D model in combination with the presentation weight corresponding to the maximum point; For each noun and the mapping discrimination value of all maximum points, the maximum point and image frame number corresponding to the maximum value of the mapping discrimination value of each noun are selected, and each noun and the corresponding image frame number are combined into a two-dimensional array to obtain all mapping data pairs of each 3D model in each frame image.

6. The method for generating a video based on text information by means of three-dimensional rendering according to claim 5, characterized in that: Determining the relative text position of each noun corresponding to each 3D model includes: For each noun corresponding to each 3D model, determine the character position of the first character of each noun in the user text; The ratio between the character position of each noun and the number of all characters in the user text is used as the relative text position of each noun corresponding to each 3D model.

7. The method for generating a video based on text information by means of three-dimensional rendering according to claim 5, characterized in that: The step of determining the mapping discriminant value between each noun and each maximum point corresponding to each 3D model includes: ,in, is the mapping discriminant value between the rth noun and the kth maximum point corresponding to the mth 3D model, is the relative text position of the rth noun corresponding to the mth 3D model, is the number of all images in the unrendered video, is the image frame number corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the presentation weight corresponding to the kth maximum point in the presentation weight sequence of the mth 3D model, is the ceiling function, The default value is greater than zero.

8. The method for generating a video based on text information by means of three-dimensional rendering according to claim 6, characterized in that: Determining the rendering weight of each 3D model in each frame of image includes: Calculating the average of the relative text positions of the nouns corresponding to all the mapping data pairs of all the 3D models in each frame of image as the video text position of each frame of image; Recording the difference between the relative text position of each noun corresponding to each 3D model in each frame of image and the video text position as a first difference; The sum of the first differences of all nouns corresponding to each 3D model in each frame of image is used as the rendering weight of each 3D model in each frame of image.

9. The method for generating a video based on text information by means of three-dimensional rendering according to claim 1, characterized in that: The determining the shortest side length of each 3D model in each frame image and reconstructing all 3D models includes: Get the maximum value of the rendering weights of all 3D models in each frame image; Recording the ratio of the rendering weight of each 3D model in each frame of image to the maximum value as a first ratio; The sum of the first ratio and a preset edge length threshold is used as the shortest edge length of each 3D model in each frame of the image; Each 3D model and the corresponding shortest side length are input into a triangulation algorithm to obtain a surface reconstructed 3D model; and a rendering engine is used to render the reconstructed 3D model.

10. A system for generating a video based on text information by means of three-dimensional rendering, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a video based on text information by means of three-dimensional rendering are implemented as described in any one of claims 1 to 9.