Video synthesis method, device, equipment and storage medium

Abstract

The present invention relates to the field of video, and discloses a video synthesis method, device, equipment and storage medium. The method comprises: obtaining a text description input by a user, and generating a scene script through a large language model according to the text description; generating multiple picture materials through a picture generation model according to the scene script; importing the picture materials into OpenGL, loading each frame of the picture material as a separate texture, and setting corresponding texture parameters for the texture; adjusting the coordinates of the texture through a shader according to the texture parameters to achieve the scaling and translation of the picture material, and rendering subtitles through OpenGL; controlling the rendering interval of each frame of the picture material through a timer to achieve the cyclic sequential playback of the frame animation, and recording the played frame animation through OpenGL to obtain a video file. In an embodiment of the present invention, users can generate high-quality video content without professional skills, reduce the time of video synthesis, and thus improve the efficiency of video generation.

Description

Technical Field

[0001] The present invention relates to the field of video, and in particular to a video synthesis method, device, equipment and storage medium. Background Art

[0002] With the advancement of multimedia technology and the popularity of social media, users' demand for audio and video content has grown rapidly, but the current video creation methods generally rely on professional tools and personnel, resulting in low efficiency and high costs. Especially for ordinary users, the existing video production software has a high threshold and it is difficult to meet the personalized and rapid creation needs. In addition, video production involves multiple complex links, such as script writing, shooting, editing and post-production, each of which requires professional knowledge and experience. Traditional methods are not only time-consuming, but also limited in flexibility, and it is difficult to meet the rapidly changing market needs.

[0003] To this end, a convenient and efficient solution is urgently needed that allows users to create video works with professional visual effects without complicated operations. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem of low video generation efficiency.

[0005] A first aspect of the present invention provides a video synthesis method, the video synthesis method comprising:

[0006] Obtaining a text description input by a user, and generating a scenario script based on the text description through a large language model;

[0007] According to the scene script, a plurality of picture materials are generated through a picture generation model;

[0008] Importing the image material into OpenGL, loading each frame of the image material as a separate texture, and setting corresponding texture parameters for the texture, wherein the texture parameters are used to control the performance of the texture during the rendering process;

[0009] According to the texture parameters, the coordinates of the texture are adjusted through a shader to achieve scaling and translation of the image material, and subtitle rendering is performed through the OpenGL;

[0010] The rendering interval of each frame of the image material is controlled by a timer to achieve loop-sequential playback of frame animations, and the played frame animations are recorded by OpenGL to obtain a video file.

[0011] Optionally, in a first implementation of the first aspect of the present invention, adjusting the coordinates of the texture through a shader according to the texture parameters to achieve scaling and translation of the image material, and performing subtitle rendering through the OpenGL includes:

[0012] In OpenGL, according to the texture parameters, the coordinates of the texture are adjusted by a scaling matrix to achieve scaling of the image material;

[0013] By using a shader and a displacement matrix, the coordinates are moved as a whole to achieve translation of the image material;

[0014] Subtitle rendering is performed through the OpenGL.

[0015] Optionally, in a second implementation of the first aspect of the present invention, the performing subtitle rendering by using the OpenGL includes:

[0016] Obtain the text content corresponding to the image material, and generate a bitmap with subtitles through Canvas according to the text content;

[0017] Converting the bitmap into a text texture by using OpenGL;

[0018] The text texture is applied to the OpenGL quadrilateral, rendered into a subtitle layer, and the font in the subtitle layer is adjusted through the shader.

[0019] Optionally, in a third implementation manner of the first aspect of the present invention, importing the picture material into OpenGL, loading each frame of the picture material as a separate texture, and setting corresponding texture parameters for the texture includes:

[0020] Import the image material into OpenGL and use the glGenTextures function to generate multiple texture identifiers;

[0021] Bind the corresponding texture identifier to each frame of the picture material in turn, and load the picture data through the glTexImage2D function to load each frame of the picture material as a separate texture;

[0022] Set corresponding texture parameters for the texture.

[0023] Optionally, in a fourth implementation of the first aspect of the present invention, the rendering interval of each frame of the picture material is controlled by a timer to achieve loop sequence playback of frame animations, and the frame animations played are recorded by OpenGL to obtain a video file, including:

[0024] The rendering interval of each frame of the image material is controlled by a timer to achieve a cyclic sequential playback of the frame animation;

[0025] Preview and play the frame animation through the interaction between UI and OpenGL;

[0026] If the previewed frame animation is consistent with the final video, the played frame animation is recorded through OpenGL to obtain a video file.

[0027] Optionally, in a fifth implementation of the first aspect of the present invention, if the previewed frame animation is consistent with the final video, recording the played frame animation through OpenGL to obtain a video file further includes:

[0028] If the previewed frame animation is inconsistent with the final video, the adjusted texture parameters are passed into the shader through the Uniform variable of OpenGL;

[0029] The adjusted texture parameters are read through the shader, and the adjusted texture parameters are applied to the rendering of each frame of the image material.

[0030] Optionally, in a sixth implementation of the first aspect of the present invention, generating a plurality of picture materials through a picture generation model according to the scene script includes:

[0031] By using the large language model, character information of each scene is set;

[0032] Generate a prompt instruction according to the scene script and the role information in the corresponding scene;

[0033] According to the prompt instruction, a model is generated through the picture to generate multiple picture materials.

[0034] A second aspect of the present invention provides a video synthesis device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the video synthesis device executes the above-mentioned video synthesis method.

[0035] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned video synthesis method.

[0036] In an embodiment of the present invention, a text description input by a user is obtained, and a scene script is generated through a large language model according to the text description; a plurality of picture materials are generated through a picture generation model according to the scene script; the picture materials are imported into OpenGL, each frame of the picture materials is loaded as a separate texture, and corresponding texture parameters are set for the texture, and the texture parameters are used to control the performance of the texture during the rendering process; according to the texture parameters, the coordinates of the texture are adjusted through a shader to achieve the scaling and translation of the picture materials, and subtitle rendering is performed through the OpenGL; the rendering interval of each frame of the picture materials is controlled through a timer to achieve the cyclic sequential playback of frame animations, and the frame animations played are recorded through the OpenGL to obtain a video file. In the present invention, a plurality of picture materials are generated through a large language model according to the text description input by the user, so that the user only needs to input a text description to automatically generate a scene image that meets the requirements to meet the personalized rapid creation requirements, and the picture materials are imported into OpenGL to achieve the scaling and translation of the pictures and the subtitle rendering, so that the user can generate high-quality video content without professional skills, reduce the time of video synthesis, and thus improve the efficiency of video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of an embodiment of a video synthesis method in an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of an embodiment of a video synthesis device in an embodiment of the present invention;

[0039] Figure 3 The figure is a schematic diagram of an embodiment of a video synthesis device in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention provide a video synthesis method, apparatus, device and storage medium.

[0041] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0042] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0043] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the video synthesis method in the embodiment of the present invention includes:

[0044] S100, obtaining a text description input by a user, and generating a scenario script based on the text description through a large language model.

[0045] In this embodiment, the user inputs a story or text description, provides basic information about the video theme, storyline, and characters, and then uses a large language model such as GPT-4 to parse the user input and generate a specific scene script. Specifically, the user needs to first clarify the basic information such as the theme, plot, and character setting of the video. These elements will be passed as input to the large language model to generate a script framework. The user can have a dialogue with the large language model and gradually input each link of the story. The large language model will generate the corresponding script content based on this information. For example, the user can provide a description of the character's appearance and personality traits, and GPT-4 will generate dialogues and scenes that meet the character settings based on this information. After generating the preliminary scene script, the user can modify and polish the scene script as needed to ensure its fluency and logic.

[0046] S200, generating a model through pictures according to the scene script to generate multiple picture materials.

[0047] In this embodiment, through the picture generation model, the system generates pictures or animated image materials that conform to the style of the video based on the scene script content, wherein the picture generation model can be a Stable Diffusion model. Specifically, first, the user needs to select a suitable Stable Diffusion model. Different models determine the style of the generated image, such as cartoon style, realistic painting style, or modern style. Then, according to the content of the scene script, enter descriptive prompts (Prompt). These prompts include information such as subject, style, color, quality requirements, etc. The prompt is the key to generating pictures. It tells AI what kind of picture to generate. Users can adjust the generation parameters to control the quality and style of the picture. For example, you can set parameters such as sampling steps, image size, and lighting effects to ensure that the generated picture meets expectations. Finally, the picture generation model will generate corresponding pictures according to the prompts.

[0048] S300, importing the image material into OpenGL, loading each frame of the image material as a separate texture, and setting corresponding texture parameters for the texture.

[0049] In this embodiment, each generated picture material is used as a frame of image in the video, and the picture material is imported into OpenGL, and each frame of the picture material is loaded as a separate texture. The texture can add details to the model and make the model look more realistic. For example, applying textures of different materials to a 3D model can significantly improve the visual effect of the model. Among them, OpenGL supports a variety of picture formats, including RGB, RGBA, grayscale images, etc., and the picture data can be stored in different formats. By loading each frame of the picture material as a separate texture, the displayed picture can be dynamically changed during the rendering process, thereby achieving animation effects, and different texture resources can be easily managed and switched, thereby improving the flexibility and scalability of the program. In order to ensure the effects of texture scaling, smoothing, etc., the texture parameters are set. The texture parameters determine the performance and details of the texture during the rendering process, such as GL_TEXTURE_MIN_FILTER and GL_TEXTURE_MAG_FILTER, which determine how to sample from the original texture image when the texture is enlarged or reduced to different sizes. Use linear filtering (GL_LINEAR) to improve the scaling effect.

[0050] S400 adjusts the texture coordinates through a shader according to the texture parameters to achieve scaling and translation of the image material, and renders subtitles through OpenGL.

[0051] In this embodiment, first, write a shader program. Shader includes vertex shader and fragment shader. Shader files usually contain image transformation and color processing logic. Vertex shader and fragment shader are two core parts in the graphics rendering pipeline, which are responsible for processing the geometric data and pixel data of the graphics respectively. Create a program object through the glCreateProgram function, and attach the vertex shader and fragment shader to the object for subsequent rendering. Then, adjust the coordinates of the texture through the shader to achieve the scaling and translation of the image material. At the same time, convert the text content corresponding to the image into a texture through OpenGL for subtitle rendering.

[0052] S500 controls the rendering interval of each frame of picture material through a timer to achieve cyclic sequential playback of frame animations, and records the played frame animations through OpenGL to obtain video files.

[0053] In this embodiment, a timer or frame rate control mechanism is used to control the rendering interval of each frame of the picture material to achieve the cyclic sequential playback of the frame animation. For example, the drawing function can be called once every certain time (such as 1000 / 60 milliseconds) by setting the timer to achieve the effect of 60 frames per second. In the rendering loop, glClear is first called to clear the color buffer to ensure that each frame is drawn on a clean canvas. Then the picture material list is traversed and each frame is drawn in turn. In the rendering loop, various input events need to be processed to respond to user operations. The frame animation is played and previewed. When the user is satisfied with the preview effect, the system will use the video recording function of OpenGL to capture the entire preview process and export it as a video file. The video encoding format, resolution and other parameters can be set according to user needs to output a video product that meets the requirements. After recording, the video file will be saved to the location specified by the user for subsequent use and sharing.

[0054] In an optional implementation of the first aspect of the present invention, adjusting the coordinates of the texture through a shader according to the texture parameters to achieve scaling and translation of the image material, and performing subtitle rendering through OpenGL includes:

[0055] In OpenGL, according to the texture parameters, the coordinates of the texture are adjusted through the scaling matrix to achieve the scaling of the image material; the coordinates are moved as a whole through the shader and the displacement matrix to achieve the translation of the image material; subtitles are rendered through OpenGL.

[0056] In this embodiment, in OpenGL, the scaling of image materials is achieved through transformation matrices, and the model view projection (MVP) matrix can be used for operation. Specifically, in the vertex shader, the texture coordinates are adjusted through the scaling matrix. For example, if the object needs to be scaled to tx, ty, and tz times along the X, Y, and Z axes respectively, the following transformation can be performed in the vertex shader:

[0057] void main() {

[0058] / / Scale the vertex coordinates

[0059] vec4 pos = vec4(position.xyz * vec3(tx, ty, tz), 1.0);

[0060] gl_Position = projectionMatrix *modelViewMatrix* pos;

[0061] }

[0062] In this process, position is the input vertex coordinate, projectionMatrix and modelViewMatrix are transformation matrices, and tx, ty, and tz are the scaling ratios along the X, Y, and Z axes, respectively.

[0063] The image material is translated through the displacement matrix, and the texture coordinates are moved as a whole using the translation vector vec3(tx, ty, 0.0) in the vertex shader. Specifically, in the vertex shader, you first need to declare a uniform variable to receive the translation vector. Then, add the translation vector to the vertex position to achieve the translation effect.

[0064] In an optional implementation of the first aspect of the present invention, performing subtitle rendering through OpenGL includes:

[0065] Get the text content corresponding to the image material, and generate a bitmap with subtitles through Canvas based on the text content; convert the bitmap into a text texture through OpenGL; apply the text texture to the OpenGL quadrilateral, render it as a subtitle layer, and adjust the font in the subtitle layer through the shader.

[0066] In this embodiment, the text content corresponding to the image material is converted into a texture. Specifically, a Canvas is used to generate a bitmap image with subtitles, which can be achieved by setting the background color, font type, font size, font color and other parameters of the Canvas. Then the generated Bitmap image is uploaded to OpenGL to generate a text texture. This step requires the use of the texture loading function provided by OpenGL, such as glTexImage2D, to load the Bitmap texture into OpenGL. Use OpenGL to draw a quadrilateral, and apply the previously generated text texture to the quadrilateral to render it as a subtitle layer, specifically including: first define the vertex coordinates and texture coordinates of the quadrilateral, then use the vertex array object (VAO) and vertex buffer object (VBO) to store the vertex and texture coordinate data, and then use the shader program of OpenGL to process texture mapping and color mixing. Use the rendering pipeline of OpenGL to render the quadrilateral to the screen, and adjust the color, size and other attributes of the font through the shader program. For example, the font color and transparency can be controlled using the fragment Shader.

[0067] In an optional implementation of the first aspect of the present invention, importing the picture material into OpenGL, loading each frame of the picture material as a separate texture, and setting corresponding texture parameters for the texture include:

[0068] Import the image material into OpenGL and use the glGenTextures function to generate multiple texture identifiers; bind the corresponding texture identifier to each frame of the image material in turn, and load the image data through the glTexImage2D function to load each frame of the image material as a separate texture; set the corresponding texture parameters for the texture.

[0069] In this embodiment, the glGenTextures function is used to generate the ID of the texture object. For example, glGenTextures(1,&texture) generates a texture ID and stores it in the texture variable, and the glBindTexture function is used to bind the generated texture ID to the specified target. After binding the texture, subsequent texture operations (such as setting the filter mode, loading image data, etc.) will act on the currently bound texture. Then, the image data is loaded into the texture through the glTexImage2D function. If there are multiple frames of images, glGenTextures can be used to generate multiple texture IDs. For example, glGenTextures(n,textures) will generate n texture IDs and store them in the textures array. For each frame of the image, the corresponding texture ID is bound in turn, and the image data is loaded using glTexImage2D. For example, for the first frame of the image, glBindTexture(GL_TEXTURE_2D,textures[0]) can be called first, and then the first frame of the image data can be loaded using glTexImage2D; then this process is repeated for the second frame of the image until all frames are loaded.

[0070] In an optional implementation of the first aspect of the present invention, a timer is used to control the rendering interval of each frame of picture material to achieve cyclic sequential playback of frame animations, and OpenGL is used to record the played frame animations to obtain a video file including:

[0071] The rendering interval of each frame of image material is controlled through a timer to achieve loop-sequential playback of frame animations; the frame animations are previewed and played through the interaction between the UI and OpenGL; if the previewed frame animation is consistent with the final video, the played frame animation is recorded through OpenGL to obtain a video file.

[0072] In this embodiment, the timer function in OpenGL is used to control the rendering interval of each frame, which can ensure that the animation runs at a consistent speed on different hardware. A user operation area is set in the main interface, and the playback and pause of the animation are controlled by UI buttons. The user can preview the effect of the frame animation through these buttons to ensure that it meets expectations. If the previewed frame animation is consistent with the final video, the played frame animation can be recorded through OpenGL to generate a video file, which usually renders each frame into a buffer and encodes the frames into a video format using an appropriate encoder.

[0073] In an optional implementation of the first aspect of the present invention, if the previewed frame animation is consistent with the final video, the played frame animation is recorded through OpenGL to obtain a video file, further comprising:

[0074] If the previewed frame animation is inconsistent with the final video, the adjusted texture parameters are passed to the shader through the Uniform variable of OpenGL; the adjusted texture parameters are read through the shader and applied to the rendering of each frame of the image material.

[0075] In this embodiment, if the previewed frame animation is inconsistent with the final video, the texture parameters can be adjusted in real time through the interaction between the UI and OpenGL, and the adjusted texture parameters are passed to the shader through the Uniform variables of OpenGL, such as scaling u_Scale, rotation u_Rotate, displacement u_Translate, etc. Through the shader, the adjusted texture parameters are read, and the adjusted texture parameters are applied to the rendering of each frame of the image material, thereby ensuring the consistency between the previewed frame animation and the final video. Specifically, the required Uniform variables are defined in the shader code. In the application, the location of the Uniform variable is obtained through the glGetUniformLocation function, and before each frame is rendered, the value of the Uniform variable is updated through the OpenGLAPI. In the shader, the rendering effect of the texture is adjusted by reading the Uniform variable. For example, the variable can be used in the shader to calculate the final texture coordinates and perform corresponding transformations.

[0076] In an optional implementation of the first aspect of the present invention, generating a plurality of picture materials by using a picture generation model according to a scene script includes:

[0077] Through a large language model, the role information of each scene is set; according to the scene script and the role information in the corresponding scene, prompt instructions are generated; according to the prompt instructions, a model is generated through the picture to generate multiple picture materials.

[0078] In this embodiment, a specific scene script is generated through a large language model such as GPT-4, and clear identity, appearance, behavior and other detailed information are set for the characters in each scene. For example, if the story description contains "young explorer", the system will automatically generate the character's age, clothing style, expression, posture and other information. The large language model generates a detailed description of the character's appearance based on the character settings in the scene, including gender, hairstyle, clothing, accessories, expression, etc. These descriptions will be used as instructions for subsequent generation of character images to ensure that the image generation meets the plot requirements. The image generation model receives the prompt instruction in the script and generates materials of different styles such as comic style, illustration style, modern style, etc. as required to match the overall visual effect.

[0079] See also Figure 2 According to a second aspect of the present invention, a video synthesis device is provided, the video synthesis device comprising:

[0080] The scenario script generation module 10 is used to obtain the text description input by the user and generate the scenario script according to the text description through a large language model;

[0081] Picture material generation module 20, used to generate a plurality of picture materials through a picture generation model according to a scene script;

[0082] The texture loading module 30 is used to import the image material into OpenGL, load each frame of the image material as a separate texture, and set corresponding texture parameters for the texture, and the texture parameters are used to control the performance of the texture during the rendering process;

[0083] The texture subtitle setting module 40 is used to adjust the coordinates of the texture through the shader according to the texture parameters to achieve the scaling and translation of the image material, and to render the subtitles through OpenGL;

[0084] The frame animation playing module 50 is used to control the rendering interval of each frame of image material through a timer to achieve cyclic sequential playing of the frame animation, and to record the played frame animation through OpenGL to obtain a video file.

[0085] In an optional implementation of the second aspect of the present invention, the texture subtitle setting module 40 is also used in OpenGL to adjust the coordinates of the texture through a scaling matrix according to texture parameters to achieve scaling of the image material; to move the coordinates as a whole through a shader and a displacement matrix to achieve translation of the image material; and to render subtitles through OpenGL.

[0086] In an optional implementation of the second aspect of the present invention, the texture subtitle setting module 40 is also used to obtain the text content corresponding to the image material, and generate a bitmap with subtitles through Canvas based on the text content; convert the bitmap into a text texture through OpenGL; apply the text texture to the OpenGL quadrilateral, render it as a subtitle layer, and adjust the font in the subtitle layer through a shader.

[0087] In an optional implementation of the second aspect of the present invention, the texture loading module 30 is used to import image materials into OpenGL, and use the glGenTextures function to generate multiple texture identifiers; bind corresponding texture identifiers to each frame of image material in turn, and load image data through the glTexImage2D function to load each frame of image material as a separate texture; set corresponding texture parameters for the texture.

[0088] In an optional implementation of the second aspect of the present invention, the frame animation playback module 50 is also used to control the rendering interval of each frame of image material through a timer to achieve loop sequence playback of the frame animation; preview and play the frame animation through the interaction between UI and OpenGL; if the previewed frame animation is consistent with the final video, the played frame animation is recorded through OpenGL to obtain a video file.

[0089] In an optional implementation of the second aspect of the present invention, the video synthesis device also includes a parameter adjustment module, which is used to pass the adjusted texture parameters to the shader through the Uniform variable of OpenGL if the previewed frame animation is inconsistent with the final video; read the adjusted texture parameters through the shader, and apply the adjusted texture parameters to the rendering of each frame of the image material.

[0090] In an optional implementation of the second aspect of the present invention, the picture material generation module 20 is also used to set the role information of each scene through a large language model; generate prompt instructions according to the scene script and the role information in the corresponding scene; and generate multiple picture materials through the picture generation model according to the prompt instructions.

[0091] Figure 31 is a schematic diagram of the structure of a video synthesis device provided by an embodiment of the present invention. The video synthesis device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the video synthesis device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the video synthesis device 500.

[0092] The video synthesis device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the video synthesis device shown does not constitute a limitation on the video synthesis device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0093] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the video synthesis method.

[0094] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0096] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video synthesis method, characterized in that: The video synthesis method comprises: Obtaining a text description input by a user, and generating a scenario script based on the text description through a large language model; According to the scene script, a plurality of picture materials are generated through a picture generation model; Importing the image material into OpenGL, loading each frame of the image material as a separate texture, and setting corresponding texture parameters for the texture, wherein the texture parameters are used to control the performance of the texture during the rendering process; According to the texture parameters, the coordinates of the texture are adjusted through a shader to achieve scaling and translation of the image material, and subtitle rendering is performed through the OpenGL; The rendering interval of each frame of the picture material is controlled by a timer to realize the cyclic sequential playback of frame animations, and the played frame animations are recorded by OpenGL to obtain a video file; The subtitle rendering by the OpenGL comprises: Obtain the text content corresponding to the image material, and generate a bitmap with subtitles through Canvas according to the text content; Converting the bitmap into a text texture by using OpenGL; Applying the text texture to the OpenGL quadrilateral, rendering it into a subtitle layer, and adjusting the font in the subtitle layer through the shader; The generating a plurality of picture materials by generating a model through a picture according to the scene script comprises: By using the large language model, character information of each scene is set; Generate a prompt instruction according to the scene script and the role information in the corresponding scene; According to the prompt instruction, a model is generated through the picture to generate multiple picture materials.

2. The video synthesis method according to claim 1, characterized in that: The step of adjusting the coordinates of the texture through a shader according to the texture parameters to achieve scaling and translation of the image material, and performing subtitle rendering through OpenGL comprises: In OpenGL, according to the texture parameters, the coordinates of the texture are adjusted by a scaling matrix to achieve scaling of the image material; By using a shader and a displacement matrix, the coordinates are moved as a whole to achieve translation of the image material; Subtitle rendering is performed through the OpenGL.

3. The video synthesis method according to claim 1, characterized in that: The step of importing the image material into OpenGL, loading each frame of the image material as a separate texture, and setting corresponding texture parameters for the texture comprises: Import the image material into OpenGL and use the glGenTextures function to generate multiple texture identifiers; Bind the corresponding texture identifier to each frame of the picture material in turn, and load the picture data through the glTexImage2D function to load each frame of the picture material as a separate texture; Set corresponding texture parameters for the texture.

4. The video synthesis method according to claim 1, characterized in that: The rendering interval of each frame of the picture material is controlled by the timer to realize the cyclic sequential playback of the frame animation, and the frame animation played is recorded by OpenGL to obtain a video file including: The rendering interval of each frame of the image material is controlled by a timer to achieve a cyclic sequential playback of the frame animation; Preview and play the frame animation through the interaction between UI and OpenGL; If the previewed frame animation is consistent with the final video, the played frame animation is recorded through OpenGL to obtain a video file.

5. The video synthesis method according to claim 4, characterized in that: If the previewed frame animation is consistent with the final video, the frame animation played is recorded through OpenGL to obtain a video file, and further includes: If the previewed frame animation is inconsistent with the final video, the adjusted texture parameters are passed into the shader through the Uniform variable of OpenGL; The adjusted texture parameters are read through the shader, and the adjusted texture parameters are applied to the rendering of each frame of the image material.

6. A video synthesis device, characterized in that: The video synthesis device comprises: A scenario script generation module is used to obtain a text description input by a user and generate a scenario script based on the text description through a large language model; A picture material generation module, used to generate a plurality of picture materials through a picture generation model according to the scene script; A texture loading module, used for importing the picture material into OpenGL, loading each frame of the picture material as a separate texture, and setting corresponding texture parameters for the texture, wherein the texture parameters are used for controlling the performance of the texture during the rendering process; A texture subtitle setting module, used to adjust the coordinates of the texture through a shader according to the texture parameters to achieve scaling and translation of the image material, and perform subtitle rendering through the OpenGL; The frame animation playback module is used to control the rendering interval of each frame of the picture material through a timer to achieve cyclic sequential playback of the frame animation, and to record the played frame animation through the OpenGL to obtain a video file; The texture subtitle setting module is also used to obtain the text content corresponding to the image material, and generate a bitmap with subtitles through Canvas according to the text content; convert the bitmap into a text texture through OpenGL; apply the text texture to the quadrilateral of OpenGL, render it as a subtitle layer, and adjust the font in the subtitle layer through the shader; The picture material generation module is also used to set the role information of each scene through a large language model; generate prompt instructions according to the scene script and the role information in the corresponding scene; and generate multiple picture materials through the picture generation model according to the prompt instruction.

7. A video synthesis device, characterized in that: The video synthesis device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the video synthesis device to execute the video synthesis method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video synthesis method according to any one of claims 1 to 5 is implemented.

Images