Video generation method and video dynamic special effect reproduction method

Abstract

The embodiment of the invention provides a video generation method and device, computer equipment, a computer readable storage medium and a computer program product, and belongs to the field of video generation. The video generation method comprises the following steps: acquiring model input information, wherein the model input information comprises a target description text; the model input information is input into a pre-trained large language model, so that a structured data object is output through the large language model, the structured data object comprises rendering function codes and a parameter set, and the parameter set comprises a plurality of adjustable parameters; in response to adjustment for at least one adjustable parameter in the plurality of adjustable parameters, updating the structured data object; and generating a target video according to the updated structured data object. According to the technical scheme of the embodiment of the invention, the updating efficiency, flexibility and individuation degree of video generation can be improved.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a video generation method, a method for reproducing dynamic video effects, an apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of generative artificial intelligence technology, automated video generation has become a hot topic in the field of content creation. Video generation solutions employing an end-to-end "black box" model can directly output a fixed stream of video pixels based on text instructions.

[0003] However, in practical applications, this model makes it difficult for creators to fine-tune the dynamic effects, motion trajectories, or element attributes in the video. When the output does not meet expectations, creators often have to resort to trial-and-error generation by repeatedly modifying the prompts, leading to extended production cycles and insufficient control precision.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] This application provides a video generation method, a video dynamic effects reproduction method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0006] One aspect of this application provides a video generation method, the method comprising:

[0007] Obtain model input information, which includes target description text; The model input information is input into a pre-trained large language model to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters. In response to an adjustment to at least one of the adjustable parameters, the structured data object is updated; Generate the target video based on the updated structured data object.

[0008] Optionally, obtain model input information, including: Receive initial description text; Based on multiple preset semantic dimensions, determine whether the initial description text meets the preset evaluation criteria; If the initial description text does not meet the preset evaluation criteria, no more than a predetermined number of target questions are output based on the initial description text, so as to receive the target results input for the target questions; Upon receiving the target result, the target description text is generated based on the target result and the initial description text.

[0009] Optionally, the method further includes: The target renderer executes the rendering function code based on the initial values ​​of each parameter in the parameter set to display the preview video; During the preview process, if an adjustment to the adjustable parameters is detected, the target renderer will execute the rendering function code again based on the values ​​of each parameter in the adjusted parameter set to display the updated preview video.

[0010] Optionally, the rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: If an error occurs during the execution of the rendering function code, obtain the code execution error information; The error message and the structured data object are input into the large language model to output the repaired structured data object; Based on the initial values ​​of each parameter in the parameter set, the rendering function code in the repaired structured data object is executed.

[0011] Optionally, the structured data object further includes a rendering mode field, which indicates the rendering mode, including a 3D rendering mode or a 2D rendering mode; the fields corresponding to the one or more adjustable parameters include fields that can set 3D model parameters; The target renderer is selected from multiple renderers, and the operation of selecting the target renderer includes: If 3D model parameters are set or the 3D rendering mode is in effect, select the 3D renderer as the target renderer. In the case where no 3D model parameters are set and the 2D rendering mode is in effect: Determine whether the structured data object includes post-processing effects code; In the case where the structured data object includes the post-processing effects code, the post-processing blend renderer is used as the target renderer, and the post-processing blend renderer includes a 2D renderer and a post-processing shader. If the structured data object does not include the post-processing effects code, then the 2D renderer is selected as the target renderer.

[0012] Optionally, the rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: With the target renderer selected, the rendering function code and the resources corresponding to each parameter are loaded through a unified interface; The rendering function code is executed after loading the rendering function code and the resources corresponding to each parameter.

[0013] Optionally, the model input information may also include multimedia materials, which may include multiple video frames; The rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: Based on the frame positions corresponding to the multiple video frames, a target random number sequence is determined. The target random number sequence includes multiple random numbers, with each random number corresponding to a video frame. The random number is used to indicate the dynamic effects in the corresponding video frame. The rendering function code is executed based on the target random number sequence and the initial values ​​of each parameter.

[0014] Optionally, the plurality of video frames includes target video frames associated with a target random number; Based on the updated structured data object, generate the target video, including: The rendering function code is executed on the target video frame according to the target random number to obtain the target rendering result corresponding to the target video frame; Obtain target pixel data based on the target rendering result; The target encoding result of the target video frame is obtained by encoding the target pixel data through the target encoder.

[0015] Optionally, the method further includes: Extract the target channel value from the target pixel data according to the target channel; The grayscale encoding result of the target video frame is obtained by the target encoder based on the target channel value.

[0016] Another aspect of this application provides a method for reproducing dynamic video effects, the method comprising: Extract multiple key video frames from the reference video; The multiple key video frames and their corresponding timestamps are input into a large language model to extract features from multiple preset dimensions to obtain a first structured data object. The first structured data object and the multiple key video frames are input into the large language model to obtain several target features; Based on the aforementioned target features, a second structured data object is obtained; The multiple key video frames and the second structured data object are input into the large language model to generate a target structured data object, which includes rendering function code for reproducing dynamic effects in the reference video.

[0017] Optionally, based on the aforementioned target features, a second structured data object is obtained, including: Determine whether the aforementioned target features meet preset standards; If one or more of the target features do not conform to the preset standard, a quantitative description is added to the one or more target features that do not conform to the preset standard to obtain the second structured data object.

[0018] Optionally, the plurality of key video frames and the second structured data object are input into the large language model to generate the target structured data object, including: Obtain supplementary correction text for the second structured data object; The second structured data object and supplementary correction text of the multiple key video frames are input into the large language model to generate the target structured data object.

[0019] Optionally, the method further includes: Multiple rendered video frames are obtained from the target video, and each rendered video frame corresponds to one key video frame; A comparison video frame sequence is generated based on the plurality of rendered video frames and the key video frames corresponding to each of the plurality of rendered video frames; The comparison video frame sequence is input into the large language model for comparison to obtain comparison results, which include a quantitative description of the differences between each rendered video frame and its corresponding key video frame.

[0020] Optionally, the comparison results may also include repair code, and the method may further include: The comparison results and the multiple key video frames are input into the large language model to modify the target part of the target structured data object through the large language model, so as to obtain the modified target structured data object. An updated video is generated based on the modified target structured data object; Based on the current round of video updates and each subsequent round of video updates, multiple rounds of modification operations are performed on the target structured data object until the final round of video updates; In each round of operations: The video frame sequence from the updated video of this round of operation is input into the large language model for comparison to obtain the comparison result of this round; If the comparison result in this round is better than the best comparison result in the previous rounds, then the comparison result in this round and multiple key video frames are input into the large language model, so that the model modifies the target part in the specified structured data object according to the repair code in the comparison result in this round, and outputs the updated target structured data object in this round, wherein the specified structured data object is the target structured data object output in the previous round.

[0021] Another aspect of this application provides a video generation apparatus, the apparatus comprising: The acquisition module is used to acquire model input information, which includes target description text; The input module is used to input model input information into a pre-trained large language model so as to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters. An update module is configured to update the structured data object in response to an adjustment of at least one of the plurality of adjustable parameters; The generation module is used to generate the target video based on the updated structured data object.

[0022] Another aspect of this application provides a video dynamic special effects reproduction device, the device comprising: The extraction module is used to extract multiple key video frames from the reference video. The first input module is used to input the plurality of key video frames and the timestamps corresponding to each of the plurality of key video frames into the large language model, so as to extract features from multiple preset dimensions through the large language model to obtain a first structured data object. The second input module is used to input the first structured data object and the multiple key video frames into the large language model to obtain several target features; The acquisition module is used to acquire a second structured data object based on the aforementioned target features; A generation module is used to input the plurality of key video frames and the second structured data object into the large language model to generate a target structured data object, wherein the target structured data object includes rendering function code for reproducing dynamic effects in the reference video.

[0023] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0024] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0025] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0026] The embodiments of this application employing the above technical solution may include the following advantages: A structured data object containing rendering function code and adjustable parameters is generated based on the model input information using a large language model, and the structured data object is updated according to the adjustment of the adjustable parameters to generate the target video. Transforming the target description text of natural language into an executable and parameterized structured data object allows creators to conveniently and quickly generate target videos with specific effects using natural language, lowering the barrier to entry for creating special effects videos. Simultaneously, the adjustable parameter settings allow creators to adjust the effects in the video in real-time according to their needs without having to re-invoke the large language model to generate new structured data objects, improving the update efficiency, flexibility, and personalization of video generation. Attached Figure Description

[0027] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0028] Figure 1 The diagram illustrates the operating environment of the video generation method according to Embodiment 1 of this application. Figure 2 A flowchart illustrating a video generation method according to Embodiment 1 of this application is shown schematically; Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step S200; Figure 4 The diagram illustrates a new addition to the video generation method according to Embodiment 1 of this application; Figure 5 Schematic illustration Figure 4Flowchart of the sub-steps in step S400; Figure 6 This schematically illustrates another additional flowchart of the video generation method according to Embodiment 1 of this application; Figure 7 Schematic illustration Figure 4 Flowchart of another sub-step in step S400; Figure 8 Schematic illustration Figure 4 Flowchart of another sub-step in step S400; Figure 9 Schematic illustration Figure 2 Flowchart of the sub-steps in step S206; Figure 10 This schematically illustrates yet another additional flowchart of the video generation method according to Embodiment 1 of this application; Figure 11 The flowchart illustrating the video dynamic effects reproduction method according to Embodiment 2 of this application is shown in the schematic diagram. Figure 12 Schematic illustration Figure 11 Flowchart of the sub-step in step S1116; Figure 13 Schematic illustration Figure 11 Flowchart of the sub-steps in step S1108; Figure 14 The diagram illustrates the addition of a new flowchart to the video dynamic effects reproduction method according to Embodiment 2 of this application; Figure 15 The illustration shows exemplary application flowcharts of the video generation method according to Embodiment 1 of this application and the video dynamic special effects reproduction method according to Embodiment 2 of this application; Figure 16 The diagram illustrates the system architecture of a video generation method according to Embodiment 1 of this application. Figure 17 The flowchart illustrating the video generation and export process of the video generation method according to Embodiment 1 of this application is shown in the illustration. Figure 18 The flowchart illustrating the video export method of the video generation method according to Embodiment 1 of this application is shown in the illustration. Figure 19 The flowchart illustrating the method for reproducing video dynamic effects according to Embodiment 2 of this application is shown in the illustration. Figure 20 A block diagram of a video generation apparatus according to Embodiment 3 of this application is shown schematically; Figure 21 A block diagram of a video motion effects reproduction device according to Embodiment 4 of this application is schematically shown; and Figure 22A schematic diagram of the hardware architecture of a computer device according to Embodiment 5 of this application is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0031] It should be noted that, in any stage of this application involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0032] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0033] First, a definition of the terminology used in this application is provided: Large Language Model: Includes LLM (Language Model) and VLM (Visual Language Model).

[0034] Structured data objects: Collections of data organized according to a predefined format (such as JSON, XML).

[0035] Abstract Syntax Tree (AST): A tree-like representation of the syntax structure of source code. By stripping away redundant syntax symbols such as parentheses and semicolons, it abstracts the code logic into a hierarchical structure composed of nodes (such as variables, declarations, and expressions).

[0036] PBR (Physically Based Rendering): A rendering method that follows the laws of physics to simulate the realistic physical interaction between light and the surface of an object.

[0037] Information entropy: A mathematical measure used to quantify the uncertainty or randomness of a system.

[0038] Alpha channel: A color channel used to define pixel transparency information.

[0039] H.264 encoding standard: A video encoding and decoding standard with a high compression ratio.

[0040] Blob URL (Object URL): A temporary URL string dynamically generated by the browser in memory, pointing to a Blob or File object in memory.

[0041] Off-screen rendering: The rendering result is output directly to the memory buffer (such as an off-screen canvas) without being displayed on the screen.

[0042] Seed pseudorandom number generator: An algorithm that generates deterministic random sequences using a specific initial value (seed).

[0043] State Management Library: A tool for centrally managing and maintaining shared data among components in an application.

[0044] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: With the growth of internet multimedia content, the demand for dynamic special effects materials in video creation, live streaming, and web design is becoming increasingly prominent. Dynamic special effects production mainly relies on professional design tools, which, while powerful, suffer from significant drawbacks such as a high learning curve, long production cycles, and low iteration efficiency. In recent years, the rise of Large Language Models (LLM) has provided a new path for automated dynamic special effects generation.

[0045] However, AI-assisted dynamic effects design still faces the following significant problems in practical applications: the tool outputs an uneditable "black box" video, making it difficult for users to fine-tune motion trajectories and parameter details; there are often visual differences between the preview effect and the final exported result, making it impossible to achieve a WYSIWYG (What You See Is What You Get) result; the lack of a unified lifecycle and parameter management method for 2D, 3D, and post-processing rendering engines leads to complex system implementation logic; and when reversing the video to restore it to code, the visual restoration effect is unstable, making it difficult to achieve high-fidelity reproduction.

[0046] Therefore, this application provides a video generation technology solution. In this technology solution, (1) the target description text of natural language is transformed into an executable and parameterized structured data object, which enables creators to conveniently and quickly generate target videos with specific effects through natural language, reducing the production threshold of special effects videos; at the same time, the adjustable parameter settings allow creators to adjust the effects of special effects in the video in real time according to their own needs, without having to call the large language model again to generate new structured data objects, which improves the update efficiency, flexibility and personalization of video generation; (2) the target random number consistent with the preview process is used to render the target video frame, ensuring that the dynamic effects implemented in the target video are consistent with the target random number in the preview process. The dynamic effects implemented in the preview video have visual consistency and are WYSIWYG; (3) The unified interface enables the calling and management of various target rendering engines of different types, shielding the implementation differences of different underlying rendering technologies and improving the compatibility of video generation; (4) The key video frames are deconstructed from multiple preset dimensions through a large language model, and by adding quantitative descriptions that conform to preset standards, the abstract visual perception is transformed into a precise structured data object that can be used for code generation, which improves the accuracy of the restoration of the visual features of dynamic effects in the reference video, reduces the difficulty and work cycle of reproducing dynamic effects, and improves the efficiency and flexibility of reproducing dynamic effects in the video. See the following text for details.

[0047] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0048] like Figure 1 As shown in the diagram, the operating environment includes: server 2 and clients (4A, 4B, ..., 4N).

[0049] Server 2 can connect to clients (4A, 4B, ..., 4N) via the network.

[0050] Server 2 can be a single server, a server cluster, or a cloud computing service center.

[0051] Server 2 can provide clients with video generation services, video dynamic effects reproduction services, and more.

[0052] Server 2 can be located in a data center, such as a single location, or distributed across different geographical locations (e.g., multiple locations). Server 2 can provide services via a network. The network includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. The network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0053] Clients (4A, 4B, ..., 4N) can be configured to access the content and services of server 2. Clients (4A, 4B, ..., 4N) can include electronic devices with built-in or external display panels, such as mobile devices, tablets, laptops, workstations, virtual reality devices, gaming devices, digital streaming media devices, vehicle terminals, smart TVs, set-top boxes, etc., and can also include virtualized computing instances. Virtualized computing instances can include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device can load the virtual machine based on the virtual image and / or other data defining specific software (e.g., operating system, dedicated applications, servers) used for simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices.

[0054] A client (4A, 4B, ..., 4N) can be associated with one or more users. A single user can also use one or more of the clients (4A, 4B, ..., 4N) to access server 2. Clients (4A, 4B, ..., 4N) can travel to various locations and use different networks to access server 2.

[0055] The client (4A, 4B, ..., 4N) may include an interface. This interface may include a touchpad, touchscreen, mouse, keyboard, or other sensing elements. For example, the input element may be configured to receive user commands that enable the client (4A, 4B, ..., 4N) to perform various operations, such as entering initial descriptive text or uploading multimedia materials.

[0056] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0057] The technical solutions of this application are described below through multiple embodiments, using server 2 or client as the execution entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0058] Example 1 Figure 2A flowchart illustrating a video generation method according to Embodiment 1 of this application is shown schematically.

[0059] like Figure 2 As shown, the video generation method may include steps S200~S206, wherein: Step S200: Obtain model input information, which includes target description text.

[0060] Step S202: Input the model input information into the pre-trained large language model to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters.

[0061] Step S204: In response to an adjustment of at least one of the plurality of adjustable parameters, update the structured data object.

[0062] Step S206: Generate the target video based on the updated structured data object.

[0063] The video generation method provided in this embodiment generates a structured data object containing rendering function code and adjustable parameters based on the model input information using a large language model. The structured data object is then updated according to adjustments to the adjustable parameters to generate the target video. By transforming the target description text in natural language into an executable and parameterized structured data object, creators can easily and quickly generate target videos with specific effects using natural language, lowering the barrier to entry for creating special effects videos. Simultaneously, the adjustable parameters allow creators to adjust the effects in the video in real-time according to their needs without having to re-invoke the large language model to generate new structured data objects, improving the update efficiency, flexibility, and personalization of video generation.

[0064] The following combination Figure 2 The steps in steps S200 to S206, as well as other optional steps, are described in detail.

[0065] Step S200 The model input information is obtained, including the target description text.

[0066] The target description text can include what dynamic effects the target object (such as a user) wants to achieve in the target video. For example, the target description text could be: "Generate a blue particle halo that continuously rotates and spreads in the center of the screen, with the particles having a slight trailing effect." In some embodiments, the model input information can also include images, videos, 3D models, design documents, or sequence frames. When the model input information includes images, the image material can be adaptively compressed (e.g., ensuring the shorter side does not exceed 1080 pixels) before being input into the model.

[0067] During the process of acquiring model input information, the quality of the acquired model input information can be optimized in various ways. The following provides an example optimization method.

[0068] In optional embodiments, such as Figure 3 As shown, step S200 includes: S300, Receive initial description text.

[0069] S302, determine whether the initial description text meets the preset evaluation criteria based on multiple preset semantic dimensions.

[0070] S304, if the initial description text does not meet the preset evaluation criteria, output no more than a predetermined number of target questions based on the initial description text, so as to receive the target results input for the target questions.

[0071] S306, Upon receiving the target result, generate the target description text based on the target result and the initial description text.

[0072] Multiple preset semantic dimensions may include one or more of the following: animation type, element shape, color scheme, animation speed, looping method, and visual style. In some embodiments, the semantic feature vectors in the initial description text can be extracted, and their information entropy or confidence level on each of the above preset dimensions can be calculated. If the confidence level of any dimension is lower than a preset threshold, it is determined that the preset evaluation criteria are not met. In this case, the generated target question can be a predetermined number (e.g., 5) of the semantic dimensions with the lowest confidence levels. In some embodiments, the generated target question may also have multiple preset options, and the number of preset options may have a minimum limit, such as no less than three. In some embodiments, the target question may also be directly selected from a preset question list, which includes multiple standard questions pre-set for the above multiple dimensions. It should be noted that if the initial description text meets the preset evaluation criteria, the initial description text can be directly used as the target description text to obtain model input information.

[0073] In this embodiment, the initial description text is analyzed from multiple semantic dimensions, and target questions are raised based on the analysis results to obtain the target results. By asking targeted questions, missing information in the initial description text is collected efficiently, thereby accurately eliminating ambiguity in the initial description text before the target video is generated. This improves the accuracy and usability of the target description text, reduces the cost of repeatedly debugging model prompts during video generation, and improves the video quality, accuracy, and efficiency of the target video generation.

[0074] Step S202 The model input information is input into a pre-trained large language model to output a structured data object. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters.

[0075] Structured data objects can be in JSON format, etc. Rendering function code can be used to implement specific dynamic effects in the target video when executed. The function signature of the rendering function code (which can indicate the rendering mode corresponding to the rendering function code, such as 2D mode or 3D mode) can include time, the current value of adjustable parameters, etc. Adjustable parameters automatically identified by the large language model can have various parameter types, such as one, multiple, or all of the following: numerical, color, boolean, selection, string, image, video, 3D model, texture map, sequence frame, etc. These adjustable parameters can also be obtained by extracting the corresponding external variables after generating the rendering function code by forming the abstract syntax tree corresponding to the rendering function code. In some embodiments, adjustable parameters for video types can include a video start time field used to dynamically calculate the start time offset of the dynamic effect in the video footage. For example, if the value of the video start time field is 1000ms, it means that the dynamic effect starts to appear when the video footage plays for 1000ms. In some embodiments, parameters for texture map types can include a texture type field used to specify the texture channel and a texture model field used to indicate the texture model, etc.

[0076] In some embodiments, the structured data object may also include multiple required or optional fields such as total duration of dynamic effects, rendering mode (2D rendering mode or 3D rendering mode, etc.), and 3D scene configuration (such as camera parameters and light source configuration, etc.).

[0077] In this embodiment, by defining the output generated by the large language model as a structured data object containing rendering function code and a set of adjustable parameters, the target text description of natural language is transformed into an executable and adjustable programming language. This allows the renderer to respond instantly by adjusting the adjustable parameters during the video generation process without having to revert to the large language model. This lowers the barrier to video production while improving the efficiency of video generation and the controllability and flexibility of the generated results.

[0078] Step S204 In response to an adjustment of at least one of the plurality of adjustable parameters, the structured data object is updated.

[0079] Adjustments to adjustable parameters can be input by the target object (such as a user) through operation controls in the parameter panel (dragging a slider, selecting a color, uploading materials, toggling a switch, etc.). In some embodiments, the parameter changes resulting from these operations can be synchronized to the renderer through a state management library, etc. The renderer can re-execute the rendering function code with the new parameter values ​​in the current frame and immediately present the visual effect of the parameter adjustment on the canvas (such as a preview video).

[0080] In this embodiment, the structured data object is updated in real time by accepting user adjustments to the adjustable parameters. When a fine-tuning of the adjustable parameters is detected, the updated structured data object can be directly applied to the renderer without calling the large language model to regenerate the rendering function code. This reduces the consumption of computing resources and interaction latency caused by model inference, and improves the efficiency, controllability and flexibility of video generation.

[0081] As mentioned earlier, the renderer can display preview videos to assist in parameter adjustments. The following describes one specific method for displaying a preview video.

[0082] In optional embodiments, such as Figure 4 As shown, the method further includes: S400, the target renderer executes the rendering function code according to the initial values ​​of each parameter in the parameter set to display the preview video.

[0083] S402, during the preview process, if an adjustment of the adjustable parameters is detected, the target renderer will execute the rendering function code again according to the values ​​of each parameter in the adjusted parameter set to display the updated preview video.

[0084] The target renderer can be selected from multiple renderers based on factors such as the rendering mode of the target video and the parameter types in the structured data object. During the execution of the rendering function code, image or video materials in the model input information can be converted into Blob URLs (temporary URLs generated locally by the browser that point to file objects) through a parameter loading service and preloaded as resource instances such as image instances or video instances. In some embodiments, the preloaded resource instances can also be injected into the corresponding objects, making them input parameters for the rendering function code. If the model input information includes video materials, the playback position of the video elements can be positioned to the current dynamic effect time point based on parameters such as the start time offset through a video synchronization service. If the structured data object contains a field for the total duration of dynamic effects, the total duration of dynamic effects can be dynamically updated after the adjustable parameters are adjusted through the duration calculation function. The updated total duration of dynamic effects also needs to undergo legality checks such as boundary checks (e.g., minimum 1000ms, maximum 60000ms) before it can be written into the structured data object.

[0085] In this embodiment, by re-executing the rendering function code in response to the adjustment of adjustable parameters to update the preview video in real time, the computational latency and resource consumption caused by calling large language models for code reconstruction can be reduced. It also allows users to see the effect after parameter adjustment in real time, improving the feedback efficiency of fine-tuning adjustable parameters and the controllability of the visual effect of the target video.

[0086] When an error occurs while executing the rendering function code, there are several ways to fix it. The following describes one exemplary method.

[0087] In optional embodiments, such as Figure 5 As shown, step S400 includes: S500: If an error occurs during the execution of the rendering function code, obtain the code execution error information.

[0088] S502, the error message and the structured data object are input into the large language model to output the repaired structured data object.

[0089] S504, based on the initial values ​​of each parameter in the parameter set, execute the rendering function code in the repaired structured data object.

[0090] The acquired error information may include the error type, error message, the specific line number where the error occurred, and the complete call stack. During the error information acquisition process, any abnormal states of the execution environment of the rendering function code can also be captured and input into the large language model along with the error information. Upon receiving the error information, the large language model can perform no more than a predetermined number of automatic repair attempts (e.g., 3). In some embodiments, if the automatic repair attempts exceed the predetermined number, the large language model can display the final error context and the history of failed repairs to the relevant objects, thus switching to manual processing mode.

[0091] In this embodiment, the large language model automatically repairs the rendering function code based on the acquired error information. By capturing the runtime error context in real time and using the logical reasoning ability of the large language model to perform precise code reconstruction, automatic closed-loop processing of rendering anomalies is achieved, which enhances the reliability and fault tolerance of video generation, reduces the technical threshold and manual debugging cost of video generation, and improves the continuity and success rate of video generation.

[0092] As mentioned earlier, the target rendering engine can be selected in several ways. The following describes one exemplary selection method.

[0093] In an optional embodiment, the structured data object further includes a rendering mode field, which indicates the rendering mode, including a 3D rendering mode or a 2D rendering mode. The fields corresponding to the one or more adjustable parameters include fields that allow setting 3D model parameters. The target renderer is selected from a plurality of renderers. Figure 6 As shown, the operation of selecting the target renderer includes: S600, when 3D model parameters are set or the 3D rendering mode is in effect, select the 3D renderer as the target renderer.

[0094] S602, when no 3D model parameters are set and the 2D rendering mode is in effect: S602A, determine whether the structured data object includes post-processing effects code.

[0095] S602B, when the structured data object includes the post-processing effects code, the post-processing blend renderer is used as the target renderer, and the post-processing blend renderer includes a 2D renderer and a post-processing shader.

[0096] S602C, if the structured data object does not include the post-processing effects code, select a 2D renderer as the target renderer.

[0097] In some embodiments, when a 3D renderer is selected as the target renderer, the target renderer can support PBR (physically based rendering) materials, covering texture processing for six channels: diffuse, normal, roughness, metallicity, emissivity, and ambient occlusion. It also has built-in multiple light source configurations (ambient light, directional light, point light, spotlight) and dynamic switching between perspective and orthographic cameras, and can also support playback and control of skeletal animation.

[0098] In some embodiments, when selecting a post-processing blending renderer as the target renderer, the rendering result can first be output to an off-screen canvas using a 2D renderer. This off-screen canvas can then be used as a texture, and a post-processing shader can perform post-processing based on post-processing effects code. The post-processing shader can perform multiple rounds of alternating read and write operations using a framebuffer mechanism. Each read / write operation can include various variables such as the previous stage output, original input, resolution, and timestamp. The encoding result of the post-processing shader can be verified using a hash value cache.

[0099] In other embodiments, when selecting a 2D renderer as the target renderer, materials such as images, videos, or sequence frames in the model input information can be converted into Blob URLs, etc., and these materials can be preloaded as image elements or video materials and injected into the parameter object before executing the rendering function code.

[0100] In this embodiment, the target rendering engine is automatically selected based on the content settings in the structured data object. By identifying the dynamic effects attributes (such as 3D model parameters, rendering mode, and post-processing code) in the structured data object, the optimal renderer is automatically selected according to priority. This ensures the visual expressiveness of the target video while reducing the dependence of business logic on the underlying rendering technology, ensuring resource utilization efficiency and performance when handling different types of rendering tasks, and improving the efficiency and reliability of video generation.

[0101] As mentioned earlier, the target rendering engine can be one of many different types of rendering engines. During the actual rendering process, there are various optimization methods for using different types of target rendering engines. The following describes an exemplary optimization method.

[0102] In optional embodiments, such as Figure 7 As shown, step S400 includes: S700, when the target renderer is selected, the rendering function code and the resources corresponding to each parameter are loaded through a unified interface.

[0103] S702, with the rendering function code and the resources corresponding to each parameter loaded, the rendering function code is executed.

[0104] In some embodiments, the unified interface may include various lifecycle management methods (such as initialization, play, pause, search, parameter update, destruction, etc.). The initialization method in the unified interface can be used to load rendering function code and resources.

[0105] In this embodiment, a unified interface is used to call and manage various target rendering engines, shielding the implementation differences of different underlying rendering technologies and improving the compatibility of video generation. Simultaneously, the preloading of rendering function code and resources improves the efficiency of video generation and ensures the smoothness and visual consistency of video output during the execution of the rendering function code.

[0106] As mentioned earlier, the model input information can also include multimedia materials such as videos and images. When using these multimedia materials, the video generation process can be optimized using various methods. The following provides an exemplary optimization method.

[0107] In an optional embodiment, the model input information further includes multimedia material, which comprises multiple video frames. For example... Figure 8 As shown, step S400 includes: S800, determine a target random number sequence based on the frame positions corresponding to the plurality of video frames. The target random number sequence includes a plurality of random numbers, one random number corresponds to one video frame, and the random number is used to indicate the dynamic effects in the corresponding video frame.

[0108] S802, Execute the rendering function code based on the target random number sequence and the initial values ​​of each parameter.

[0109] In some embodiments, the frame position of a video frame can be determined based on its animation timestamp (time, in milliseconds). The animation timestamp times[i] of the i-th video frame among multiple video frames can be calculated using the following formula:

[0110] Here, fps refers to the frame rate of the target video specified in the model input information. In some embodiments, the animation timestamp of the video frame can be used as a seed to generate a random number corresponding to that video frame using a seed pseudo-random number generator or the like.

[0111] For example, if the model input information includes a video frame sequence consisting of 10 video frames (i.e., multimedia material) with a specified frame rate of 24 (frames per second), then the animation timestamp times[4]≈166.7 of the fourth video frame is used as a seed to calculate its corresponding random number using a pseudo-random number algorithm. If the purpose of this video generation is to achieve a rain effect, the random number corresponding to each video frame indicates how many raindrops should be displayed in that video frame, then 4 raindrops should be displayed in the fourth video frame.

[0112] In this embodiment, the corresponding random number is determined based on the frame position of the video frame. By establishing a deterministic mapping relationship between frame position and random number sequence, it is ensured that the same dynamic effects can be generated at the same time point or frame position, thus guaranteeing the consistency of visual effects during the video generation process.

[0113] Step S206 The target video is generated based on the updated structured data object.

[0114] In some embodiments, before generating the target video, the complete sequence of frame timestamps can be pre-calculated using the formula for calculating the animation timestamps of each video frame mentioned above. During the generation of the target video, an off-screen canvas that conforms to the target resolution in the model input information can be created, and the rendering function code can be called frame by frame to render on the off-screen canvas. During frame-by-frame rendering, relevant methods (such as setTimeout(0)) can be used to allow the main thread to be released once every fixed number of frames (e.g., 10) to avoid blocking the browser. The generated target video can be in MP4 or WebM video format, or in GIF or APNG animation format. In some embodiments, to adapt to H.264 encoding requirements, non-even width and height can be automatically aligned by adding one pixel.

[0115] The random numbers mentioned earlier can be used not only in generating preview videos but also in generating target videos. The following describes an exemplary method for applying these random numbers in the target video generation process.

[0116] In an optional embodiment, the plurality of video frames includes target video frames associated with a target random number. For example... Figure 9 As shown, step S206 includes: S900, the rendering function code is executed on the target video frame according to the target random number to obtain the target rendering result corresponding to the target video frame.

[0117] S902, Obtain target pixel data based on the target rendering result.

[0118] S904, the target pixel data is encoded by the target encoder to obtain the target encoding result of the target video frame.

[0119] The target random number can be generated and stored locally during the preview video generation process. In some embodiments, the target random number can also be generated in real time during the target video generation process using the same method (including the same random number function) as during the preview video generation process. The acquired target pixel data can be RGBA four-channel pixel data captured from an off-screen canvas. After acquiring the target pixel data, it can be encoded using an H.264 encoder or similar method to obtain the target encoding result. After obtaining the target encoding result, the corresponding method in the unified interface described above can be called to encapsulate the target encoding result into a specific file format (such as MP4 container format). In some embodiments, all drawing instructions (including the drawing API call sequence) can also be recorded during the preview process, and the recorded drawing instructions can be directly played back when generating the target video to obtain the target video.

[0120] In this embodiment, the target video frames are rendered using the same target random number as during the preview process, ensuring visual consistency between the dynamic effects implemented in the target video and those implemented in the preview video, achieving a WYSIWYG (What You See Is What You Get) effect. Simultaneously, the target pixel data is rendered frame by frame sequentially and captured, then frame-encoded using an independent target encoder, guaranteeing precise alignment of video frame times and high-quality image output, thus improving the video quality of the target video.

[0121] During the generation of the target video, various methods can be used to achieve specific video effects or requirements. The following provides an example method.

[0122] In optional embodiments, such as Figure 10 As shown, the method further includes: S1000, extract the target channel value from the target pixel data according to the target channel.

[0123] S1002, the grayscale encoding result of the target video frame is obtained by the target encoder based on the target channel value.

[0124] The target channel can be the alpha channel of the RGBA four-channel array. In some embodiments, the target channel value can be extracted during the acquisition of target pixel data using a specific method (such as getImageData()). After extracting the target channel value, it can be mapped to grayscale pixel values, and the obtained grayscale pixel values ​​can be encoded to obtain a grayscale encoding result (such as grayscale video, where white represents complete opacity and black represents complete transparency). In some embodiments, the grayscale encoding result and the target video can be packaged together into the same compressed file, and the grayscale encoding result and the target video in the compressed file can be combined into a single video using video processing tools.

[0125] In this embodiment, target channel values ​​are extracted from target pixel data and encoded to obtain grayscale encoding results. By accurately extracting transparency information from pixel data and mapping it to grayscale values ​​for independent encoding, the resulting grayscale video can serve as a transparent background for the target video. This not only ensures high-fidelity transparency of dynamic effects against complex backgrounds but also provides material for post-production compositing, improving the practicality and compatibility of video generation.

[0126] Example 2 Figure 11 The flowchart illustrating the video dynamic effects reproduction method according to Embodiment 2 of this application is shown in the illustration.

[0127] like Figure 11 As shown, the method for reproducing dynamic video effects may include steps S1100~S1108, wherein: S1100 extracts multiple key video frames from a reference video.

[0128] S1102, the multiple key video frames and their corresponding timestamps are input into a large language model to extract features from multiple preset dimensions through the large language model to obtain a first structured data object.

[0129] S1104, the first structured data object and the multiple key video frames are input into the large language model to obtain several target features.

[0130] S1106, Based on the aforementioned target features, obtain a second structured data object.

[0131] S1108, the plurality of key video frames and the second structured data object are input into the large language model to generate a target structured data object, the target structured data object including rendering function code for reproducing dynamic effects in the reference video.

[0132] Key video frames can be extracted uniformly from the reference video, such as sampling 16 key video frames at the same interval. After extracting the key video frames, they can be scaled to ensure that each key video frame meets the preset size requirements (e.g., the shorter side does not exceed 768 pixels), and encoded in a specified format (e.g., JPEG).

[0133] The large language model used for feature extraction can be a visual language model. Multiple preset dimensions for feature extraction can include some or all of the following: element shape, direction of motion, dynamic effect rhythm, blur level, highlight effect, color scheme, etc. The resulting first structured data object can include motion phase divisions, visual element descriptions, temporal information, background information, effect parameters, and code implementation suggestions.

[0134] In some embodiments, the obtained target features can be evaluated from multiple dimensions, and a second structured data object can be obtained based on the evaluation results. For example, target features with data anomalies can be re-extracted, or target features with description accuracy lower than a preset level can be modified or supplemented.

[0135] In this embodiment, through two stages of analysis—initial feature extraction and target feature refinement—along with supplementary quantitative descriptions, dynamic effects in the reference video are reverse-engineered into editable structured data objects. By deconstructing key video frames from multiple preset dimensions using a large language model, abstract visual perceptions are transformed into precise, code-generating structured data objects. This improves the accuracy of visual feature reconstruction of dynamic effects in the reference video, reduces the difficulty and cycle time of reproducing dynamic effects, and enhances the efficiency and flexibility of reproducing video dynamic effects.

[0136] In the process of reproducing dynamic video effects, various optimization methods can be used to improve the reproduction effect. Several exemplary optimization methods are provided below.

[0137] Method 1, in an optional embodiment, such as Figure 12 As shown, step S1106 includes: S1200, determine whether the plurality of target features meet the preset standard.

[0138] S1202 If one or more of the target features do not conform to the preset standard, add a quantitative description to the one or more target features that do not conform to the preset standard to obtain the second structured data object.

[0139] Target features can be core features that determine the visual identity of dynamic special effects. In some embodiments, whether the target features contain a sufficiently precise quantized description can be used as a preset standard. The added quantized description can be a description that supports pixel-by-pixel code replication of the reference video, such as "the halo diameter is 150% of the element's body width" or "the gradient color stop position is x: [0.8, 1.0]". The technical details of the process of generating the target structured data object can be found in the process of generating the structured data object in Embodiment 1. It should be noted that if the obtained target features all meet the preset standard, these target features can be directly used as the second structured data object.

[0140] In this embodiment, the obtained target features are evaluated according to preset standards, and quantitative descriptions are added to target features that do not meet the preset standards. The addition of quantitative descriptions can improve the description accuracy of target features, thereby improving the reproduction accuracy of subsequent dynamic special effects reproduction, and enhancing the reproduction quality and effect of dynamic special effects reproduction.

[0141] Method 2, in optional embodiments, such as Figure 13 As shown, step S1108 includes: S1300, Obtain supplementary correction text for the second structured data object.

[0142] S1302, the second structured data object and supplementary correction text of the multiple key video frames are input into the large language model to generate the target structured data object.

[0143] Supplementary and corrective text can be obtained by presenting a first structured data object and a second structured data object (including key video frame previews, motion phases, element lists, target feature evaluations, etc.) to the target object (such as a creator) and based on the target object's input. In some embodiments, the content input into the large language model may also include constraints or coding standards for rendering performance, such as a maximum particle count, prohibition of nested loops with O(n²) or higher complexity, and requirements for using object pooling techniques to reduce memory jitter.

[0144] In this embodiment, by obtaining supplementary correction text for the second structured data object, the subjective correction from relevant objects (such as creators) is transformed into explicit constraints for the generation of the target structured data object. This can improve the visual accuracy of dynamic special effects reproduction and the logical accuracy of the generation of the target structured data object, while also enhancing the controllability of the dynamic special effects reproduction process and improving the reproduction quality of dynamic special effects.

[0145] Method 3, in optional embodiments, such as Figure 14 As shown, the method further includes: S1400, obtain multiple rendered video frames from the target video, with each rendered video frame corresponding to a key video frame.

[0146] S1402, Generate a comparison video frame sequence based on the plurality of rendered video frames and the key video frames corresponding to each of the plurality of rendered video frames; S1404, The comparison video frame sequence is input into the large language model for comparison to obtain the comparison result, which includes a quantitative description of the differences between each rendered video frame and its corresponding key video frame.

[0147] Rendered video frames can be obtained in the same way as key video frames, such as uniformly sampling 16 rendered video frames at the same frame interval. In some embodiments, the number of rendered video frames can be less than the number of key video frames. For example, 8 key video frames can be uniformly selected from the 16 key video frames, and their corresponding rendered video frames can be obtained to form a comparison video frame sequence. After obtaining the rendered video frames, they can be scaled to ensure that the rendered video frames and key video frames have consistent resolution and other parameters. During the generation of the comparison video frame sequence, the rendered video frames and key video frames can be interleaved. That is, if the first video frame in the comparison video frame sequence is a rendered video frame, then the second video frame should be the key video frame corresponding to the first video frame, and so on. In some embodiments, the large language model can obtain the comparison results through a three-step analysis method: first, identify the core visual features in the key video frames (such as particle emission, rotation speed, blur radius, etc.); then evaluate the performance of the rendered video frames on these specific features; and finally, give a comprehensive score of 1-10. The quantitative descriptions in the comparison results should include specific units of measurement, such as "the arc width of the rendered video frame is 5% of the canvas width, while that of the key video frame is 2%", etc.

[0148] In this embodiment, by aligning and comparing the generated rendered video frames with the original key video frames frame by frame, the abstract visual sensory differences are transformed into concrete and measurable quantitative descriptions using a large language model. This provides deterministic data guidance for the subsequent iterative optimization of the generated dynamic effects, improving the evaluation efficiency and restoration accuracy of the video dynamic effects reproduction process.

[0149] As mentioned earlier, the comparison results can be used in subsequent iterative optimization processes of the generated dynamic effects. The following provides an exemplary iterative optimization method.

[0150] In an optional embodiment, the comparison results further include repair code. The method also includes: The comparison results and the multiple key video frames are input into the large language model to modify the target part of the target structured data object, thereby obtaining the modified target structured data object.

[0151] An updated video is generated based on the modified target structured data object.

[0152] Based on the first round of video updates and subsequent rounds of video updates, multiple rounds of modification operations are performed on the target structured data object until the last round of video updates; in each round of operations: The video frame sequence from the updated video of this round of operation is input into the large language model for comparison to obtain the comparison result of this round.

[0153] If the comparison result in this round is better than the best comparison result in the previous rounds, then the comparison result in this round and multiple key video frames are input into the large language model, so that the model modifies the target part in the specified structured data object according to the repair code in the comparison result in this round, and outputs the updated target structured data object in this round, wherein the specified structured data object is the target structured data object output in the previous round.

[0154] In some embodiments, the visual feature with the highest score can be selected based on the comparison results (the highest score indicates the largest difference in the implementation effect of that visual feature between the rendered video frame and the key video frame), and the code segment (i.e., the target part) responsible for implementing this visual feature in the target structured data object can be modified. During the round of iteration, an iterative version library can be established to record the comparison results of each round of iteration. Modifications for that round are only accepted and updated to the current optimal version if the comparison result of the new round is better than the current optimal version (i.e., the best comparison result among the previous rounds). If the comparison result of this round is not better than the current optimal version, the current optimal version can be re-analyzed and modified according to the aforementioned steps, and the re-modification operation should be different from the previous modification operation. In some embodiments, the current optimal version can be determined as the final updated video when the comparison result reaches a preset evaluation standard (e.g., a comprehensive score higher than 8 points, the higher the comprehensive score, the smaller the difference) or reaches a preset iteration round limit (e.g., 10 rounds).

[0155] In this embodiment, if the current version is better than the current best version, the modification results from the previous round are accepted and modifications continue. This monotonically increasing quality modification strategy avoids the degradation of dynamic effect reproduction quality during iteration. Simultaneously, modifying parts of the target structured data object and specified structured data objects enhances the targeting and repair accuracy of the large language model in the dynamic effect reproduction process, thereby improving the visual fidelity of the dynamic effect reproduction.

[0156] To make this application easier to understand, the following is combined with... Figure 15-19 An example application is provided. Wherein: S1. Receive the initial animation description text input by the user (such as "scattered golden particles"), and perform semantic ambiguity detection from six preset dimensions such as animation type, color, and speed.

[0157] S2. If the description is not clear, the system outputs a targeted question (such as "Is the particle moving upward or downward?") and generates the final target description text based on the user's answer.

[0158] S3. Input the text into the large language model and output a structured data object containing rendering function code, a set of ten types of parameters, and a rendering mode field.

[0159] S4. Based on whether the structured data contains 3D model parameters or post-processing shader code, automatically select the target renderer from 2D renderers, 3D renderers, or post-processing hybrid renderers.

[0160] S5, the target renderer loads rendering code and multimedia materials through a unified interface, performs rendering based on the initial parameter values, and displays a real-time animation preview video.

[0161] S6. If an error occurs during rendering, the error stack information is automatically obtained and fed back to the model. The corrected structured data is output through up to three automatic iterations.

[0162] S7. Users can adjust the values ​​or colors on the parameter panel during the preview process, and the video preview effect will be updated in real time in response to the adjustment, without having to call the model generation code again.

[0163] S8. Calculate the precise frame time and inject a seed random number sequence to ensure that the preview and the exported dynamic effects are consistent pixel by pixel.

[0164] S9: Renders frame by frame to obtain RGBA pixel data, generates target video through target encoder, and supports extracting Alpha channel values ​​separately to output grayscale encoded files with transparency.

[0165] S10. Extract 16 keyframes and timestamps from the reference video, input them into a large language model for two-stage feature extraction to obtain a structured object containing quantified descriptions.

[0166] S11. Based on the corrections manually provided by the user, the model generates the initial rendering function code to reproduce the motion effects of the reference video.

[0167] S12. The system submits the generated video frames and reference frames to the model in an alternating arrangement to obtain a quantitative scoring comparison result containing specific difference metrics and repair codes.

[0168] S13. Based on the scoring-driven approach, perform multiple rounds of iterative modifications, retaining only the code with a score better than the previous version, until the visual reproduction accuracy meets the standard or the iteration limit is reached.

[0169] Example 3 Figure 20 The diagram schematically illustrates a video generation apparatus according to Embodiment 3 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 20 As shown, the device 1000 may include: an acquisition module 1100, an input module 1200, an update module 1300, and a generation module 1400, wherein: The acquisition module 1100 is used to acquire model input information, which includes target description text; The input module 1200 is used to input model input information into a pre-trained large language model so as to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters. The update module 1300 is configured to update the structured data object in response to an adjustment of at least one of the plurality of adjustable parameters; The generation module 1400 is used to generate the target video based on the updated structured data object.

[0170] As an optional embodiment, the acquisition module 1100 is further configured to: Receive initial description text; Based on multiple preset semantic dimensions, determine whether the initial description text meets the preset evaluation criteria; If the initial description text does not meet the preset evaluation criteria, no more than a predetermined number of target questions are output based on the initial description text, so as to receive the target results input for the target questions; Upon receiving the target result, the target description text is generated based on the target result and the initial description text.

[0171] As an optional embodiment, the device 1000 is also used for: The target renderer executes the rendering function code based on the initial values ​​of each parameter in the parameter set to display the preview video; During the preview process, if an adjustment to the adjustable parameters is detected, the target renderer will execute the rendering function code again based on the values ​​of each parameter in the adjusted parameter set to display the updated preview video.

[0172] As an optional embodiment, the device 1000 is also used for: If an error occurs during the execution of the rendering function code, obtain the code execution error information; The error message and the structured data object are input into the large language model to output the repaired structured data object; Based on the initial values ​​of each parameter in the parameter set, the rendering function code in the repaired structured data object is executed.

[0173] As an optional embodiment, the structured data object further includes a rendering mode field, which indicates the rendering mode, including a 3D rendering mode or a 2D rendering mode; the fields corresponding to the one or more adjustable parameters include fields that can set 3D model parameters. The target renderer is selected from a plurality of renderers, and the apparatus 1000 is further configured to: If 3D model parameters are set or the 3D rendering mode is in effect, select the 3D renderer as the target renderer. In the case where no 3D model parameters are set and the 2D rendering mode is in effect: Determine whether the structured data object includes post-processing effects code; In the case where the structured data object includes the post-processing effects code, the post-processing blend renderer is used as the target renderer, and the post-processing blend renderer includes a 2D renderer and a post-processing shader. If the structured data object does not include the post-processing effects code, then the 2D renderer is selected as the target renderer.

[0174] As an optional embodiment, the device 1000 is also used for: With the target renderer selected, the rendering function code and the resources corresponding to each parameter are loaded through a unified interface; The rendering function code is executed after loading the rendering function code and the resources corresponding to each parameter.

[0175] As an optional embodiment, the model input information further includes multimedia materials, which include multiple video frames, and the device 1000 is further used for: Based on the frame positions corresponding to the multiple video frames, a target random number sequence is determined. The target random number sequence includes multiple random numbers, with each random number corresponding to a video frame. The random number is used to indicate the dynamic effects in the corresponding video frame. The rendering function code is executed based on the target random number sequence and the initial values ​​of each parameter.

[0176] As an optional embodiment, the plurality of video frames includes target video frames associated with a target random number, and the apparatus 1000 is further configured to: The rendering function code is executed on the target video frame according to the target random number to obtain the target rendering result corresponding to the target video frame; Obtain target pixel data based on the target rendering result; The target encoding result of the target video frame is obtained by encoding the target pixel data through the target encoder.

[0177] As an optional embodiment, the device 1000 is also used for: Extract the target channel value from the target pixel data according to the target channel; The grayscale encoding result of the target video frame is obtained by the target encoder based on the target channel value.

[0178] Example 4 Figure 21 The diagram schematically illustrates a video motion effects reproduction device according to Embodiment 4 of this application. This device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of this application. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. For example... Figure 21 As shown, the device 2000 may include: an extraction module 2100, a first input module 2200, a second input module 2300, an acquisition module 2400, and a generation module 2500, wherein: Extraction module 2100 is used to extract multiple key video frames from a reference video; The first input module 2200 is used to input the plurality of key video frames and the timestamps corresponding to each of the plurality of key video frames into the large language model, so as to extract features from multiple preset dimensions through the large language model to obtain a first structured data object. The second input module 2300 is used to input the first structured data object and the multiple key video frames into the large language model to obtain several target features; The acquisition module 2400 is used to acquire a second structured data object based on the plurality of target features; The generation module 2500 is used to input the plurality of key video frames and the second structured data object into the large language model to generate a target structured data object, wherein the target structured data object includes rendering function code for reproducing dynamic effects in the reference video.

[0179] As an optional embodiment, the acquisition module 2400 is also used for: Determine whether the aforementioned target features meet preset standards; If one or more of the target features do not conform to the preset standard, a quantitative description is added to the one or more target features that do not conform to the preset standard to obtain the second structured data object.

[0180] As an optional embodiment, the generation module 2500 is also used for: Obtain supplementary correction text for the second structured data object; The second structured data object and supplementary correction text of the multiple key video frames are input into the large language model to generate the target structured data object.

[0181] As an optional embodiment, the device 2000 is also used for: Multiple rendered video frames are obtained from the target video, and each rendered video frame corresponds to one key video frame; A comparison video frame sequence is generated based on the plurality of rendered video frames and the key video frames corresponding to each of the plurality of rendered video frames; The comparison video frame sequence is input into the large language model for comparison to obtain comparison results, which include a quantitative description of the differences between each rendered video frame and its corresponding key video frame.

[0182] As an optional embodiment, the comparison results further include repair code, and the device 2000 is also used for: The comparison results and the multiple key video frames are input into the large language model to modify the target part of the target structured data object through the large language model, so as to obtain the modified target structured data object. An updated video is generated based on the modified target structured data object; Based on the current round of video updates and each subsequent round of video updates, multiple rounds of modification operations are performed on the target structured data object until the final round of video updates; In each round of operations: The video frame sequence from the updated video of this round of operation is input into the large language model for comparison to obtain the comparison result of this round; If the comparison result in this round is better than the best comparison result in the previous rounds, then the comparison result in this round and multiple key video frames are input into the large language model, so that the model modifies the target part in the specified structured data object according to the repair code in the comparison result in this round, and outputs the updated target structured data object in this round, wherein the specified structured data object is the target structured data object output in the previous round.

[0183] Example 5 Figure 22 This illustration schematically depicts the hardware architecture of a computer device 10000 suitable for implementing a video life method or a video dynamic special effects reproduction method according to Embodiment 5 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 21 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for video life methods or video dynamic effects reproduction methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0184] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0185] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0186] It should be pointed out that, Figure 22 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0187] In this embodiment, the video life method or video dynamic special effects reproduction method stored in the memory 10010 can be further divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0188] Example 6 This application also provides a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of the video life method or video dynamic special effects reproduction method in the embodiments.

[0189] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the video life method or video dynamic special effects reproduction method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0190] Example 7 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0191] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0192] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that, The method includes: Obtain model input information, which includes target description text; The model input information is input into a pre-trained large language model to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters. In response to an adjustment to at least one of the adjustable parameters, the structured data object is updated; Generate the target video based on the updated structured data object.

2. The method according to claim 1, characterized in that, Obtain model input information, including: Receive initial description text; Based on multiple preset semantic dimensions, determine whether the initial description text meets the preset evaluation criteria; If the initial description text does not meet the preset evaluation criteria, no more than a predetermined number of target questions are output based on the initial description text, so as to receive the target results input for the target questions; Upon receiving the target result, the target description text is generated based on the target result and the initial description text.

3. The method according to claim 1, characterized in that, The method further includes: The target renderer executes the rendering function code based on the initial values ​​of each parameter in the parameter set to display the preview video; During the preview process, if an adjustment to the adjustable parameters is detected, the target renderer will execute the rendering function code again based on the values ​​of each parameter in the adjusted parameter set to display the updated preview video.

4. The method according to claim 3, characterized in that, The rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: If an error occurs during the execution of the rendering function code, obtain the code execution error information; The error message and the structured data object are input into the large language model to output the repaired structured data object; Based on the initial values ​​of each parameter in the parameter set, the rendering function code in the repaired structured data object is executed.

5. The method according to claim 3, characterized in that, The structured data object also includes a rendering mode field, which indicates the rendering mode, including a 3D rendering mode or a 2D rendering mode; the fields corresponding to the one or more adjustable parameters include fields that can set 3D model parameters; The target renderer is selected from multiple renderers, and the operation of selecting the target renderer includes: If 3D model parameters are set or the 3D rendering mode is in effect, select the 3D renderer as the target renderer. In the case where no 3D model parameters are set and the 2D rendering mode is in effect: Determine whether the structured data object includes post-processing effects code; In the case where the structured data object includes the post-processing effects code, the post-processing blend renderer is used as the target renderer, and the post-processing blend renderer includes a 2D renderer and a post-processing shader. If the structured data object does not include the post-processing effects code, then the 2D renderer is selected as the target renderer.

6. The method according to claim 3, characterized in that, The rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: With the target renderer selected, the rendering function code and the resources corresponding to each parameter are loaded through a unified interface; The rendering function code is executed after loading the rendering function code and the resources corresponding to each parameter.

7. The method according to claim 3, characterized in that, The model input information also includes multimedia materials, which include multiple video frames. The rendering function code is executed by the target renderer based on the initial values ​​of each parameter in the parameter set, including: Based on the frame positions corresponding to the multiple video frames, a target random number sequence is determined. The target random number sequence includes multiple random numbers, with each random number corresponding to a video frame. The random number is used to indicate the dynamic effects in the corresponding video frame. The rendering function code is executed based on the target random number sequence and the initial values ​​of each parameter.

8. The method according to claim 7, characterized in that, The plurality of video frames includes target video frames associated with target random numbers; Based on the updated structured data object, generate the target video, including: The rendering function code is executed on the target video frame according to the target random number to obtain the target rendering result corresponding to the target video frame; Obtain target pixel data based on the target rendering result; The target encoding result of the target video frame is obtained by encoding the target pixel data through the target encoder.

9. The method according to claim 8, characterized in that, The method further includes: Extract the target channel value from the target pixel data according to the target channel; The grayscale encoding result of the target video frame is obtained by the target encoder based on the target channel value.

10. A method for reproducing dynamic special effects in video, characterized in that, The method includes: Extract multiple key video frames from the reference video; The multiple key video frames and their corresponding timestamps are input into a large language model to extract features from multiple preset dimensions to obtain a first structured data object. The first structured data object and the multiple key video frames are input into the large language model to obtain several target features; Based on the aforementioned target features, a second structured data object is obtained; The multiple key video frames and the second structured data object are input into the large language model to generate a target structured data object, which includes rendering function code for reproducing dynamic effects in the reference video.

11. The method according to claim 10, characterized in that, Based on the aforementioned target features, a second structured data object is obtained, including: Determine whether the aforementioned target features meet preset standards; If one or more of the target features do not conform to the preset standard, a quantitative description is added to the one or more target features that do not conform to the preset standard to obtain the second structured data object.

12. The method according to claim 10, characterized in that, The multiple key video frames and the second structured data object are input into the large language model to generate the target structured data object, including: Obtain supplementary correction text for the second structured data object; The second structured data object and supplementary correction text of the multiple key video frames are input into the large language model to generate the target structured data object.

13. The method according to claim 10, characterized in that, The method further includes: Multiple rendered video frames are obtained from the target video, and each rendered video frame corresponds to one key video frame; A comparison video frame sequence is generated based on the plurality of rendered video frames and the key video frames corresponding to each of the plurality of rendered video frames; The comparison video frame sequence is input into the large language model for comparison to obtain comparison results, which include a quantitative description of the differences between each rendered video frame and its corresponding key video frame.

14. The method according to claim 13, characterized in that, The comparison results also include repair code, and the method further includes: The comparison results and the multiple key video frames are input into the large language model to modify the target part of the target structured data object through the large language model, so as to obtain the modified target structured data object. An updated video is generated based on the modified target structured data object; Based on the current round of video updates and each subsequent round of video updates, multiple rounds of modification operations are performed on the target structured data object until the final round of video updates; In each round of operations: The video frame sequence from the updated video of this round of operation is input into the large language model for comparison to obtain the comparison result of this round; If the comparison result in this round is better than the best comparison result in the previous rounds, then the comparison result in this round and multiple key video frames are input into the large language model, so that the model modifies the target part in the specified structured data object according to the repair code in the comparison result in this round, and outputs the updated target structured data object in this round, wherein the specified structured data object is the target structured data object output in the previous round.

15. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire model input information, which includes target description text; The input module is used to input model input information into a pre-trained large language model so as to output a structured data object through the large language model. The structured data object includes rendering function code and a parameter set, and the parameter set includes several adjustable parameters. An update module is configured to update the structured data object in response to an adjustment of at least one of the plurality of adjustable parameters; The generation module is used to generate the target video based on the updated structured data object.

16. A device for reproducing dynamic video effects, characterized in that, The device includes: The extraction module is used to extract multiple key video frames from the reference video. The first input module is used to input the plurality of key video frames and the timestamps corresponding to each of the plurality of key video frames into the large language model, so as to extract features from multiple preset dimensions through the large language model to obtain a first structured data object. The second input module is used to input the first structured data object and the multiple key video frames into the large language model to obtain several target features; The acquisition module is used to acquire a second structured data object based on the aforementioned target features; A generation module is used to input the plurality of key video frames and the second structured data object into the large language model to generate a target structured data object, wherein the target structured data object includes rendering function code for reproducing dynamic effects in the reference video.

17. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 14.

19. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 14.

Images