A method and apparatus for generating an explanatory video, a computing device, and a storage medium

Abstract

The specification provides a method, device, computing equipment and storage medium for generating an explanation video. The specification provides a code generation model for generating animation code according to a flow video explanation script; determines a basic reward value of the animation code and a video quality reward value according to whether the animation code can successfully obtain an explanation video and a picture quality condition of key frame images in the explanation video; optimizes the code generation model according to the basic reward value of the animation code and the video quality reward value, so that the animation code obtained by the optimized code generation model has a higher probability of being successfully rendered to obtain an explanation video, and the picture quality condition of key frame images in the explanation video is better. In this way, manual modification and debugging of the animation code obtained by the code generation model can be reduced, and manual adjustment of the explanation video obtained by the animation code can be reduced.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computing device and storage medium for generating explanatory videos. Background Technology

[0002] Currently, the production of instructional videos, such as problem-solving videos, mainly relies on manual or semi-automatic tools to write explanation scripts and animation code. This method suffers from inefficiency and inconsistent quality: on the one hand, manually creating videos frame by frame or writing code is time-consuming and labor-intensive, making it difficult to quickly generate a large amount of high-quality content; on the other hand, the quality of videos generated by different creators or tools varies.

[0003] In recent years, large language models with powerful reasoning and code generation capabilities (such as GPT-4 and Google Gemini) have emerged. They have not only achieved breakthroughs in natural language understanding but can also automatically generate executable code based on text requirements. This means we have the opportunity to have large language models simultaneously act as "teachers" and "programmers": first, the large language model understands the content to be explained (e.g., a problem) and generates explanatory text, like a teacher; then, like a programmer, it converts the explanatory text into code that can be used to generate videos. This allows for the automatic generation of instructional videos through the code generated by the large language model. In particular, some large models with enhanced reasoning capabilities can ensure the correctness and logic of problem-solving steps, thereby improving the quality of the explanations.

[0004] Existing methods for automatically generating instructional videos using large models have alleviated the production burden to some extent, but still have many shortcomings: On the one hand, the code generated by the large language model in existing solutions often fails to generate videos successfully, requiring manual modification and debugging to run successfully, resulting in a lengthy generation process; on the other hand, the quality of videos generated using the above code is often unsatisfactory, requiring modification and adjustment. For example, elements frequently exceed the screen boundaries or overlap with each other, and these "out-of-frame" and "overlapping" issues affect the video's visual appeal, requiring additional manual adjustments to the video's layout. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this specification provides a method, apparatus, computing device, and storage medium for generating explanatory videos.

[0006] According to a first aspect of the embodiments of this specification, a method for generating explanatory videos is provided, the method comprising:

[0007] The streamlined video tutorial script, which is output by the large model based on the input information, is input into the code generation model to obtain N segments of animation code corresponding to the input information; N is greater than 1.

[0008] For each animation code segment, compile and render the video. If an explanatory video is successfully obtained, determine the base reward value of the animation code segment as the first value. Select at least two keyframe images from the explanatory video and obtain the image quality of each keyframe image. Determine the video quality reward value of the animation code segment based on the image quality of each keyframe image. Based on the base reward value and the video quality reward value of the animation code segment, determine the relative advantage value of the animation code segment relative to the N animation code segments.

[0009] Based on the relative advantage value of each animation code segment relative to the N animation code segments, an objective function is determined; the code generation model is optimized based on the objective function to adjust the code generation model toward the optimal objective; the optimal objective indicates that the animation code generated by the code generation model is successfully rendered into an explanatory video and the image quality of the keyframe images in the explanatory video meets the set optimal requirements.

[0010] According to a second aspect of the embodiments of this specification, a video generation apparatus is provided, the apparatus comprising:

[0011] The generation module is used to input the streamlined video explanation script, which is output by the large model based on the input information, into the code generation model to obtain N segments of animation code corresponding to the input information; N is greater than 1.

[0012] The module is used to compile and render the animation code for each segment. If an explanatory video is successfully obtained, the base reward value of the animation code segment is determined to be the first value. At least two keyframe images are selected from the explanatory video, and the image quality of each keyframe image is obtained. The video quality reward value of the animation code segment is determined based on the image quality of each keyframe image. Based on the base reward value and the video quality reward value of the animation code segment, the relative advantage value of the animation code segment relative to the N animation code segments is determined.

[0013] An optimization module is used to determine an objective function based on the relative advantage value of each animation code segment relative to the N animation code segments; optimize the code generation model based on the objective function so that the code generation model is adjusted towards the optimal objective; the optimal objective indicates that the animation code generated by the code generation model is successfully rendered into an explanatory video and the image quality of the key frame images in the explanatory video meets the set optimal requirements.

[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, the computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the explanatory video generation method described in the first aspect above.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a program is stored, the program being executed by a processor of the operations performed by the explanatory video generation method described in the first aspect above.

[0016] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, performs the operations performed by the explanatory video generation method described in the first aspect above.

[0017] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0018] This paper provides a code generation model for generating animated code based on a streamlined video tutorial script. A base reward value for the animated code and a video quality reward value are determined by whether the animated code successfully renders into the tutorial video and the image quality of keyframes within the video. The code generation model is then optimized based on these base and quality reward values, resulting in an optimized model that increases the probability of successfully rendering the animated code into the tutorial video and improves the image quality of keyframes within the video. This reduces the need for manual modification and debugging of the animated code generated by the code generation model, as well as manual adjustments to the resulting tutorial video.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0021] Figure 1 This is a flowchart of a video generation method provided by the present invention;

[0022] Figure 2 A schematic diagram of a keyframe image with out-of-frame issues provided by the present invention;

[0023] Figure 3 A schematic diagram of a keyframe image with overlapping issues provided by the present invention;

[0024] Figure 4 A flowchart of a method for generating explanatory videos for mathematical problems provided by the present invention;

[0025] Figure 5 This is a schematic diagram of an explanatory video generation device provided by the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of a computing device provided by the present invention. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed herein.

[0028] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “described,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0030] See Figure 1 , Figure 1 This is a flowchart illustrating a video generation method provided by the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0031] S101. Input the streamlined video explanation script, which is output by the large model based on the input information, into the code generation model to obtain N segments of animation code corresponding to the input information.

[0032] Where N is greater than 1.

[0033] The large model in this invention possesses reasoning capabilities and can output a streamlined video explanation script based on input information. For example, the input information can be a math problem to be explained, and the output streamlined video explanation script can be a detailed step-by-step explanation script. In simpler terms, the large model, like a math teacher, understands the math problem to be explained and writes out each step of the solution process—that is, a streamlined video explanation script corresponding to the math problem.

[0034] The process-oriented video tutorial script is input into the code generation model, which generates N segments of animated code corresponding to the input information. Each segment of animated code is a programming language, describing how to display the process-oriented video tutorial script within the video. In simpler terms, the code generation model simulates programmer behavior, converting the process-oriented video tutorial script into N segments of animated code, each segment using a programming language to describe how to display the process-oriented video tutorial script within the video. For example, the animated code could be a Python script.

[0035] S102. For each piece of animation code, compile and render the video. If an explanatory video is successfully obtained, determine that the basic reward value of the animation code is the first value.

[0036] Whether the animation code can successfully generate an explanatory video is a key indicator for evaluating the quality of the animation code in this invention, which is also the basic reward value.

[0037] In some embodiments, the animation code is a Python script, and the Manim animation library can be used to render the video content when the animation code is compiled and the video is rendered.

[0038] S103. Select at least two keyframe images from the explanatory video and obtain the image quality of each keyframe image; determine the video quality bonus value of the animation code segment based on the image quality of each keyframe image.

[0039] The quality of the explanatory video obtained through the animation code is also one of the indicators for quality evaluation of the animation code in this invention, namely the video quality bonus value.

[0040] In this invention, the explanatory video is composed of multiple image frames, and the image quality of the image frames that make up the explanatory video can reflect the quality of the explanatory video.

[0041] Keyframe images are frames in a video that are of critical significance, representing a condensation of visual information. To improve the efficiency and effectiveness of video quality assessment, a video quality bonus value for the animation code segment can be determined based on the image quality of each keyframe image.

[0042] As one possible implementation, in order to ensure that the selected keyframe images cover as much of the core content and key information of the explanatory video as possible, at least two keyframe images are selected from the explanatory video, including: dividing the explanatory video into at least two video segments according to a set event and / or a set duration; extracting representative images from each video segment; and determining keyframe images based on the extracted images.

[0043] In some embodiments, when segmenting video clips, the segments can be divided according to key nodes such as changes in the topic of explanation, scene switching, and changes in demonstration operations, to ensure that each clip contains a relatively independent and complete unit; when extracting representative images, the video clips can be selected to contain iconic scenes that contain the core explanation content and scene transitions.

[0044] In some embodiments, the narration video can be segmented according to the set duration of the video segments. For example, if the preset duration is 5 seconds, and the total duration of the narration video is 15 seconds, it can be evenly divided into 3 video segments, each with a duration of 5 seconds, and each video segment corresponds to a consecutive time interval.

[0045] As one possible implementation, extracting representative images from each video segment includes: for each video segment, using the first and / or last frame images of that video segment as representative images.

[0046] In some embodiments, the first frame of a video clip presents the initial visual information of the video clip, such as the initial state of elements, and serves as a reference for subsequent changes in the scene; the last frame of a video clip accumulates the visual changes of the entire video clip, and can cover the final state of elements, dynamic results, etc.

[0047] Furthermore, extracting the first and / or last frame images from video clips is a simple and efficient operation that requires no complex algorithm calculations.

[0048] One possible implementation is to obtain the image quality of each keyframe image, which includes: for each keyframe image, inputting the keyframe image into a trained image detector, and determining the image quality of the keyframe image based on the output of the image detector.

[0049] In some embodiments, a general image detector can be used to assess the image quality of keyframe images.

[0050] In some embodiments, the image detector can be supervisedly fine-tuned based on the image quality that the user is particularly concerned about.

[0051] As one possible implementation, determining the video quality reward value of the animation code segment based on the image quality of each keyframe image includes: counting the number of keyframe images with the same image problem in the image quality, determining the score corresponding to the problem based on the number and the total number of keyframe images; and determining the video quality reward value of the animation code segment based on the score corresponding to each problem and the weight set for each problem.

[0052] In some embodiments, the score corresponding to a problem can be determined by the ratio of the number of keyframe images showing the same image problem to the total number of keyframe images in the image quality condition.

[0053] In some embodiments, a weighted calculation can be performed based on the scores corresponding to each question and the weights set for each question, and the video quality bonus value of the animation code segment can be determined based on the calculation result.

[0054] As one possible implementation, the image quality indicates whether there are out-of-frame issues and / or overlap issues in the keyframe image; image issues are out-of-frame issues and / or overlap issues.

[0055] In some embodiments, common image problems in keyframe images include out-of-frame and / or overlap problems. Therefore, when assessing image quality, the focus is on out-of-frame and / or overlap problems in keyframe images.

[0056] The "out-of-frame" problem refers to situations where visual elements in a keyframe image extend beyond the image boundaries of that keyframe. For example, Figure 2 A schematic diagram of a keyframe image with out-of-frame issues provided by the present invention, as shown below. Figure 2 As shown, in this keyframe image, several visual elements, such as "8a+b=7" on the left side, extend beyond the image boundary.

[0057] The overlap problem refers to the situation where two or more visual elements in a keyframe image overlap each other, for example, Figure 3 This is a schematic diagram of a keyframe image with overlapping issues provided by the present invention, as shown below. Figure 3 As shown, the visual elements “4-cent stamps: 30” and “100x8=800” in the keyframe image overlap each other.

[0058] In some embodiments, the image detector used to determine the image quality of a keyframe image can be fine-tuned based on an InternVL-2 model. The image detector can binary classify and output whether the keyframe image has out-of-frame or overlapping issues. For example, two image detectors are used to determine the image quality of a keyframe image: one image detector is used to determine whether the keyframe image has out-of-frame issues. The keyframe image is input to this image detector. If the keyframe image has out-of-frame issues, the image detector outputs 1; if the keyframe image does not have out-of-frame issues, the image detector outputs 0. The other image detector is used to determine whether the keyframe image has overlapping issues. The keyframe image is input to this image detector. If the keyframe image has overlapping issues, the image detector outputs 1; if the keyframe image does not have overlapping issues, the image detector outputs 0.

[0059] As a possible implementation, if the explanatory video is not successfully obtained, the method further includes: determining the base reward value of the animation code segment as a second value, the second value being less than the first value; determining the video quality reward value of the animation code segment as a specified value; and returning the steps of determining the relative advantage value of the animation code segment relative to N animation codes based on the base reward value of the animation code segment and the video quality reward value of the animation code segment.

[0060] This invention aims to successfully obtain an explanatory video after compiling and rendering the animation code, therefore the second value is set to be less than the first value.

[0061] In some embodiments, the first value is 1 and the second value is 0. That is, when the animation code is successfully compiled and rendered to obtain an explanatory video, the base reward value of the animation code is determined to be 1; when the explanatory video is not successfully obtained, the base reward value of the animation code is determined to be 0.

[0062] In some embodiments, the specified value is 0. That is, if no explanatory video is successfully obtained, the video quality bonus value for that animation code segment is determined to be 0.

[0063] S104. Based on the base reward value of the animation code segment and the video quality reward value of the animation code segment, determine the relative advantage value of the animation code segment relative to N animation codes segment.

[0064] In this invention, the relative advantage value of each animation code segment can reflect the relative advantage of the basic reward value and video quality reward value of that animation code segment in a group of N animation code segments.

[0065] For example, for each piece of animation code, the overall reward value of that animation code can be determined based on its base reward value and its video quality reward value. Based on the overall reward values ​​of N animation codes, the average and standard deviation of the overall reward values ​​of the N animation codes can be determined. The relative advantage value of that animation code is calculated as: (Overall reward value of that animation code - Average of the overall reward values ​​of N animation codes) / Standard deviation of the overall reward values ​​of N animation codes.

[0066] S105. Determine the objective function based on the relative advantage value of each animation code segment relative to N animation code segments; optimize the code generation model based on the objective function so that the code generation model is adjusted towards the optimal objective.

[0067] Among them, the animation code generated by the optimal target indication code generation model was successfully rendered into an explanatory video, and the image quality of the key frame images in the explanatory video met the set optimal requirements.

[0068] In this invention, the objective function is a function of the generation probability of each animation code segment in the code generation model. The code generation model is optimized by using the objective function determined by the relative advantage value of each animation code segment relative to N animation codes. This can increase the generation probability of animation codes with larger relative advantage values ​​in the code generation model, thereby making the animation codes generated by the code generation model more likely to be successfully rendered into the explanatory video, and resulting in better image quality of keyframe images in the explanatory video.

[0069] In some embodiments, the model parameters of the code generation model can be updated according to the gradient direction of the objective function to maximize the objective function.

[0070] Specifically, based on the relative advantage value of each animation code segment relative to N animation codes, the objective function is determined, including: for each action in the objective process corresponding to each animation code segment, multiplying the importance sampling ratio of that action in the objective process corresponding to that animation code segment by the relative advantage value of that animation code segment to obtain the objective product of that action in the objective process corresponding to that animation code segment; wherein, the objective process corresponding to that objective code segment is the process of inputting the streamlined video explanation script output by the large model based on the input information into the code generation model to obtain that animation code segment.

[0071] For each action in the target process corresponding to each animation code segment, if the importance sampling ratio of that action in the target process corresponding to that animation code segment is greater than a preset ratio threshold, then the target product of that action in the target process corresponding to that animation code segment is updated to the product of the preset ratio threshold and the relative advantage value of that animation code segment.

[0072] The objective function is determined based on the objective product of each action in the target process corresponding to each animation code segment and the KL divergence constraint corresponding to the code generation model; wherein, the KL divergence constraint corresponding to the code generation model indicates the degree of difference between the code generation model before and after optimization based on the objective function.

[0073] It should be noted that this method is applied to the training phase of the code generation model.

[0074] In some embodiments, the optimization can end when a preset number of iterations is reached, such as 10 iterations; or when the code generation model converges, that is, when the change in the code generation model is less than a threshold in several consecutive iterations.

[0075] The following describes a method for generating explanatory videos provided by the present invention, using specific examples.

[0076] Currently, the production of instructional videos mainly relies on manual or semi-automatic tools to write instructional scripts and animation code. This approach suffers from inefficiency and inconsistent quality: on the one hand, manually creating or writing code frame by frame is time-consuming and labor-intensive, making it difficult to quickly generate large amounts of high-quality content; on the other hand, the quality of videos generated by different creators or tools varies considerably. Some automated attempts have been made, such as using the programming animation library Manim in conjunction with large language models to generate instructional video code, which alleviates the production burden to some extent, but still has many shortcomings.

[0077] 1. Low efficiency: Existing solutions often require multiple rounds of manual intervention and code debugging. For example, the Manim code generated by large language models often requires manual modification to run successfully, resulting in a lengthy generation process.

[0078] 2. Unstable quality: Automatically generated video content sometimes lacks logical coherence or fails to highlight key points. Due to the lack of a stable verification mechanism, the generation quality may fluctuate greatly for different questions.

[0079] 3. Disconnect between content and presentation: In existing technologies, the textual explanation and the animation presentation may be out of sync or inconsistent. For example, the model may provide correct textual explanations, but the generated animation may fail to accurately reflect the solution steps, resulting in a disconnect between the two.

[0080] 4. Video frame out-of-frame / overlapping issues: Animations generated by existing methods often exhibit elements that extend beyond the frame boundaries or overlap with each other. These "out-of-frame" and "overlapping" issues affect the video's visual appeal and require additional manual adjustments to the layout.

[0081] The aforementioned problems have not yet been effectively solved in existing technologies. However, recent advances in large language models have brought new opportunities to this field.

[0082] In recent years, large language models with powerful reasoning and code generation capabilities (such as GPT-4 and Google Gemini) have emerged continuously. They have not only achieved breakthroughs in natural language understanding but can also automatically generate executable code based on text requirements. This means we have the opportunity to allow AI (Artificial Intelligence) to simultaneously play the roles of "teacher" and "programmer": first, understanding the problem and explaining the steps like a teacher, and then, like a programmer, converting the explanation into an animated script. In particular, some large models with enhanced reasoning capabilities can ensure the correctness and logic of the problem-solving steps, thereby improving the quality of the explanations.

[0083] While LLM (Large Language Model) provides a powerful engine for automatic video generation, further optimizing the generation results remains a challenge. Existing technologies have not yet applied reinforcement learning methods to this task. For example, no literature reports using reinforcement learning algorithms such as GRPO (Generalized Reinforcement Policy Optimization) to adjust the LLM's generation strategy based on the success or failure of video generation and its presentation quality. In other words, there is no precedent in the industry for using "generation result feedback" to train models to improve script generation quality and animation presentation. This invention differs significantly from existing technologies in its comprehensive utilization of large language models and reinforcement learning, effectively addressing the aforementioned efficiency and quality issues.

[0084] This invention aims to provide an end-to-end automated video generation system to significantly improve the efficiency and quality of script generation and animation rendering. Through this system, users only need to provide a math problem, and the system can automatically generate a complete video containing step-by-step explanations and animated demonstrations.

[0085] Specifically, the purpose of this invention is to address several key pain points in the prior art:

[0086] 1. Improve generation success rate: Resolve issues with Manim code generation failures or inability to run, ensuring that every question can be successfully compiled into an animated video.

[0087] 2. Improve content and visual quality: Eliminate out-of-frame and overlapping issues in video content to make the animation clear and beautiful; ensure that the narration and animation demonstration are synchronized, logically rigorous, and highlight key points.

[0088] 3. Reduce manual intervention costs: Through automated script generation and error correction mechanisms, the workload of manual editing and debugging is reduced, shortening the video production cycle.

[0089] In summary, the goal of this invention is to construct a complete technical process that includes modules such as automatic thinking, code generation, visual inspection, and reinforcement learning optimization, achieving full automation from question input to explanation video output. Through this system, the production of explanation videos will become efficient and reliable, and can be widely applied to online education platforms, teaching and tutoring tools, and other scenarios.

[0090] Figure 4 A flowchart of a method for generating explanatory videos for mathematical problems provided by this invention is shown below. Figure 4 As shown, the process of this method includes the following steps.

[0091] For example, the following is the math problem that will be explained in the video, which will serve as the input to the large language model.

[0092] Xiaoming chases a thief who is 5 meters ahead and running away at a speed of 0.5 meters per second. How long will it take Xiaoming to catch up with the thief?

[0093] S401. First, DeepSeek, a deep thinking model, analyzes the knowledge points involved in the question and generates a corresponding streamlined video explanation script according to a pre-set explanation script format. It is required that the script incorporate how to use the knowledge points.

[0094] The following is an example of a workflow-based video tutorial script format:

[0095]

[0096]

[0097] Here, animation is an animation script in a clip, background_text is the content that needs to be explained in this clip, and it will be connected to an audio generation model for later dubbing of this video.

[0098] S402. Input the streamlined video tutorial script into the code generation model to generate Manim's script (i.e., animation code). Manim's script is an animation engine developed based on Python, which can be used to generate corresponding videos.

[0099] S403. For the script for Manim, there may be errors in the code. If an error occurs and the explanatory video cannot be obtained successfully, return to S401. If the explanatory video can be obtained successfully, then combine the output of the audio generation model to add voice-over to the explanatory video.

[0100] The training strategy for optimizing the code generation model in the video generation method is as follows:

[0101] S501. Large-scale model analysis of problems and generation of explanation text: Based on a large language model with reasoning capabilities (i.e., an LLM with enhanced reasoning function), after inputting a math problem, the model automatically generates detailed step-by-step explanation text, that is, a process-oriented video explanation script. In layman's terms, this step allows the AI ​​to understand the problem and write out the explanation of each step of the solution process, just like a math teacher.

[0102] S502. Generate Animation Code: The system inputs the above-mentioned flowchart video explanation text into the same or another large language model (i.e., code generation model) to generate the corresponding Manim code (i.e., animation code). In other words, the AI ​​acts as a "programmer," converting the teacher's explanation into animation code that describes how to display each step in the explanation video.

[0103] S503. Compile and render the video: Compile the generated Manim code and render the explanatory video. If the Manim code has errors and cannot be compiled, the failure information is recorded. This stage is equivalent to testing whether the animation code can run correctly and generate the actual video footage.

[0104] S504. Video Segmentation and Keyframe Extraction: The rendered explanation video is segmented into multiple 5-second segments. Two representative keyframe images are extracted from each video segment, and these are combined to form the keyframe set of the explanation video. In short, two keyframe images are extracted every 5 seconds to represent the animation scenes in the video, allowing for automatic checking of the visual effects later.

[0105] S505. Multimodal Visual Detection: Keyframe images are input into a pre-trained image detector model for analysis. This image detector is fine-tuned based on the InternVL-2 model and can binary classify and output whether each keyframe image has out-of-frame or overlap issues (the image quality of the keyframe image). Combining the detection results of all keyframe images, the image quality score of the narration video is calculated, and the video quality reward value of the animation code is determined based on this image quality score. Specifically, the image quality score of the narration video is between 0 and 1. If there are many keyframe images with image problems in the narration video, the image quality score of the narration video is close to 0; if there are few keyframe images with image problems in the narration video, the image quality score of the narration video is close to 1.

[0106] S506. Obtain compilation status signal: Record whether the Manim code was successfully compiled and rendered into an explanatory video, and determine the base reward value of the animation code based on success or failure. In other words, if the animation code successfully generates an explanatory video in S503, record a "success" flag and set the base reward value of the animation code to 1; if the animation code encounters an error and fails to generate an explanatory video, record a "failure" flag and set the base reward value of the animation code to 0.

[0107] S507. Reinforcement Learning Optimization (GRPO): This method uses the base reward value of the animation code and the video quality reward value as feedback for reinforcement learning. A generalized reinforcement policy optimization algorithm is employed to train and optimize the code generation model that generates Manim code. Specifically, these two reward values ​​are introduced into the strategy of the code generation model when generating animation code: if the generated animation code successfully runs to produce an explanatory video with high image quality, the parameters of the code generation model will be adjusted to improve this type of output; conversely, if the animation code fails to produce an explanatory video or the video quality reward value is low, the code generation model will be penalized to correct the generation strategy. In simple terms, the code generation model learns through a "reward and punishment mechanism," rewarding good animation code outputs and penalizing poor animation code outputs, continuously increasing the probability that the next animation code generated by the code generation model will successfully render an explanatory video with keyframe images meeting the set optimal requirements.

[0108] Through the above process, this invention achieves a closed-loop optimization system from automatic problem-solving and explanation to video generation. Its technical advantages are reflected in:

[0109] 1. Improved the success rate of generating explanatory videos using Manim code. After reinforcement learning, the code generation model can more reliably output executable animation code, reducing video generation failures caused by code errors.

[0110] 2. Improved video clarity and layout aesthetics, automatically detected and penalized issues such as out-of-frame and overlapping, ensuring that all elements in the final video are arranged reasonably within the frame, thus enhancing the effectiveness of teaching demonstrations.

[0111] 3. It ensures the accuracy and rigor of the explanation logic. The reasoning ability of the large model used to generate the process-oriented video explanation script makes it pay more attention to the correctness of the problem-solving steps. The video content of the explanation video is more in line with the requirements of the textbook in terms of knowledge point explanation and has a clear hierarchy.

[0112] 4. Significantly reduces the cost of manual editing and repeated debugging. The video generation process is highly automated, reducing human intervention, and even non-professionals can quickly obtain high-quality explanatory videos.

[0113] 5. The system has good generalization ability and adaptability, and can automatically adjust the output for questions of different difficulty and types (e.g., from basic arithmetic to advanced mathematics) to meet the needs of a wide range of educational scenarios.

[0114] In summary, the technical solution of this invention significantly improves the efficiency and quality of existing mathematical video generation, and can provide a practical solution for the field of intelligent education.

[0115] Corresponding to the embodiments of the methods described above, this specification also provides embodiments of the apparatus and the computing devices on which it is applied.

[0116] See Figure 5 , Figure 5 This is a block diagram of a video generation device provided by the present invention, such as... Figure 5 As shown, the device includes:

[0117] The generation module 501 is used to input the streamlined video explanation script, which is output by the large model based on the input information, into the code generation model to obtain N segments of animation code corresponding to the input information; N is greater than 1.

[0118] The determination module 502 is used to compile and render the animation code for each segment. If an explanatory video is successfully obtained, the base reward value of the animation code segment is determined to be the first value. At least two keyframe images are selected from the explanatory video, and the image quality of each keyframe image is obtained. The video quality reward value of the animation code segment is determined based on the image quality of each keyframe image. Based on the base reward value and the video quality reward value of the animation code segment, the relative advantage value of the animation code segment relative to N animation code segments is determined.

[0119] The optimization module 503 is used to determine the objective function based on the relative advantage value of each animation code segment relative to N animation code segments; optimize the code generation model based on the objective function so that the code generation model is adjusted towards the optimal objective; the optimal objective indicates that the animation code generated by the code generation model is successfully rendered to obtain the explanatory video and the picture quality of the key frame images in the explanatory video meets the set optimal requirements.

[0120] As one possible implementation, module 502 is further configured to, if the explanatory video is not successfully obtained, determine that the basic reward value of the animation code segment is a second value, the second value being less than the first value; determine that the video quality reward value of the animation code segment is a specified value; and return the steps of determining the relative advantage value of the animation code segment relative to N animation codes based on the basic reward value of the animation code segment and the video quality reward value of the animation code segment.

[0121] As one possible implementation, module 502 is specifically used to divide the explanatory video into at least two video segments according to a set event and / or a set duration; extract representative images from each video segment, and determine keyframe images based on the extracted images.

[0122] As one possible implementation, the determining module 502 is specifically used to determine, for each video segment, the first frame image and / or the last frame image in the video segment as representative images.

[0123] As one possible implementation, the determining module 502 is specifically used to input the keyframe image into the trained image detector for each keyframe image, and determine the image quality of the keyframe image based on the output of the image detector.

[0124] As one possible implementation, module 502 is specifically used to count the number of keyframe images with the same image problem in the picture quality situation, and determine the score corresponding to the problem based on the number and the total number of keyframe images; and determine the video quality bonus value of the animation code segment based on the score corresponding to each problem and the weight set for each problem.

[0125] As one possible implementation, the image quality indicates whether there are out-of-frame issues and / or overlap issues in the keyframe image; image issues are out-of-frame issues and / or overlap issues.

[0126] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0127] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0128] This specification also provides a computing device, see [link to documentation] Figure 6 , Figure 6 This is a schematic diagram of the structure of a computing device provided by the present invention. Figure 6As shown, the computing device includes a processor 610, a memory 620, and a network interface 630. The memory 620 stores computer instructions that can run on the processor 610. The processor 610 is used to implement a method for generating explanatory videos provided in any embodiment of this specification when executing the computer instructions. The network interface 630 is used to implement input / output functions. In many possible implementations, the computing device may also include other hardware, which is not limited in this specification.

[0129] This specification also provides a computer-readable storage medium, which can take many forms, such as RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (e.g., hard disk drives), solid-state drives, any type of storage disk (e.g., optical discs, DVDs), or similar storage media, or combinations thereof. Specifically, the computer-readable medium can also be paper or other suitable media capable of printing programs. A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements a method for generating explanatory videos provided in any embodiment of this specification.

[0130] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements a method for generating explanatory videos provided in any embodiment of this specification.

[0131] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, apparatus, computing device, computer-readable storage medium, or computer program product. Therefore, one or more embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments corresponding to computing devices are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0133] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of this specification. In some cases, the actions or steps described in this specification may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information and transmit it to a suitable receiving device for execution by a narration video generation apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0135] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0136] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0137] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0138] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0139] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0140] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of this specification. In some cases, the actions described herein may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the accompanying drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0141] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. That is, this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

[0142] The above description is merely an optional embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. A method for generating explanatory videos, characterized in that, The method includes: The streamlined video tutorial script, which is output by the large model based on the input information, is input into the code generation model to obtain N segments of animation code corresponding to the input information; N is greater than 1. For each animation code segment, compile and render the video. If an explanatory video is successfully obtained, determine the base reward value of the animation code segment as the first value. Select at least two keyframe images from the explanatory video and obtain the image quality of each keyframe image. Determine the video quality reward value of the animation code segment based on the image quality of each keyframe image. Based on the base reward value and the video quality reward value of the animation code segment, determine the relative advantage value of the animation code segment relative to the N animation code segments. For each action in the target process corresponding to each animation code segment, the importance sampling ratio of that action in the target process corresponding to that animation code segment is multiplied by the relative advantage value of that animation code segment to obtain the target product of that action in the target process corresponding to that animation code segment. The target process corresponding to that target code segment is the process of inputting a streamlined video explanation script, output by a large model based on input information, into the code generation model to obtain that animation code segment. Based on the target products corresponding to each action in the target process of each animation code segment and the KL divergence constraint corresponding to the code generation model, the objective function is determined according to the relative advantage value of each animation code segment relative to the N animation code segments. The code generation model is optimized according to the objective function to adjust the code generation model toward the optimal objective; the optimal objective indicates that the animation code generated by the code generation model is successfully rendered into an explanatory video and the image quality of the key frame images in the explanatory video meets the set optimal requirements.

2. The method according to claim 1, characterized in that, If the instructional video is not successfully obtained, the method also includes: The base reward value for this animation code segment is determined to be a second value, which is less than the first value. The video quality bonus value of the animation code segment is determined to be a specified value, and the step of determining the relative advantage value of the animation code segment relative to the N animation code segments is returned based on the base bonus value of the animation code segment and the video quality bonus value of the animation code segment.

3. The method according to claim 1, characterized in that, The selection of at least two keyframe images from the explanatory video includes: The explanatory video is divided into at least two video segments according to the set event and / or set duration; Representative images are extracted from each video segment, and keyframe images are determined based on the extracted images.

4. The method according to claim 3, characterized in that, Extracting representative images from each video segment includes: For each video segment, the first frame and / or the last frame of that video segment are used as representative images.

5. The method according to claim 1, characterized in that, The process of obtaining the image quality of each keyframe includes: For each keyframe image, the keyframe image is input into the trained image detector, and the image quality of the keyframe image is determined based on the output of the image detector.

6. The method according to claim 1 or 5, characterized in that, The video quality bonus value for determining the animation code segment based on the image quality of each keyframe image includes: The number of keyframe images showing the same image problem in the image quality statistics is counted, and the score corresponding to the problem is determined based on this number and the total number of keyframe images. Based on the scores corresponding to each question and the weights assigned to each question, the video quality bonus value for this animation code segment is determined.

7. The method according to claim 6, characterized in that, The image quality indicates whether there are out-of-frame issues and / or overlap issues in the keyframe image; The image problems mentioned are out-of-frame problems and / or overlap problems.

8. A device for generating explanatory videos, characterized in that, The device includes: The generation module is used to input the streamlined video explanation script, which is output by the large model based on the input information, into the code generation model to obtain N segments of animation code corresponding to the input information; N is greater than 1. The module is used to compile and render the animation code for each segment. If an explanatory video is successfully obtained, the base reward value of the animation code segment is determined to be the first value. At least two keyframe images are selected from the explanatory video, and the image quality of each keyframe image is obtained. The video quality reward value of the animation code segment is determined based on the image quality of each keyframe image. Based on the base reward value and the video quality reward value of the animation code segment, the relative advantage value of the animation code segment relative to the N animation code segments is determined. The optimization module is used to multiply the importance sampling ratio of each action in the target process corresponding to each animation code segment by the relative advantage value of that animation code segment, to obtain the target product of that action in the target process corresponding to that animation code segment. The target process corresponding to that target code segment is the process of inputting a streamlined video explanation script, output by a large model based on input information, into the code generation model to obtain that animation code segment. Based on the target products corresponding to each action in the target process of each animation code segment and the KL divergence constraint corresponding to the code generation model, the objective function is determined according to the relative advantage value of each animation code segment relative to the N animation code segments. The code generation model is optimized according to the objective function to adjust the code generation model toward the optimal objective; the optimal objective indicates that the animation code generated by the code generation model is successfully rendered into an explanatory video and the image quality of the key frame images in the explanatory video meets the set optimal requirements.

9. A computing device, characterized in that, The computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the explanatory video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that is executed by a processor as described in any one of claims 1 to 7, representing the operations performed by the explanatory video generation method.

Images