Mathematical problem solving video generation method and system based on intermediate state structured script

Abstract

The invention belongs to the technical field of computer-aided education and multimedia content generation, and relates to a mathematical problem solving video generation method and system based on an intermediate-state structured script, and the method comprises the steps that a first system receives mathematical problem data, carries out the semantic analysis and logic disassembly through a large language model, and generates a mathematical problem video; generating an intermediate-state structured script for separately storing visual content and auditory content of the problem solving step; calling a voice synthesis service to generate an audio and obtain a playing time length, dynamically calculating a time sequence parameter of the visual animation based on the playing time length, and realizing audio and picture alignment by inserting a waiting instruction or adjusting a playing rate; and the rendering engine analyzes the script, compiles the visual content into a graphic object, and synthesizes a teaching video in combination with the time sequence parameters and the audio data. By introducing the audio-visual decoupling intermediate state script and a dynamic time sequence calculation mechanism based on the audio duration, the technical problems of difficulty in audio and picture synchronization and high content modification cost in video generation are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided education and multimedia content generation technology, specifically involving a method and system for generating intermediate structured scripts using a large language model and achieving automatic audio-visual synchronization through dynamic timing calculation based on audio duration, thereby generating mathematical problem-solving videos. Background Technology

[0002] With the development of online education, the demand for mathematics teaching videos is increasing. Currently, the main technical challenges in producing mathematics teaching videos are as follows:

[0003] First, traditional manual recording methods are inefficient and costly to modify. If a verbal or written error occurs, the entire video often needs to be re-recorded, making fine-grained local corrections impossible.

[0004] Second, end-to-end AI video generation technology suffers from uncontrollable logic. The model may generate logically flawed derivations, and the output content is difficult to correct.

[0005] Third, audio-visual synchronization is a common challenge in the production of math teaching videos. High-quality math explanations require that the audio content and the formulas written on the screen correspond on the timeline. Current technologies for achieving audio-visual synchronization typically require manual frame-by-frame adjustments on the timeline, a tedious process.

[0006] In summary, there is an urgent need in this field for a video generation technology solution that can automatically achieve audio-visual synchronization and facilitate content modification. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for generating mathematical problem-solving videos based on intermediate-state structured scripts, thereby solving the technical problems of difficulty in synchronizing audio and video and high cost of content modification in the prior art.

[0008] To achieve the above objectives, the core idea of ​​the technical solution proposed in this invention is to introduce an intermediate data layer in the video generation process. This data layer separates and stores the visual presentation content and auditory playback content of each step in the problem-solving process, thereby achieving audiovisual decoupling. Based on this, the timing parameters of the visual animation are dynamically calculated based on the audio playback duration returned by the speech synthesis service, and audio-visual alignment is automatically achieved.

[0009] The technical solution of the present invention includes the following steps:

[0010] First, there's the script generation step. The system receives math problem data, uses structured prompt word engineering technology to call a large language model, performs semantic analysis and logical decomposition of the problem, and generates an intermediate structured script. The core feature of this script is audiovisual decoupling: the data object for each problem-solving step is decomposed into independent visual content fields (mathematical formulas for screen display) and auditory content fields (natural language text for speech synthesis). This decoupling design allows visual and auditory content to be viewed, modified, and processed independently.

[0011] Secondly, there is the timing calculation step. The system parses the intermediate script, iterates through the auditory content fields, calls the speech synthesis service to generate audio data, and obtains the precise playback duration of the audio. Subsequently, the system obtains the base execution time of the animation instruction sequence used to present the corresponding visual content and compares it with the audio duration. If the base animation time is shorter than the audio duration, waiting instructions are automatically inserted into the animation sequence to fill the visual gap; if the base animation time is longer than the audio duration, the animation playback rate is adjusted to compress the runtime. Through this dynamic timing calculation mechanism, automatic alignment between visual animation and audio playback is achieved.

[0012] Finally, there is the rendering and compositing step. The rendering engine compiles the mathematical formulas in the visual content fields into vector graphics objects, binds dynamic effects such as writing, transformation, highlighting, and fade-in / fade-out to them according to the animation timing parameters, generates a video stream frame by frame, and merges it with the audio data along the timeline to output the final video.

[0013] Beneficial effects of the present invention

[0014] First, through the intermediate script design that decouples audio and video, when it is necessary to modify a certain formula or a certain audio segment in the video, only the corresponding field in the script needs to be modified, without having to re-record the entire video, which significantly reduces the cost of content maintenance.

[0015] Second, the dynamic timing calculation mechanism based on audio duration enables automatic audio-visual synchronization, eliminating the tedious work of manually aligning the timeline frame by frame.

[0016] Third, intermediate scripts can serve as an interface for human-computer collaboration, allowing users to verify and modify AI-generated content, thus solving the problem of uncontrollable end-to-end model output. Attached Figure Description Figure 1 This is a flowchart illustrating a mathematical problem-solving video generation method and system based on intermediate-state structured scripts provided in an embodiment of the present invention. Detailed Implementation

[0018] The core of the method proposed in this invention lies in a complete data processing and video generation pipeline. This pipeline begins by receiving mathematical problem data and ends with outputting a synchronized audio-visual teaching video. The specific implementation process will be described in detail below according to each stage of the pipeline.

[0019] Step 1: Script Generation

[0020] The goal of this step is to convert mathematical problems into an intermediate structured script that decouples audiovisual elements.

[0021] [1-1] Data Reception and Preprocessing: The system receives mathematical problem data input by the user through an interface. The problem data can be a text description or a structured data object containing problem information. The system preprocesses the input data to ensure that its format is standardized, preparing it for subsequent calls to the large language model.

[0022] [1-2] Structured prompt word construction: The backend service constructs structured prompt words to convey the task objective to the large language model. The prompt words contain the following core elements: the definition of the target data pattern, which clarifies the hierarchical structure of the output and the meaning of each field; output format constraints, which require the model to strictly output according to the predetermined data structure format; and problem-solving specifications, which guide the model to decompose complex problems into multiple logically clear steps.

[0023] [1-3] Large Language Model Invocation and Script Generation: The system sends the constructed prompts to the large language model service. The model performs semantic understanding, logical analysis, and step decomposition on the questions, and outputs an intermediate structured script that follows a preset pattern.

[0024] [1-4] Script Structure Description: The generated script adopts a hierarchical data structure. The top level contains fields such as question metadata and a step list. Each step object in the step list contains one or more content objects. Each content object contains two core fields: a visual content field, which stores the mathematical formula representation for screen display; and an auditory content field, which stores the natural language narration text for speech synthesis. When generating the auditory content field, the large language model converts mathematical symbols into spoken text that conforms to spoken expression habits, ensuring natural and fluent speech.

[0025] Step Two: Collaborative Editing

[0026] This step is optional and provides the ability to verify and modify content collaboratively with humans.

[0027] [2-1] Script Visualization Mapping: The system provides an interactive interface that maps intermediate scripts into a structured, editable view. The interface displays the hierarchical structure of the script in an intuitive way, allowing users to clearly see each step and its included visual and auditory content.

[0028] [2-2] Independent editing support: Users can independently modify any field. For visual content fields, users can correct errors in mathematical formulas or optimize expressions; for auditory content fields, users can polish the audio text to make it more natural and fluent. Due to the decoupled storage of audio-visual content, modifications to one side will not affect the other, enabling flexible local adjustments.

[0029] [2-3] Modification Confirmation: After the user completes the editing, they confirm the changes. Once confirmed, the script will proceed to the next processing step.

[0030] Step 3: Timing Calculation

[0031] This step, which enables dynamic timing calculation based on audio duration, is the core component of automatic audio-visual synchronization.

[0032] [3-1] Audio Generation and Duration Acquisition: The system iterates through the auditory content fields in the script and calls the speech synthesis service to generate audio data for each piece of speech text. Before the call, an identifier can be generated based on the text content and the cache can be retrieved: if the cache is hit, the stored audio and its duration information are read directly; if the cache is not hit, the remote service is called to generate audio and store it. The caching mechanism can avoid the repeated generation of audio with the same content, thus improving processing efficiency.

[0033] [3-2] Obtaining the basic animation time: For each visual content unit in the script that needs to be animated, the system obtains the basic runtime of its animation instruction sequence. The basic runtime refers to the time required to complete the animation at the default playback speed.

[0034] [3-3] Timing Alignment Calculation: The system compares the basic running time of the animation with the playback duration of the corresponding audio and executes the following logic: If the basic running time is less than the audio playback duration, it means that the animation finishes playing before the audio, resulting in a visual window. In this case, a wait instruction with a duration equal to the difference between the two is inserted at the end of the animation instruction sequence. If the basic running time is greater than or equal to the audio playback duration, it means that the animation is slower than the audio or is just in sync. In this case, a rate factor is calculated and the animation playback rate is adjusted to match the actual running time of the animation with the audio duration. Through this dynamic timing calculation, the visual animation and audio playback of each problem-solving step are precisely aligned on the timeline.

[0035] Step 4: Layout Calculation

[0036] This step is optional and addresses the spatial layout of multiple elements within the image.

[0037] [4-1] Virtual Canvas Initialization: The system maintains a virtual canvas coordinate system, defining the boundaries and usable areas of the screen. The canvas can be divided into different functional areas, such as the question display area, the solution steps area, and the answer area.

[0038] [4-2] Element localization and collision detection: When a new visual content element needs to be rendered, the system calculates its target rendering coordinates. During localization, the system detects whether the element overlaps with already rendered elements based on its bounding box. If overlap is detected, the system calculates the coordinate offset to avoid occlusion and ensures that all content is clearly visible.

[0039] [4-3] Viewport scrolling: When the cumulative height of the content exceeds a preset threshold of the canvas's visible area, the viewport scrolling logic is triggered. The scrolling operation moves the earlier presented content out of the visible area to make room for new content and ensure the rationality of the screen layout.

[0040] Step 5: Rendering and Compositing

[0041] This step converts the data in the script into the final video output.

[0042] [5-1] Graphics object compilation: The rendering engine compiles the mathematical formulas in the visual content fields into vector graphics objects. Vector graphics have good scaling properties, which can ensure display quality at different resolutions.

[0043] [5-2] Dynamic Effect Binding: Bind dynamic effects to graphic objects according to animation timing parameters. Available dynamic effect types include: writing effect, which simulates the handwriting process and presents the content stroke by stroke; transformation effect, which smoothly transitions the source object to the target object; highlight effect, which guides the viewer's attention through visual emphasis; and fade-in / fade-out effect, which controls the gradual change of element transparency to achieve a smooth appearance and disappearance.

[0044] [5-3] Video stream generation and compositing: The rendering engine generates a video stream frame by frame, and simultaneously merges the pre-generated audio data with the video frames along the timeline. The final output is a video file containing synchronized audio and video.

[0045] Through the above steps, this invention achieves the automated generation of teaching videos from mathematical problems. The intermediate structured script, as the core hub, decouples visual and auditory content, facilitating independent content modification; the dynamic timing calculation mechanism based on audio duration solves the audio-visual synchronization problem; and the parameterized rendering process ensures the quality and consistency of the output video.

Claims

1. A method for generating mathematical problem-solving videos based on intermediate-state structured scripts, characterized in that, Includes the following steps: (1) Script generation steps: Receive mathematical problem data, call the large language model to perform semantic analysis and logical decomposition on the mathematical problem data, and generate intermediate structured script; the intermediate structured script decouples the data object of the problem-solving steps into independent visual content fields and auditory content fields; (2) Timing calculation steps: parse the intermediate structured script, extract the auditory content field, call the speech synthesis service to generate audio data, and obtain the playback duration of the audio data; Based on the playback duration, the animation timing parameters of the corresponding visual content field are dynamically calculated to align the duration of the visual animation with the audio playback duration. (3) Rendering and compositing steps: Drive the rendering engine according to the animation timing parameters, compile the visual content fields into graphic objects and bind dynamic effects, and combine the audio data to synthesize video.

2. The method according to claim 1, characterized in that, The intermediate structured script mentioned in step (1) includes a hierarchical data structure, which at least includes: (1-1) A step list field, where each step object contains one or more content objects; (1-2) Each content object contains a visual content field and an auditory content field. The visual content field stores a mathematical formula representation for screen display, and the auditory content field stores natural language text for speech synthesis.

3. The method according to claim 1, characterized in that, The dynamic calculation of the animation timing parameters of the corresponding visual content field in step (2) is specifically as follows: (2-1) Obtain the basic runtime of the animation instruction sequence used to render the visual content field; (2-2) Compare the audio playback duration with the basic running time: if the basic running time is less than the audio playback duration, insert a waiting instruction into the animation instruction sequence; if the basic running time is greater than or equal to the audio playback duration, adjust the playback rate of the animation instruction sequence.

4. The method according to claim 1, characterized in that, It may also include a collaborative editing step: providing an interactive interface to map the intermediate structured script into an editable view; and independently modifying the visual content field or the auditory content field in response to the user's editing instructions.

5. The method according to claim 1, characterized in that, Step (2) also includes a caching mechanism: (2-3) Generate an identifier based on the content of the auditory content field; (2-4) Search for whether there is a cached audio that matches the identifier; (2-5) If a match exists, read the duration information of the cached audio; if no match exists, call the speech synthesis service to generate audio and store it.

6. The method according to claim 1, characterized in that, It also includes layout calculation steps: maintaining a virtual canvas coordinate system and calculating the rendering coordinates of each visual content element in turn; detecting whether there is spatial overlap between elements based on bounding boxes, and calculating the coordinate offset if there is overlap. When the cumulative height of the content exceeds a preset threshold, viewport scrolling is triggered.

7. The method according to claim 1, characterized in that, The rendering and compositing steps in step (3) include: (3-1) Compile the mathematical formula representation in the visual content field into a vector graphic object; (3-2) Bind dynamic effects to the vector graphic object, the dynamic effects including one or more of the following: writing effect, transformation effect, highlighting effect or fade-in / fade-out effect; (3-3) Generate a video stream frame by frame and merge it with the audio data in timeline alignment.

8. The method according to claim 1, characterized in that, When calling the large language model in step (1), structured prompt word engineering technology is used, and the prompt words include the definition of the target data pattern and output format constraints.

9. A mathematical problem-solving video generation system based on intermediate-state structured scripts, characterized in that, include: A script generation module is configured to receive mathematical problem data and call a large language model to generate an intermediate structured script that separates visual and auditory content. A timing calculation module is configured to parse the auditory content field in the intermediate structured script, call the speech synthesis service to generate audio and obtain the playback duration, and calculate the animation timing parameters based on the playback duration; a video rendering module is configured to receive the timing parameters and the audio data, compile the visual content into a graphic object and bind dynamic effects, and synthesize a video aligned with the audio timeline.

10. The system according to claim 9, characterized in that, Also includes: A collaborative editing module is configured to provide an interactive interface that supports independent editing of visual content fields and auditory content fields in the intermediate structured script.

Images