A teaching video script generation method and related device

Abstract

The application discloses a teaching video script generation method and related device, and relates to the technical field of intelligent education, comprising: obtaining a target test question image of a target subject; performing text extraction and image standardization processing on the target test question image respectively to obtain structured text and a standardized test question image; based on a target multi-modal large model adapted to the target subject, performing cross-modal feature alignment processing on the structured text and the standardized test question image, building the association between the test question text semantics and the image visual elements, and generating a teaching video script fused with visual semantics according to the association, wherein the explanation steps in the teaching video script refer to the related visual elements in the standardized test question image as reasoning basis. The application realizes reasoning type explanation combining text and images, so that students can intuitively correspond the reasoning steps to specific graphic areas in the image when watching the teaching video, and significantly improves the intuitiveness and understandability of teaching explanation.

Description

Technical Field

[0001] This invention relates to the field of intelligent education technology, and in particular to a method and related apparatus for generating teaching video scripts. Background Technology

[0002] With the deepening application of artificial intelligence technology in education, automatically generating teaching video scripts using AI has become an important direction for improving the efficiency of teaching resource production. In the teaching of certain subjects (such as mathematics), test questions often contain a large number of visual elements, such as the vertices, sides, angles, and auxiliary line labels in geometric figures, the data distribution in statistical charts, the curve features in function graphs, and the row and column correspondences in tabular data. According to educational industry survey data, multi-model test questions currently account for 68% of primary and junior high school mathematics tests, of which visually dependent questions such as geometric proofs, graphical calculations, and tabular data analysis account for over 90%.

[0003] Currently, mainstream teaching video script generation solutions typically adopt a two-tier architecture of "OCR text extraction + general large language model generation". The basic process is as follows: first, OCR tools are used to detect and recognize text in the test question image to extract the plain text content; then, the extracted text is input into a general large language model to generate the corresponding teaching video script according to a preset template.

[0004] While the above solution achieves a certain degree of automated conversion from test question images to teaching video scripts, the generated teaching video scripts are completely detached from the visual information in the test question images, resulting in a lack of intuitive basis for the explanation process. Summary of the Invention

[0005] In view of this, the present invention provides a method and related apparatus for generating teaching video scripts, to solve the problem that existing teaching video script generation schemes generate teaching video scripts that are completely detached from the visual information in the test question images, resulting in a lack of intuitive basis for the explanation process. The technical solution is as follows:

[0006] The first aspect of this invention provides a method for generating instructional video scripts, comprising:

[0007] Obtain the target test question image for the target subject, wherein the target test question image is an image of a test question containing visual elements;

[0008] Text extraction and image standardization are performed on the target test question images to obtain structured text and standardized test question images;

[0009] Based on the target multimodal large model, cross-modal feature alignment processing is performed on the structured text and the standardized test question image to establish the association between the semantics of the test question text and the visual elements of the image. Based on the association, a teaching video script that integrates visual semantics is generated. The explanation steps in the teaching video script refer to the relevant visual elements in the standardized test question image as the basis for reasoning.

[0010] The target multimodal large model is obtained by fine-tuning the basic multimodal large model using the training sample set of the target subject. Each training sample in the training sample set includes a test question image sample, the structured text corresponding to the test question image sample, and the step-by-step explanation text. The explanation steps in the step-by-step explanation text refer to the visual elements in the test question image sample as the basis for reasoning.

[0011] In one possible implementation, the instructional video script includes interactive guidance instructions set during the explanation process, which are used to guide the user to perform interactive operations related to solving the standardized test question images.

[0012] In one possible implementation, the process of obtaining the training sample set includes:

[0013] Obtain an initial training sample set, wherein each training sample in the initial training sample set includes a test question image sample of the target subject and a step-by-step explanation text corresponding to the test question image sample;

[0014] The initial training sample set is processed to obtain a processed training sample set. Each training sample in the processed training sample set includes a test question image sample and the corresponding structured text and step-by-step explanation text. The structured text includes the question text and visual element information.

[0015] Difficult samples are selected from the processed training sample set, and labels are added to the selected difficult samples to characterize their features;

[0016] The selected difficult samples are resampled, and the resampled difficult samples are added to the processed training sample set to obtain the final training sample set.

[0017] In one possible implementation, the label of each difficult sample includes some or all of the following labels: visual interference level label, difficulty coefficient label, and core visual element label;

[0018] The visual interference level label is used to characterize the intensity of visual interference in the difficult sample, the difficulty coefficient label is used to characterize the difficulty of the difficult sample, and the core visual element label is used to indicate the key visual information in the difficult sample that affects the solution of the problem.

[0019] In one possible implementation, the step of filtering difficult samples from the processed training sample set includes:

[0020] From the processed training sample set, one or more of the following types of samples are selected as difficult samples: image-text overlap samples, handwritten samples, complex formula samples, and multi-image linkage samples.

[0021] Among them, the text-image overlap type sample is a sample in the test question image where the area of ​​text covering the graphic exceeds a preset area threshold; the handwritten character type sample is a sample in the test question image where the handwritten character recognition accuracy is lower than a preset accuracy threshold; the complex formula type sample is a sample in the test question image where the number of nested mathematical formulas exceeds a preset number threshold; and the multi-image linkage type sample is a sample in the test question image where the number of associated step images exceeds a preset number threshold.

[0022] In one possible implementation, the selected difficult samples include multiple categories of difficult samples;

[0023] The resampling process for the selected difficult samples includes:

[0024] Obtain the frequency of occurrence of various difficult samples in actual teaching;

[0025] Based on the obtained frequency of occurrence, differentiated resampling ratios are assigned to various types of difficult samples;

[0026] Based on the resampling ratio assigned to each type of difficult sample, resampling is performed on each type of difficult sample.

[0027] In one possible implementation, the basic multimodal large model is fine-tuned using a training sample set of the target discipline to obtain the target multimodal large model, including:

[0028] Freeze the parameters of the visual encoder and the base layer of the language model in the basic multimodal large model;

[0029] Using the training sample set of the target discipline, the parameters of the cross-modal alignment layer of the basic multimodal large model and the top layer of the language model are finely adjusted to obtain the target multimodal large model.

[0030] In one possible implementation, the basic multimodal large model is fine-tuned using a training sample set of the target discipline to obtain the target multimodal large model, including:

[0031] Obtain a prompt instruction template configured for the target subject, wherein the prompt instruction template contains visual association constraint information;

[0032] For each training sample in the training sample set, the structured text in the training sample is combined with the prompt instruction template to obtain a script-generated prompt instruction. The script-generated prompt instruction and the test question image samples in the training sample are used to fine-tune the basic multimodal large model.

[0033] The visual association constraint information is used to force the basic multimodal large model to reference visual elements in the test question image samples when generating explanation content.

[0034] In one possible implementation, the prompt instruction template includes: an explanation of requirements, a question element section, and an output format section;

[0035] The explanation requirement section is used to carry the explanation requirements of the basic multimodal large model, the question element section is used to contain the structured text in the training samples, and the output format section is used to standardize the output structure of the explanation content generated by the basic multimodal large model. The output structure includes some or all of the following modules: graphical observation module, known condition extraction module, problem-solving steps module, interactive guidance module, and knowledge expansion module.

[0036] In one possible implementation, the training sample set covers multiple different educational stages;

[0037] Each training sample in the training sample set has a learning segment label, which is used to identify the learning segment to which the corresponding training sample belongs, so that the target multimodal large model can adapt to the teaching style of the corresponding learning segment when generating teaching video scripts.

[0038] In one possible implementation, the fine-tuning employs a joint loss function to optimize the parameters of the underlying multimodal large model;

[0039] The joint loss function includes a text semantic loss term and a visual association loss term;

[0040] The text semantic loss term is used to measure the semantic difference between the teaching video script generated by the basic multimodal large model and the corresponding step-by-step explanation text in the training sample set, and the visual association loss term is used to indicate whether the core visual elements in the test question image sample are mentioned in the teaching video script generated by the basic multimodal large model.

[0041] In one possible implementation, the step of performing cross-modal feature alignment processing on the structured text and the standardized test question image to establish a correlation between the semantics of the test question text and the visual elements of the image, and generating a teaching video script that integrates visual semantics based on the correlation, includes:

[0042] Extract the textual semantic features of the structured text and the image visual features of the standardized test question image, respectively;

[0043] Cross-modal feature alignment processing is performed on the text semantic features and the image visual features to establish an association mapping between the concepts related to the target subject in the structured text and the corresponding visual elements in the standardized test question image, thereby obtaining joint features that integrate text semantics and visual features;

[0044] Based on the aforementioned joint features, an interactive teaching video script is generated according to a preset script structure.

[0045] A second aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the electronic device implements any of the above-described methods for generating instructional video scripts.

[0048] A third aspect of this application provides a computer-readable storage medium storing computer instructions for causing an electronic device to implement any of the above-described methods for generating instructional video scripts.

[0049] By utilizing the aforementioned technical solution, the teaching video script generation method provided by this invention performs cross-modal feature alignment processing on structured text and standardized test question images by calling a target multimodal large model. This establishes a correlation between the semantics of the test question text and the visual elements of the image, and generates a teaching video script based on this correlation. This allows the explanation steps in the script to reference relevant visual elements in the standardized test question images as reasoning basis, achieving a reasoning-based explanation that combines text and images. Students can intuitively correspond the reasoning steps with specific graphic regions in the image when watching the teaching video, significantly improving the intuitiveness and comprehensibility of the teaching explanation. Simultaneously, during the training phase, this invention uses training samples containing test question image samples, structured text, and step-by-step explanation text to fine-tune the basic multimodal large model. Furthermore, the explanation steps in the step-by-step explanation text reference visual elements in the test question image samples as reasoning basis, enabling the model to establish a strong correlation supervision signal between visual evidence and reasoning steps from the training phase. The consistency of the visual correlation mechanism between the training and reasoning phases ensures that the model can stably generate explanation scripts referencing visual elements during the reasoning phase, fundamentally solving the problem of the disconnect between explanation and visual information in existing technologies. Furthermore, this invention uses a training sample set specific to the target discipline to fine-tune the basic multimodal large model, enabling the model to learn the terminology system, reasoning logic, and explanation style unique to the target discipline. This significantly improves the performance of the generated script in terms of terminology standardization and teaching logic coherence, making the output teaching video scripts more in line with the actual teaching needs of the target discipline. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the teaching video script generation method provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the overall scheme for generating teaching video scripts provided in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram illustrating the process of obtaining a training dataset for a target subject according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of the teaching video script generation device provided in an embodiment of the present invention. Detailed Implementation

[0055] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0056] The embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] In the process of developing this case, the inventors discovered that current teaching video script generation solutions based on "OCR text extraction + large language model generation" have many shortcomings:

[0059] Firstly, visual semantic information is completely discarded during the OCR extraction process.

[0060] Conventional OCR tools can only extract text information from images and cannot analyze spatial relationships, formatting information, and special symbols in test questions. For example, key visual information such as the positional relationships of vertices in geometric figures, the parallel or perpendicular relationships of edges, and the way auxiliary lines are added cannot be recognized and recorded; the correspondence between rows and columns in tabular data and the positional information of data in the table are lost during the textification process; information containing pedagogical value, such as correction marks and writing order in handwritten solutions, is ignored; and special symbols (such as "∽" (similar), "⊥" (perpendicular), and "⊙" (circle)) are easily misidentified or lost. Taking a middle school math test as an example, the question is: "As shown in the figure, in right triangle ABC, ∠C=90°, BC=4cm, AC=3cm, find the length of the altitude CD on the hypotenuse AB." The existing solution extracts only the text "In right triangle ABC, ∠C=90°, BC=4cm, AC=3cm, find the length of the altitude CD on the hypotenuse AB," while the core visual information "CD is the altitude on side AB" (the position of CD, the vertical symbol, the dotted line markings, etc.) in the image is completely discarded. The explanation content in the final generated teaching video script is only based on the Pythagorean theorem calculation and the derivation of the area method based on pure text. It cannot explain why the area method is chosen to find the altitude, which line segment CD corresponds to in the figure, and other reasoning logic that relies on visual information, making it difficult for students to understand the origin of the solution.

[0061] Secondly, the generated teaching video scripts lack visual connections and cannot achieve teaching reasoning that combines text and images.

[0062] Since the input to the large language model is only a plain text sequence, there is no information related to visual elements in the model. Therefore, the generated teaching video script cannot use the visual elements in the test question image as the basis for reasoning. This leads to the following problems: (1) For pure graphic test questions (such as picture recognition questions that only contain geometric figures and have no text descriptions), OCR cannot extract any effective information, and the large language model cannot generate teaching video scripts at all; (2) For test questions that combine text and graphics, the generated teaching video script lacks reference to key visual information in the graphics, and the problem-solving steps lack intuitive basis, making it difficult to achieve the teaching effect of combining text and graphics; (3) The teaching video script cannot contain visual teaching guidance such as "observe the right angle symbol of ∠C in the figure" or "refer to the label position of CD in the figure".

[0063] Third, there is a lack of specific adaptations for mathematics teaching scenarios.

[0064] The general language model has not been specifically trained in the teaching scenarios of target subjects such as mathematics. The scripts generated have the following deficiencies in teaching content: (1) mismatch of learning stages, such as using high school trigonometric function knowledge to explain junior high school geometry problems, or using terms that are beyond the students' cognitive scope; (2) skipping steps, such as omitting basic explanation links such as "the applicable conditions of the Pythagorean theorem" and "the derivation of the triangle area formula", which does not conform to the principle of gradual teaching; (3) non-standard terminology, such as expressing "hypotenuse" as "the longest side" and "perpendicular" as "vertical"; (4) lack of teaching interaction, such as the lack of interactive nodes in the script that conform to the teaching rules, such as observing the characteristics of the figure, marking the known conditions, and verifying the calculation results.

[0065] To address the shortcomings of existing solutions, the inventors of this invention conducted research and, through continuous investigation, ultimately proposed a method for generating instructional video scripts that overcomes the deficiencies of existing solutions. The following embodiments will further illustrate the instructional video script generation method provided by this invention.

[0066] Please see Figure 1 The diagram illustrates a flowchart of a teaching video script generation method provided by an embodiment of the present invention. This teaching video script generation method may include:

[0067] Step S101: Obtain the target test question image for the target subject.

[0068] The target subject can be subjects such as mathematics, physics, and chemistry that include graphical questions. The target question image is an image of a graphical question. Graphical questions refer to questions that contain visual elements such as geometric figures, statistical charts, function graphs, tabular data, or handwritten formulas.

[0069] Images of target test questions can be obtained through methods such as taking photos, scanning, screenshotting, or uploading from local storage. For example, a student can take a photo of a geometry problem on a test paper using a learning device to obtain an image of the target test question. Obtaining these images provides the data foundation for subsequent text extraction and image standardization processing.

[0070] Step S102: Perform text extraction and image standardization on the target test question image to obtain structured text and standardized test question image.

[0071] Text extraction is performed on the target test question image, which involves recognizing and extracting text information from the image. This embodiment preferably uses a dedicated OCR tool for the target subject (e.g., mathematics) to perform text detection and recognition on the target test question image. This OCR tool is specifically optimized for common symbols, formula structures, and handwritten characters in the target subject. After extraction, format correction is required, such as subscript semantic restoration (e.g., correcting "x2" to "x²"), special symbol correction (e.g., correcting "┬" to "⊥"), and standardizing handwritten characters (e.g., adjusting cursive mathematical symbols to standard symbols). Simultaneously, text extraction also extracts visual element information from the target test question image, such as vertex coordinates, side length numerical labels, and auxiliary line markings. This information is recorded in a structured form and, together with the question text, constitutes structured text. The structured text contains both the question text content in the image and information about visual elements in the image (e.g., coordinates, attributes), providing a text-side data foundation for subsequent cross-modal alignment.

[0072] Image standardization of the target test image refers to the unified format conversion and image quality optimization of the target test image. Image standardization may include some or all of the following processes: (1) Channel conversion: converting the 4-channel RGBA image to a 3-channel RGB image to avoid the transparency channel interfering with the model's recognition of graphic boundaries; (2) Size normalization: uniformly scaling to a preset size (e.g., 512×512 pixels) while maintaining the aspect ratio to avoid graphic distortion caused by scaling; (3) Blur enhancement: using Gaussian filtering to enhance the edge texture of low-quality images with a sharpness lower than a preset threshold (e.g., 0.4) to improve the recognizability of symbols and graphic boundaries. Through the above standardization processes, a standardized test image is obtained, providing a standardized image input for subsequent visual coding.

[0073] Step S103: Based on the target multimodal large model, perform cross-modal feature alignment processing on the structured text and standardized test question images, establish the association between the semantics of the test question text and the visual elements of the image, and generate a teaching video script that integrates visual semantics based on the association.

[0074] Among them, the target multimodal large model refers to a pre-trained and fine-tuned visual-language large model that can simultaneously receive text and image inputs and achieve deep fusion of visual features and text features through a cross-modal attention mechanism.

[0075] In this embodiment, as Figure 2As shown, the target multimodal large model is obtained by fine-tuning the basic multimodal large model (such as Qwen2.5-VL) using a training sample set of the target subject (such as mathematics). The training sample set contains multiple training samples, each including a test question image sample, the corresponding structured text (question text and visual element information), and step-by-step explanation text. The explanation steps in the step-by-step explanation text reference visual elements from the test question image sample as the basis for reasoning. Through the design of the above training data, the model learns a generation strategy that integrates image visual elements into the explanation steps during the fine-tuning process.

[0076] Preferably, the training sample set covers multiple different educational stages. For example, the training sample set covers primary, junior high, and senior high school, meaning it simultaneously contains training samples from all three stages. Each training sample in the training sample set has a grade level label, which identifies the grade level to which the corresponding training sample belongs. Specifically, during the data construction phase, each training sample is labeled with a grade level label, such as "primary," "junior high," or "senior high." These grade level labels serve as metadata for the training samples and are learned by the model during fine-tuning, enabling the model to associate specific explanation styles, terminology depth, and reasoning complexity with the corresponding grade level. For instance, when a junior high school geometry problem is input, the model will adopt a junior high school math explanation style based on the grade level label, including using core knowledge points such as the Pythagorean theorem and the triangle area formula, and employing terminology and expressions suitable for junior high school students' cognitive level. When a primary school arithmetic problem is input, the model will use more conversational language and more detailed step-by-step demonstrations. When a senior high school function problem is input, the model will introduce more abstract concepts and more rigorous logical derivations. By training a sample set that covers multiple learning stages and labeling them with stage tags, the model can adaptively adjust its explanation style according to the learning stage to which the test questions belong, thus achieving precise adaptation of explanation styles for different learning stages.

[0077] After fine-tuning to obtain a target multimodal large model adapted to the target discipline, such as Figure 2 As shown, cross-modal feature alignment processing can be performed on structured text and standardized test question images based on the target multimodal large model, establishing the association between the semantics of the test question text and the visual elements of the image, and generating a teaching video script that integrates visual semantics based on the association. The explanation steps in the teaching video script refer to the relevant visual elements in the standardized test question images as the basis for reasoning.

[0078] For example, the explanation content in the teaching video script is "Step 1: Observation" Figure 1Identify the right-angle vertex of right triangle ABC as C (marked '∠C=90°' in the diagram), and extract the known conditions AC=3cm and BC=4cm (side labeling information). Step 2: Calculate the length of the hypotenuse AB using the Pythagorean theorem: AB²=AC²+BC²=3²+4²=25, therefore AB=5cm. Step 3: Use the area method to find the height CD. From the position of CD in the diagram, we know that it is the height on the hypotenuse AB. …” The explanation steps in the above example use specific visual elements in the question image as a basis.

[0079] This embodiment implements a reasoning-based explanation that combines text and images, enabling students to intuitively match the reasoning steps with specific graphic areas in the images when watching the instructional videos, significantly improving the intuitiveness and comprehensibility of the instruction.

[0080] Preferably, the teaching video script generated based on the target multimodal large model can include interactive guidance instructions set during the explanation process. These instructions guide users to perform problem-solving-related interactive operations on standardized test question images. For example, after calculating the length of hypotenuse AB using the Pythagorean theorem and before proceeding to the area method for finding the altitude, the following interactive guidance instruction can be embedded: "Now, please pause and draw this right triangle in your exercise book, marking each vertex and the known side lengths. Then, try to verify the length of CD using another method. Think about how CD will change if AC becomes longer?" Similarly, after explaining the area method for finding the altitude, the interactive instruction "Please mark the positions of hypotenuse AB and altitude CD in the diagram" can be embedded. These interactive guidance instructions trigger a pause during video playback, guiding students to perform marking and verification operations on the test question images or exercise books, thus realizing a teaching process that transitions from one-way explanation to two-way interaction.

[0081] The teaching video script generation method provided in this invention performs cross-modal feature alignment processing on structured text and standardized test question images by calling a target multimodal large model. This establishes a correlation between the semantics of the test question text and the visual elements of the image, and generates a teaching video script based on this correlation. This allows the explanation steps in the script to reference relevant visual elements in the standardized test question image as the basis for reasoning, achieving a reasoning-based explanation that combines text and images. Compared to existing OCR solutions that only extract text, resulting in the complete loss of visual information, this embodiment enables students to intuitively match the reasoning steps with specific graphic areas in the image when watching the teaching video, significantly improving the intuitiveness and comprehensibility of the teaching explanation.

[0082] Meanwhile, in the training phase, this embodiment of the invention uses training samples containing test question image samples, structured text, and step-by-step explanation text to fine-tune the basic multimodal large model. Furthermore, the explanation steps in the step-by-step explanation text reference visual elements in the test question image samples as the basis for reasoning. This enables the model to establish a strong correlation supervision signal between visual evidence and reasoning steps from the training phase. The consistency of the visual correlation mechanism between the training and reasoning phases ensures that the model can stably generate explanation scripts that reference visual elements during the reasoning phase, fundamentally solving the problem of the disconnect between explanation and graphics in the prior art.

[0083] Furthermore, this embodiment of the invention uses a training sample set of the target subject to fine-tune the basic multimodal large model, enabling the model to learn the terminology system, reasoning logic, and explanation style unique to the target subject. This significantly improves the performance of the generated script in terms of terminology standardization and teaching logic coherence, making the output teaching video scripts more in line with actual teaching needs. This embodiment of the invention covers multiple learning stages by using a training sample set labeled with learning stage tags, allowing the model to adaptively adjust its explanation style according to the learning stage to which the test questions belong. This achieves precise adaptation of the explanation style of mathematical knowledge across different learning stages. This embodiment also upgrades the explanation of test questions from one-way instruction to interactive teaching by setting interactive guidance instructions in the script, guiding students to actively participate in the reasoning process and further improving teaching effectiveness.

[0084] As mentioned in the above embodiments, the target multimodal large model is trained using the training dataset of the target discipline. In some embodiments of this application, the process of obtaining the training dataset of the target discipline is described.

[0085] Please see Figure 3 The diagram illustrates the process of obtaining a training dataset for a target subject, which may include:

[0086] Step S301: Obtain the initial training sample set.

[0087] Each training sample in the initial training sample set includes a test question image sample for the target subject and corresponding step-by-step explanation text. It may also include the answers corresponding to the test question image sample. The step-by-step explanation text in the training sample can serve as a teacher's explanation template. It should be noted that the initial training sample set may include image samples of single-model test questions (pure text questions, pure graphic questions) and multi-model test questions (combining text and graphics), or it may only include image samples of multi-model test questions.

[0088] Preferably, the initial training sample set covers multiple different educational stages (such as primary school, junior high school, and senior high school) and multiple different subject modules (such as geometry, algebra, and statistics). Of course, the initial training sample set may also only cover a specific educational stage.

[0089] Step S302: Process the initial training sample set to obtain the processed training sample set.

[0090] Each training sample in the processed training sample set may include: a test question image sample and the corresponding structured text and step-by-step explanation text, wherein the structured text may include the question text and visual element information.

[0091] The processing of the initial training sample set includes some or all of the following:

[0092] (1) Data preprocessing.

[0093] The target subject-specific OCR tool was used to perform text detection and recognition on each test question image sample in the initial training sample set. Special processing was applied to common symbols, formula structures, and handwritten characters in the target subject. The OCR extraction results were formatted, including but not limited to: subscript semantic restoration (e.g., correcting "x2" to "x²"), special symbol correction (e.g., correcting "┬" to "⊥"), and handwritten character standardization (e.g., retaining cursive symbols as standard symbols).

[0094] (2) Data cleaning.

[0095] The data cleaning process can include three levels: The first level is deduplication, which uses a dual comparison of text cosine similarity (threshold 0.95) and the dHash hash value (Hamming distance ≤ 2) of the test question image sample images to remove duplicate test questions; the second level is noise reduction, which corrects deep OCR errors, removes low-quality images with a resolution below a preset resolution threshold, and removes out-of-syllabus test questions based on the subject knowledge point database; the third level is format unification, which processes all data into a unified structured format. Each data entry can include a test question image sample, the corresponding structured text (including the question text and visual element information), and step-by-step explanation text. It can also include information such as grade level tags, subject modules, and interactive guidance nodes, as shown below:

[0096] <Problem Elements>{Text: [“In right triangle ABC, ∠C = 90°, find the altitude CD on hypotenuse AB”], Image: [Image path: / data / geo_001.jpg; Visual element information: {Vertex A (80, 120), Vertex B (280, 120), Vertex C (80, 220), Side BC labeled “4cm”, Side AC labeled “3cm”, CD is the dashed altitude on side AB}]

[0097] <Explanation Steps>

[0098] Step 1: Observation Figure 1Identify the right-angle vertex of Rt△ABC as C (marked "∠C=90°" in the figure), and extract the known conditions AC=3cm and BC=4cm (side labeling information).

[0099] Step 2: Calculate the length of the hypotenuse AB using the Pythagorean theorem: AB² = AC² + BC² = 3² + 4² = 25, therefore AB = 5cm;

[0100] Step 3: Use the area method to find the height CD. The area of ​​the triangle S = 1 / 2 × AC × BC = 1 / 2 × 3 × 4 = 6 cm².

[0101] Step 4: Substitute the data into S = 1 / 2 × AB × CD to get 6 = 1 / 2 × 5 × CD, and solve for CD = 12 / 5 = 2.4 cm. + <School Segment Tag> [Junior High School Mathematics] + <Subject Module> [Geometry] + <Interactive Guidance Node> ["Pause: Please mark the hypotenuse AB and height CD in the figure"].

[0102] (3) Image preprocessing.

[0103] Convert the 4-channel RGBA test image samples into 3-channel RGB images; uniformly scale the test image samples to a preset size (e.g., 512×512 pixels) while maintaining the aspect ratio; apply Gaussian filtering (e.g., Gaussian filtering with a radius of 1.5) to blur test image samples with a sharpness lower than the preset sharpness threshold.

[0104] (4) Data augmentation. Augmentation strategies were designed based on the characteristics of the test questions in the target subject: For graphic questions, rotation and horizontal flipping were performed at preset angles (e.g., ±15°) to simulate different shooting angles; for questions containing formulas, equivalent expression replacements were performed (e.g., replacing "x²" with "x squared"); and slight ink noise was added to handwritten questions to simulate real writing scenarios. The amount of data after augmentation was increased to a preset multiple (e.g., 3 times) of the original data.

[0105] Step S303: Select difficult samples from the processed training sample set and add labels to the selected difficult samples to characterize the sample characteristics.

[0106] After obtaining the processed training sample set, difficult samples are selected from the processed training sample set. Specifically, one or more of the following categories of samples can be selected as difficult samples from the processed training sample set: image-text overlap samples, handwritten samples, complex formula samples, and multi-image linkage samples.

[0107] Image-text overlap samples are those in which the text in a test question image covers an area exceeding a preset area threshold. For example, if the side length labels in a rectangle area calculation problem fill the entire interior of the image, and the text covers an area exceeding a preset area threshold (e.g., 40%), it is classified as a difficult image-text overlap sample. The core challenge of this type of sample is that the text occludes key elements of the image, preventing the model from recognizing geometric relationships.

[0108] Handwritten samples are those in the test question image samples where the accuracy of handwritten character recognition is lower than a preset accuracy threshold. For example, if a student's handwritten "2x+3y=10" contains cursive writing and correction marks, and the OCR recognition accuracy is below 70%, it is judged as a difficult handwritten sample. These samples closely resemble students' real-life writing scenarios, and the difficulty lies in the irregularity of handwritten characters, which makes recognition difficult.

[0109] Complex formula samples are those in the test question image samples where the number of nested mathematical formulas exceeds a preset threshold. For example, a formula with more than three nested levels is classified as a complex formula sample. The difficulty of these samples lies in the complexity of the formula structure and the large number of nested symbols, which can easily lead to the model misinterpreting the formula's semantics.

[0110] Multi-image linked samples are those in the test question image sample where the number of associated step images exceeds a preset threshold. For example, when a geometry proof question contains three step images: "before adding auxiliary lines - after adding them - proof complete," it is judged as a difficult multi-image linked sample. The difficulty of this type of sample lies in the need for cross-image linked reasoning, and existing technologies cannot handle the visual associations and logical reasoning between multiple images.

[0111] The selection of the above four types of difficult samples was not random, but based on teaching practice. It identified four core scenarios that students frequently encounter when solving problems and that existing models are most prone to errors, thus achieving the goal of selecting samples that are adapted to teaching needs.

[0112] After identifying the difficult samples, a label is added to each selected difficult sample to characterize its properties. The label added to each difficult sample may include, but is not limited to, some or all of the following labels: visual interference level label, difficulty coefficient label, and core visual element label.

[0113] Visual interference level labels are used to characterize the intensity of visual interference in difficult samples. For example, for difficult samples with overlapping text and images, where the text covers more than 40% of the graphic area and severely obscures key graphic elements, the visual interference level is labeled as level 4 (out of 5) to inform the model that the sample has strong visual interference and that the visual feature extraction strategy needs to be optimized accordingly.

[0114] Difficulty coefficient labels are used to characterize the difficulty of difficult samples. For example, for complex formula questions with more than three levels of nesting, the difficulty coefficient is labeled as 0.9 (range 0.1-1.0, 1.0 being the most difficult), to adapt to the model fine-tuning needs of different educational levels and avoid the model using a uniform difficulty logic to handle difficult samples of different educational levels.

[0115] Core visual element labels are used to indicate key visual information in difficult samples that affects problem-solving. For example, for difficult samples involving multiple related images, the label is "core element is auxiliary line DE"; for difficult samples involving overlapping images and text, the label is "core element is the occluded rectangle vertex". This allows the model to clearly know the visual areas that need to be focused on, avoiding blindly learning irrelevant features.

[0116] This embodiment provides multi-dimensional feature guidance for difficult samples through a joint labeling system consisting of visual interference level labels, difficulty coefficient labels, and core visual element labels. This enables the model to differentiate and focus on the key features of different difficult samples during training, thereby improving its ability to handle complex multi-mode test questions.

[0117] Step S304: Resample the selected difficult samples and add the resampled difficult samples to the processed training sample set to obtain the final training sample set.

[0118] The selected difficult samples include multiple types of difficult samples (such as text and image overlap samples, handwritten samples, complex formula samples, and multi-image linkage samples). Given that the frequency of occurrence of various types of difficult samples in actual teaching is different, this embodiment adopts a stratified resampling strategy for resampling processing.

[0119] First, we need to determine the frequency of various challenging questions in actual teaching. For example, according to teaching practice statistics, handwritten questions appear relatively frequently, while questions involving multiple related images appear relatively infrequently.

[0120] Then, based on the frequency of occurrence of various types of difficult samples in actual teaching, differentiated resampling ratios are assigned to each type of difficult sample. Specifically, a higher resampling ratio is assigned to difficult samples with higher frequency of occurrence, and a lower resampling ratio is assigned to difficult samples with lower frequency of occurrence. For example, a 12% resampling ratio is assigned to handwritten samples, an 8% resampling ratio is assigned to images and text overlapping samples, a 5% resampling ratio is assigned to complex formula samples, and a 5% resampling ratio is assigned to multi-image linked samples.

[0121] Finally, based on the resampling ratio assigned to each type of difficult sample, the difficult samples are resampled.

[0122] According to the resampling ratio allocated to each type of difficult sample, each type of difficult sample is copied, and the copied samples are added to the processed training sample set, thus increasing the proportion of difficult samples in the training sample set.

[0123] By employing the aforementioned hierarchical resampling strategy, we can avoid over-sampling of single-type difficult samples that could lead to model overfitting, while ensuring that all types of difficult samples receive training weights that are appropriate for their actual teaching needs. This allows the model to focus on high-frequency difficult samples during training, strengthen feature learning for various difficult scenarios, and significantly improve its ability to handle complex test questions.

[0124] This embodiment overcomes the core limitations of existing technologies, such as lack of targeted screening, lack of dedicated annotation, and insufficient weighting of difficult samples, through a three-stage linkage process of "screening-labeling-resampling," and precisely enhances the model's ability to process complex multi-mode test questions.

[0125] After obtaining the training dataset for the target subject, the basic multimodal large model can be fine-tuned using the training dataset to obtain a target multimodal large model adapted to the target subject.

[0126] In one possible implementation, when fine-tuning the basic multimodal large model using the training dataset of the target subject, all parameters of the visual encoder and the basic layer of the language model of the basic multimodal large model can be frozen, and then only the parameters of the cross-modal alignment layer of the basic multimodal large model and the top layer of the language model can be fine-tuned using the training dataset of the target subject.

[0127] Optionally, the basic multimodal large model can be, but is not limited to, the Qwen2.5-VL series model, which includes a visual encoder and a language model (Transformer decoder structure). Freezing all parameters of the visual encoder means that the parameters of the visual encoder are not updated during fine-tuning, thus preserving its general visual feature extraction capabilities learned during the pre-training phase.

[0128] It should be noted that the language model base layer refers to the layer closest to the input in the Transformer decoder architecture. It is mainly responsible for extracting general syntactic and semantic features. These features have good generalization ability across tasks. Freezing the parameters of the language model base layer can effectively prevent the model from overfitting on specific task data and significantly reduce the computational cost of fine-tuning, enabling the model to maintain general language comprehension ability while learning target subject knowledge.

[0129] The top layer of the language model refers to the last few layers in the Transformer decoder architecture closest to the output, responsible for transforming abstract semantic representations into the final output text. During fine-tuning, only the parameters of the top layer of the language model are updated to adapt to the terminology, reasoning logic, and presentation style of the target subject. Optionally, the top layer of the language model can be, but is not limited to, the last two Transformer layers.

[0130] The technical principle behind the aforementioned fine-tuning method lies in: directly reusing the built-in visual encoder capabilities of the basic multimodal large model, eliminating the need to construct an additional dual-encoder architecture; and directly reusing the model's built-in alignment layer in the feature fusion stage, eliminating the need for custom projection layers and normalization processing, and enhancing the alignment effect solely through training data. By freezing most parameters and only fine-tuning a small number of top-level parameters, the method ensures the adaptation effect to the target discipline while minimizing computational costs.

[0131] Building upon the lightweight fine-tuning strategy described above to obtain the target multimodal large model, this embodiment further enhances the model's ability to reference visual elements when generating explanatory content. This approach introduces a fine-tuning method based on prompt instruction templates. By constructing structured instruction templates containing visual association constraint information, this method injects visual association constraints into the model's fine-tuning process. This enables the model to establish an output pattern where each explanatory step must reference visual elements while learning the target subject knowledge.

[0132] The process of fine-tuning the target subject's training sample set in conjunction with the prompt instruction template may include: obtaining a prompt instruction template configured for the target subject, which contains visual association constraint information; for each training sample in the training sample set, combining the structured text (including question text and visual element information) in the training sample with the prompt instruction template to obtain script-generated prompt instructions; and using the script-generated prompt instructions and the test question image samples in the training sample to fine-tune the basic multimodal large model.

[0133] Among them, visual association constraint information is used to force the basic multimodal large model to use visual elements in the test question image samples as the basis for reasoning when generating explanation content.

[0134] This embodiment introduces a prompt instruction template containing visual association constraint information, enabling the model to learn the output pattern that must refer to visual elements during the fine-tuning process, thereby stably generating teaching video scripts that combine text and images during the inference stage.

[0135] In one possible implementation, the prompt instruction template may include the following parts: question element part, explanation requirement part, and output format part.

[0136] The question element section is used to contain the structured text in the training samples (including question text and visual element information, for example, the question text is "In Rt△ABC, ∠C=90°, find the length of the height CD on the hypotenuse AB", and the visual element information is "vertex A(80,120), B(280,120), C(80,220), side BC is labeled "4cm", side AC is labeled "3cm", and CD is a dashed line on side AB (labeled "height")"). That is, the above structured text is filled into this part of the template.

[0137] The explanation requirements section is designed to cover the explanation of the fundamental multimodal model. Different explanation requirements are specified for different grade levels and subject modules. For example, for the junior high school geometry module, the explanation requirements section might include the following: "Explain each step in relation to the visual elements of the problem," "Include one interactive node to guide observation of the figure," and "Supplement with explanations of the applicable conditions for the Pythagorean theorem." The explanation requirements section can be configured differently for different grade levels and math modules. For example, the elementary school arithmetic module requires "Using everyday language" and "Demonstrating the calculation process step by step," while the high school functions module requires "Supplementing with function graph analysis" and "Related to the concept of derivatives."

[0138] The output format section standardizes the output structure of the explanation content generated from the basic multimodal large model. The output structure may include some or all of the following modules: graphical observation module, known condition extraction module, problem-solving steps module, interactive guidance module, and knowledge expansion module. It is preferable to include all modules, and the above modules should implicitly contain visual association requirements.

[0139] The graphic observation module requires the model to describe the graphic features in the test question image at the beginning of the explanation and refer to the corresponding visual elements. The known condition extraction module requires the model to extract known conditions by combining the visual element information of the image. The problem-solving step module requires each problem-solving step to be associated with the corresponding visual evidence. The interactive guidance module requires the model to embed interactive instructions to guide students to think or operate in an appropriate position. The knowledge extension section requires the model to associate similar question types or common mistakes for extended explanation.

[0140] The structured text from the training samples is used to populate the prompt instruction template to generate script-generated prompt instructions. An example of a script-generated prompt instruction is shown below:

[0141] Please generate an interactive video script based on the following math problem elements: [Grade Level: Junior High Mathematics] [Module: Geometry]. Requirements: 1. Explain the relevant visual elements of the problem at each step; 2. Include one interactive node to guide observation of the figure; 3. Supplement with an explanation of the applicable conditions for the Pythagorean theorem. <Problem Elements>

[0142] Text: In right triangle ABC, ∠C = 90°, find the length of the altitude CD on the hypotenuse AB;

[0143] image: Figure 1 The vertices A(80,120), B(280,120), and C(80,220) are labeled. Edge BC is labeled "4cm", edge AC is labeled "3cm", and CD is a dashed line on edge AB (labeled "height").

[0144] <Output Format>

[0145] 1. Graphic Observation: [Describe the graphic characteristics by associating visual elements];

[0146] 2. Extraction of known conditions: [Extracting mathematical conditions by combining visual annotations];

[0147] 3. Solution steps: [Each step relates to visual evidence, including theorem explanations];

[0148] 4. Interactive Nodes: [Interactive elements that guide thinking or action];

[0149] 5. Knowledge Expansion: [Related to similar question types or common mistakes].

[0150] This embodiment achieves full-process teaching adaptation of "input-demand-output" through the design of structured instruction templates. It forces the model to integrate visual elements into every explanation step from the entry point, solving the problems of scripts generated by existing solutions having no visual association and disjointed explanations.

[0151] In one possible implementation, the parameters of the basic multimodal large model can be optimized using a joint loss function during the fine-tuning of the basic multimodal large model.

[0152] Specifically, the joint loss function can include a textual semantic loss term and a visual association loss term. The expression for the joint loss function is as follows:

[0153] Loss = α × text semantic loss + β × visual association loss (1).

[0154] Where α and β are weighting coefficients, for example, α=0.7 and β=0.3.

[0155] Text semantic loss measures the semantic difference between the instructional video scripts generated by the basic multimodal large model and the corresponding step-by-step explanation texts in the training sample set. Cross-entropy loss can be used to optimize the correctness of the problem-solving logic and the standardization of subject-specific terminology. For example, when there is a semantic discrepancy between the instructional video scripts generated by the model and the corresponding step-by-step explanation texts, the text semantic loss term will output a larger loss value, driving the model to adjust its parameters to generate more accurate explanations.

[0156] The visual association loss is used to indicate whether the instructional video scripts generated by the basic multimodal large model mention the core visual elements in the test question image samples. The visual association loss term can use simple matching loss, and its core function is to determine whether the generated instructional text mentions the core visual elements (such as "AC=3cm", "right angle symbol", "CD is height", etc.). If the generated instructional video script does not mention the core visual elements, a loss penalty is triggered, driving the model to add references to the visual elements in subsequent generation. For example, when the model generates the step of "calculating the area of ​​the triangle," if it fails to mention the visual evidence that "AC and BC in the figure are right angle sides," the visual association loss term will incur a loss penalty.

[0157] This embodiment optimizes the quality of text generation by designing a joint loss function, while forcing the model to establish visual-semantic associations from the training target end. This ensures that the scripts generated by the model not only conform to the teaching logic of the target subject, but are also highly related to visual elements.

[0158] After fine-tuning the target multimodal large model using a training sample set specific to the target subject, and obtaining structured text and standardized test question images by performing text extraction and image standardization on the target test question image samples, the structured text and standardized test question images can be processed based on the target multimodal large model to generate teaching video scripts. In some embodiments of this application, the process of processing structured text and standardized test question images based on the target multimodal large model to generate teaching video scripts is described.

[0159] Specifically, the process of generating instructional video scripts by processing structured text and standardized test item images based on a target multimodal large model can include:

[0160] Step b1: Using the target multimodal large model, extract the textual semantic features of the structured text and extract the image visual features of the standardized test question images.

[0161] Specifically, the structured text is encoded through the embedding and encoding layers of the language model of the target multimodal large model to obtain text semantic features. The text semantic features focus on the core concepts, condition descriptions and solution objectives in the question stem. The standardized test question images are encoded through the visual encoder of the target multimodal large model to extract image visual features. The image visual features focus on the visual information in the image, such as vertex positions, side length labels, formula layout, and graphic outlines.

[0162] Step b2: Using the target multimodal large model, perform cross-modal feature alignment processing on text semantic features and image visual features, establish the association mapping between concepts related to the target subject in the structured text and corresponding visual elements in the standardized test question image, and obtain joint features that integrate text semantics and visual features.

[0163] Specifically, through the cross-modal alignment layer of the target multimodal large model, a cross-attention mechanism is used to calculate the attention weights between text semantic features and image visual features. This establishes an association mapping between concepts related to the target subject in the structured text and corresponding visual elements in the standardized test question image (such as the association between "hypotenuse AB" and the longest side label in the image, and the association between "quadratic function" and the function graph curve). This generates text-visual joint features, which contain the association information between subject-related concepts in the structured text and corresponding visual elements in the standardized test question image.

[0164] Step b2: Using the target multimodal large model, generate interactive teaching video scripts according to the joint features and the preset script structure.

[0165] Specifically, the target multimodal large model generates interactive teaching video scripts based on joint features, following the standardized output format specified in the script generation prompts. These interactive teaching video scripts include the following modules: a graphical observation generation module, a known condition extraction module, a problem-solving step module, an interactive guidance module, and a knowledge expansion module. Each step in the problem-solving step module references corresponding visual evidence.

[0166] Below is an example of an interactive teaching video script:

[0167] Graphical observation module: Explicitly mentions mathematical visual elements, for example: "Students, first look at the image on the screen..." Figure 1 This is a right triangle ABC. The diagram shows that ∠C is a right angle. The positions of the three vertices A, B, and C are clearly marked. Side AC is marked 3cm, and side BC is marked 4cm. There is also a dotted line connecting point C to side AB, with the 'altitude' marked next to it. This is the CD we are looking for.

[0168] Known condition extraction module: Extracts mathematical conditions by combining visual annotations, example: "from..." Figure 1 From the annotation, we can extract the known conditions: ① △ABC is a right triangle, with the right angle at point C; ② The right-angled side AC = 3cm, and the right-angled side BC = 4cm; ③ What is required is the length of the altitude CD on the hypotenuse AB.

[0169] Problem-solving steps module: Each step connects visual evidence with mathematical theorems. Example: "Step 1: Determine the solution method. Since this is a right triangle, we can first use the Pythagorean theorem to find the length of the hypotenuse AB (the Pythagorean theorem applies to right triangles and states that the sum of the squares of the legs equals the square of the hypotenuse). From the diagram, we can see that AB is the longest side, which is the hypotenuse; Step 2: Substitute the data to calculate AB, Therefore, AB = 5cm; Step 3: Find the height CD using the area method. There are two ways to calculate the area of ​​a right triangle: one is 1 / 2 × leg × leg, and the other is 1 / 2 × hypotenuse × altitude to the hypotenuse. From the position of CD in the diagram, we know that it is the altitude to the hypotenuse, so we can set up the equation: Step 4: Substitute the data to solve. The left side is 6, and the right side is... ,so ".

[0170] Interactive Node Module: Design interactive activities that align with mathematics teaching. Example: "Now, let's pause for a moment. Please draw this right triangle in your exercise book, mark each vertex and the known side lengths, and then try to verify the length of CD using another method. Think about how CD will change if AC becomes longer?"

[0171] Knowledge Extension Module: Connecting common math mistakes with similar problem types. Example: "Please note that the prerequisite for using the area method to find the height is knowing the area of ​​the triangle and the length of its corresponding base. Here, because it is a right triangle, we can quickly calculate the area using the two legs. A similar problem type is 'Given the length of the legs and the length of the base of an isosceles triangle, find the height on the base.' The core idea is to first calculate the area and then work backward to deduce the length of the height."

[0172] This embodiment establishes a precise association between subject-related concepts in text and visual elements in images through cross-modal feature alignment processing, and generates an interactive teaching video script that integrates text and images based on this association, so that each reasoning step is supported by corresponding visual evidence, which significantly improves the intuitiveness and comprehensibility of teaching explanations.

[0173] The teaching video script generation method provided by this invention has the following advantages:

[0174] Firstly, it achieves reasoning-based explanations combining text and images, significantly enhancing the intuitiveness of teaching. This invention uses a target multimodal large model to perform cross-modal feature alignment processing on structured text and standardized test question images, establishing a connection between the semantics of the test question text and the visual elements of the image. This allows the explanation steps in the generated script to reference relevant visual elements in the standardized test question image as the basis for reasoning, achieving reasoning-based explanations combining text and images. This enables students to intuitively match the reasoning steps with specific graphic areas in the image when watching the teaching video, significantly improving the intuitiveness and comprehensibility of the teaching explanations.

[0175] Secondly, the stability of visual references is ensured through the consistent design of the visual association mechanism between the training data and the inference stage. During the training stage, this invention uses a training sample set containing a triplet of "test question image sample - structured text - step-by-step explanation text" to fine-tune the basic multimodal large model. Each step in the step-by-step explanation text references visual elements from the test question image sample as the basis for inference. Similarly, the script generated during the inference stage references visual elements from the standardized test question image. This consistency in the visual association mechanism between the training and inference stages creates a complete closed loop from learning to application, fundamentally solving the problem of "disconnect between explanation and graphics" in existing technologies. This ensures that the model can stably generate explanation scripts that reference visual elements during the inference stage.

[0176] Third, adaptive teaching style matching across different educational stages is achieved through a dedicated training sample set covering multiple educational stages. This invention uses a training sample set covering multiple different educational stages to fine-tune the basic multimodal model, and labels each training sample with its educational stage, enabling the model to learn the terminology, reasoning logic, and teaching style of different educational stages. During the reasoning stage, the model can adaptively adjust the teaching method according to the educational stage to which the question belongs, achieving precise adaptation of teaching styles across different educational stages.

[0177] Fourth, by employing a joint labeling system and a hierarchical resampling strategy, the model's ability to handle complex multi-modal test questions is significantly improved. This invention selects four types of challenging samples based on teaching practice: overlapping text and images, handwritten text, complex formulas, and multiple interconnected images. A joint labeling system is constructed using three categories of labels: visual interference level, difficulty coefficient, and core visual elements. This provides multi-dimensional feature guidance for challenging samples. Simultaneously, a hierarchical resampling strategy is adopted, allocating differentiated resampling ratios based on the frequency of occurrence of each type of challenging sample in actual teaching. This allows the model to focus on high-frequency challenging samples during training, strengthening feature learning for various challenging scenarios and significantly improving its ability to handle complex multi-modal test questions.

[0178] Fifth, a lightweight fine-tuning strategy is adopted to adapt to edge deployment and lower the hardware threshold. This invention adopts a lightweight fine-tuning strategy that freezes the visual encoder and the basic layer of the language model, and only fine-tunes the cross-modal alignment layer and the top layer of the language model. It directly reuses the built-in visual encoding and feature alignment capabilities of the basic multimodal large model. By freezing most parameters and only fine-tuning a small number of top-level parameters, the computational cost is significantly reduced while ensuring the adaptation effect of the target subject. The memory usage of a single card is reduced to less than 10GB, so that the trained target multimodal large model can be adapted to edge educational hardware such as learning machines and tablets.

[0179] Sixth, by using structured instruction templates and a joint loss function, visual association is enforced from both the entry point and the training target. This invention designs a structured instruction template that includes a description requirement section, a question element section, and an output format section. Visual association constraint information is embedded in the template, forcing the model to reference visual elements when generating explanations from the entry point. At the same time, a joint loss function that includes textual semantic loss and visual association loss is designed to penalize the generated results that do not mention core visual elements from the training target. Through the dual constraints of instruction templates and loss functions, it is ensured that the scripts generated by the model have both the accuracy of teaching logic and sufficient reference to visual elements in the question images.

[0180] This invention also provides a teaching video script generation device, such as... Figure 4 As shown, the teaching video script generation device may include: a test question image acquisition module 401, a test question image processing module 402, and a teaching video script generation module 403.

[0181] The test question image acquisition module 401 is used to acquire the target test question image of the target subject. The target test question image is an image of the test question containing visual elements.

[0182] The test question image processing module 402 is used to perform text extraction and image standardization processing on the target test question image to obtain structured text and standardized test question image.

[0183] The teaching video script generation module 403 is used to perform cross-modal feature alignment processing on structured text and standardized test question images based on the target multimodal large model, establish the association between the semantics of the test question text and the visual elements of the image, and generate a teaching video script that integrates visual semantics based on the association. The explanation steps in the teaching video script refer to the relevant visual elements in the standardized test question images as the basis for reasoning.

[0184] Among them, the target multimodal large model is obtained by fine-tuning the basic multimodal large model using the training sample set of the target subject. Each training sample in the training sample set includes a test question image sample, the structured text corresponding to the test question image sample, and the step-by-step explanation text. The explanation steps in the step-by-step explanation text refer to the visual elements in the test question image sample as the basis for reasoning.

[0185] In one possible implementation, the instructional video script includes interactive guidance instructions set during the explanation process. These instructions guide the user to perform interactive operations related to solving standardized test question images.

[0186] In one possible implementation, the instructional video script generation device may further include a training sample set acquisition module. This module is used to acquire a training sample set for the target subject.

[0187] The training sample set acquisition module is specifically used to acquire the training sample set for the target subject as follows:

[0188] Obtain an initial training sample set. Each training sample in the initial training sample set includes a test question image sample for the target subject and the corresponding step-by-step explanation text for the test question image sample.

[0189] The initial training sample set is processed to obtain the processed training sample set. Each training sample in the processed training sample set includes a test question image sample and the corresponding structured text and step-by-step explanation text. The structured text includes the question text and visual element information.

[0190] Difficult samples are selected from the processed training sample set, and labels are added to the selected difficult samples to characterize their features;

[0191] The selected difficult samples are resampled, and the resampled difficult samples are added to the processed training sample set to obtain the final training sample set.

[0192] In one possible implementation, the label for each difficult sample includes some or all of the following labels: visual interference level label, difficulty coefficient label, and core visual element label;

[0193] Among them, the visual interference level label is used to characterize the intensity of visual interference in difficult samples, the difficulty coefficient label is used to characterize the difficulty of the questions in difficult samples, and the core visual element label is used to indicate the key visual information in difficult samples that affects the solution of the questions.

[0194] In one possible implementation, the training sample set acquisition module, when filtering difficult samples from the processed training sample set, is specifically used for:

[0195] From the processed training sample set, one or more of the following categories of samples are selected as difficult samples: overlapping text and image samples, handwritten samples, complex formula samples, and multi-image linked samples.

[0196] Among them, the text-image overlap type samples are those in the test question image samples where the area of ​​text covering the graphic exceeds a preset area threshold; the handwritten type samples are those in the test question image samples where the handwritten character recognition accuracy is lower than a preset accuracy threshold; the complex formula type samples are those in the test question image samples where the number of nested mathematical formulas exceeds a preset number threshold; and the multi-image linkage type samples are those in the test question image samples where the number of associated step images exceeds a preset number threshold.

[0197] In one possible implementation, the selected difficult samples include multiple categories of difficult samples;

[0198] The training sample set acquisition module is specifically used for resampling the selected difficult samples when:

[0199] Obtain the frequency of occurrence of various difficult samples in actual teaching;

[0200] Based on the frequency of occurrence, differentiated resampling ratios are assigned to various types of difficult samples;

[0201] Based on the resampling ratio assigned to each type of difficult sample, resampling is performed on each type of difficult sample.

[0202] In one possible implementation, the instructional video script generation device may further include: a model training model, which is used to fine-tune a basic multimodal large model using a training sample set of the target subject.

[0203] When fine-tuning a basic multimodal large model using a training sample set from the target discipline, the model is specifically used for:

[0204] Freeze the parameters of the visual encoder and the base layer of the language model in the basic multimodal large model;

[0205] By using the training sample set of the target discipline, the parameters of the cross-modal alignment layer of the basic multimodal large model and the top layer of the language model are finely adjusted to obtain the target multimodal large model.

[0206] In one possible implementation, when fine-tuning the basic multimodal large model using a training sample set from the target discipline, the model training is specifically used for:

[0207] Obtain prompt instruction templates configured for the target subject, which contain visual association constraint information;

[0208] For each training sample in the training sample set, the structured text in the training sample is combined with the prompt instruction template to obtain the script-generated prompt instruction. The script-generated prompt instruction and the test question image samples in the training sample are used to fine-tune the basic multimodal large model.

[0209] Among them, visual association constraint information is used to force the basic multimodal large model to reference visual elements in the test question image samples when generating explanation content.

[0210] In one possible implementation, the prompt instruction template includes: an explanation of the requirements, a question element section, and an output format section;

[0211] The explanation requirements section is used to carry the explanation requirements of the basic multimodal large model. The question element section is used to contain the structured text in the training samples. The output format section is used to standardize the output structure of the explanation content generated by the basic multimodal large model. The output structure includes some or all of the following modules: graphical observation module, known condition extraction module, problem-solving steps module, interactive guidance module, and knowledge expansion module.

[0212] In one possible implementation, the training sample set covers multiple different educational stages, and each training sample in the training sample set has an educational stage label. The educational stage label is used to identify the educational stage to which the corresponding training sample belongs, so that the target multimodal large model can adapt to the teaching style of the corresponding educational stage when generating teaching video scripts.

[0213] In one possible implementation, fine-tuning is performed using a joint loss function to optimize the parameters of the base multimodal large model. The joint loss function includes a text semantic loss term and a visual association loss term. The text semantic loss term measures the semantic difference between the instructional video script generated by the base multimodal large model and the corresponding step-by-step explanation text in the training sample set, while the visual association loss term indicates whether the instructional video script generated by the base multimodal large model mentions the core visual elements in the test question image samples.

[0214] In one possible implementation, the teaching video script generation module 403, when performing cross-modal feature alignment processing on structured text and standardized test question images based on a target multimodal large model, establishing the association between the semantics of the test question text and the visual elements of the image, and generating a teaching video script that integrates visual semantics based on the association, is specifically used for:

[0215] Based on the target multimodal large model, the text semantic features of the structured text and the image visual features of the standardized test questions are extracted respectively. Cross-modal feature alignment processing is performed on the text semantic features and image visual features to establish the association mapping between the concepts related to the target subject in the structured text and the corresponding visual elements in the standardized test questions, so as to obtain the joint features that integrate the text semantic and visual features. Based on the joint features, the interactive teaching video script is generated according to the preset script structure.

[0216] The teaching video script generation device provided in this embodiment of the invention performs cross-modal feature alignment processing on structured text and standardized test question images by calling a target multimodal large model, establishes the association between the semantics of the test question text and the visual elements of the image, and generates a teaching video script based on the association, so that the explanation steps in the script can refer to the relevant visual elements in the standardized test question images as the basis for reasoning, thus realizing a reasoning-based explanation that combines text and images.

[0217] This invention also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0218] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0219] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the teaching video script generation method provided in the above embodiments.

[0220] This invention also provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement the teaching video script generation method provided in the above embodiments.

[0221] This invention also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the teaching video script generation method provided in the above embodiments.

[0222] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by this invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0224] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0225] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for generating instructional video scripts, characterized in that, include: Obtain the target test question image for the target subject, wherein the target test question image is an image of a test question containing visual elements; Text extraction and image standardization are performed on the target test question images to obtain structured text and standardized test question images; Based on the target multimodal large model, cross-modal feature alignment processing is performed on the structured text and the standardized test question image to establish the association between the semantics of the test question text and the visual elements of the image. Based on the association, a teaching video script that integrates visual semantics is generated. The explanation steps in the teaching video script refer to the relevant visual elements in the standardized test question image as the basis for reasoning. The target multimodal large model is obtained by fine-tuning the basic multimodal large model using the training sample set of the target subject. Each training sample in the training sample set includes a test question image sample, the structured text corresponding to the test question image sample, and the step-by-step explanation text. The explanation steps in the step-by-step explanation text refer to the visual elements in the test question image sample as the basis for reasoning.

2. The teaching video script generation method according to claim 1, characterized in that, The teaching video script includes interactive guidance instructions set during the explanation process. These interactive guidance instructions are used to guide users to perform interactive operations related to solving the standardized test question images.

3. The teaching video script generation method according to claim 1, characterized in that, The process of obtaining the training sample set includes: Obtain an initial training sample set, wherein each training sample in the initial training sample set includes a test question image sample of the target subject and a step-by-step explanation text corresponding to the test question image sample; The initial training sample set is processed to obtain a processed training sample set. Each training sample in the processed training sample set includes a test question image sample and the corresponding structured text and step-by-step explanation text. The structured text includes the question text and visual element information. Difficult samples are selected from the processed training sample set, and labels are added to the selected difficult samples to characterize their features; The selected difficult samples are resampled, and the resampled difficult samples are added to the processed training sample set to obtain the final training sample set.

4. The teaching video script generation method according to claim 3, characterized in that, Each of the difficult samples is labeled with some or all of the following labels: visual interference level label, difficulty coefficient label, and core visual element label; The visual interference level label is used to characterize the intensity of visual interference in the difficult sample, the difficulty coefficient label is used to characterize the difficulty of the difficult sample, and the core visual element label is used to indicate the key visual information in the difficult sample that affects the solution of the problem.

5. The teaching video script generation method according to claim 3, characterized in that, The step of screening difficult samples from the processed training sample set includes: From the processed training sample set, one or more of the following types of samples are selected as difficult samples: image-text overlap samples, handwritten samples, complex formula samples, and multi-image linkage samples. Among them, the text-image overlap type sample is a sample in the test question image where the area of ​​text covering the graphic exceeds a preset area threshold; the handwritten character type sample is a sample in the test question image where the handwritten character recognition accuracy is lower than a preset accuracy threshold; the complex formula type sample is a sample in the test question image where the number of nested mathematical formulas exceeds a preset number threshold; and the multi-image linkage type sample is a sample in the test question image where the number of associated step images exceeds a preset number threshold.

6. The teaching video script generation method according to claim 3, characterized in that, The selected difficult samples include multiple categories of difficult samples; The resampling process for the selected difficult samples includes: Obtain the frequency of occurrence of various difficult samples in actual teaching; Based on the obtained frequency of occurrence, differentiated resampling ratios are assigned to various types of difficult samples; Based on the resampling ratio assigned to each type of difficult sample, resampling is performed on each type of difficult sample.

7. The teaching video script generation method according to claim 1, characterized in that, The basic multimodal large model is fine-tuned using the training sample set of the target discipline to obtain the target multimodal large model, including: Freeze the parameters of the visual encoder and the base layer of the language model in the basic multimodal large model; Using the training sample set of the target discipline, the parameters of the cross-modal alignment layer of the basic multimodal large model and the top layer of the language model are finely adjusted to obtain the target multimodal large model.

8. The teaching video script generation method according to claim 1, characterized in that, The basic multimodal large model is fine-tuned using the training sample set of the target discipline to obtain the target multimodal large model, including: Obtain a prompt instruction template configured for the target subject, wherein the prompt instruction template contains visual association constraint information; For each training sample in the training sample set, the structured text in the training sample is combined with the prompt instruction template to obtain a script-generated prompt instruction. The script-generated prompt instruction and the test question image samples in the training sample are used to fine-tune the basic multimodal large model. The visual association constraint information is used to force the basic multimodal large model to reference visual elements in the test question image samples when generating explanation content.

9. The teaching video script generation method according to claim 8, characterized in that, The prompt instruction template includes: an explanation of requirements, a question element section, and an output format section; The explanation requirement section is used to carry the explanation requirements of the basic multimodal large model, the question element section is used to contain the structured text in the training samples, and the output format section is used to standardize the output structure of the explanation content generated by the basic multimodal large model. The output structure includes some or all of the following modules: graphical observation module, known condition extraction module, problem-solving steps module, interactive guidance module, and knowledge expansion module.

10. The teaching video script generation method according to claim 1, characterized in that, The training sample set covers multiple different educational stages; Each training sample in the training sample set has a learning segment label, which is used to identify the learning segment to which the corresponding training sample belongs, so that the target multimodal large model can adapt to the teaching style of the corresponding learning segment when generating teaching video scripts.

11. The method for generating teaching video scripts according to claim 1, characterized in that, The fine-tuning employs a joint loss function to optimize the parameters of the basic multimodal large model; The joint loss function includes a text semantic loss term and a visual association loss term; The text semantic loss term is used to measure the semantic difference between the teaching video script generated by the basic multimodal large model and the corresponding step-by-step explanation text in the training sample set, and the visual association loss term is used to indicate whether the core visual elements in the test question image sample are mentioned in the teaching video script generated by the basic multimodal large model.

12. The teaching video script generation method according to claim 1, characterized in that, The step of performing cross-modal feature alignment processing on the structured text and the standardized test question images to establish the association between the semantics of the test question text and the visual elements of the images, and generating a teaching video script that integrates visual semantics based on the association, includes: Extract the textual semantic features of the structured text and the image visual features of the standardized test question image, respectively; Cross-modal feature alignment processing is performed on the text semantic features and the image visual features to establish an association mapping between the concepts related to the target subject in the structured text and the corresponding visual elements in the standardized test question image, thereby obtaining joint features that integrate text semantics and visual features; Based on the aforementioned joint features, an interactive teaching video script is generated according to a preset script structure.

13. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the teaching video script generation method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing an electronic device to execute the instructional video script generation method as described in any one of claims 1 to 12.

Images