An enhanced captioning method based on an expressive code diffusion model

Abstract

The application provides an enhanced subtitle method based on an expression code diffusion model; in order to solve the problems that a subtitle is difficult to reflect an intention and a rhythm and a style switching is unstable in a classroom scene, the method receives a classroom voice stream to generate a transcription text sequence, and extracts a rhythm vector and a semantic vector to form a voice text pair; under the constraint of the intention and the rhythm, an expression code diffusion network is driven to output an expression code, an expression strength and an expression unit boundary, so that a sequence of expression code triplets is obtained; the expression code is one-to-one mapped to a unique main symbol template, and style parameters such as intensity binding color, font weight, outline, background, size and animation are used to generate a main symbol style stream under the constraint of a single main channel; a template switching instruction set is generated according to the difference between adjacent expression units, and a subordinate labeled object superposition is generated according to a trigger rule, so that an enhanced subtitle frame sequence is synthesized and output; the method realizes controllable rendering and switching with an expression unit as a period, and improves the consistency of the subtitle presentation.

Description

Technical Field

[0001] This invention relates to the field of speech transcription and subtitle visualization technology, and in particular to an enhanced subtitle method based on an expression code diffusion model. Background Technology

[0002] In classroom teaching, teachers' lectures are often accompanied by variations in speaking speed, pauses, emphasis, and intonation. These rhythmic information, along with semantic intent, constitute expressive effects such as "emphasis, transition, questioning, and summarizing." For deaf or hearing-impaired students, classroom information is primarily obtained through subtitles. If subtitles only provide word-for-word transcription, key points and structural clues are easily missed, affecting comprehension and follow-along. Therefore, subtitle systems designed for classroom scenarios not only need to transcribe speech into text but also need to align it on a timeline and present the layers of expression visually, highlighting key content without interfering with reading. Furthermore, the classroom environment is subject to multiple sources of interference, including echoes, student interruptions, and blackboard explanations, making consistency in subtitle style and clear hierarchy particularly crucial.

[0003] Current classroom subtitle generation typically employs speech recognition models to transcribe speech streams in real time, combining timestamps for subtitle segmentation and scrolling display. To improve readability, some solutions introduce sentence segmentation, punctuation restoration, speaker separation, and keyword extraction to organize the text into fragments. Other solutions utilize acoustic features such as energy, fundamental frequency, and pauses, or emotion / intonation classification networks, to infer tags such as stress, question, and emotion, then map these tags to visual styles such as font color, font size, bolding, outlines, background bars, or simple animations. For prompts, common practices include overlaying icons, underlines, prompt boxes, or side notes to indicate questions, reminders, key points, etc., and displaying them synchronously with the subtitles on the player or classroom terminal interface. Some systems link subtitles with courseware and blackboard screenshots, generating chapter prompts by detecting teacher instructions or courseware titles. Style generation often uses a preset template library or rule engine to render output on the client side according to timestamps, and provides subtitle export, retrieval, and playback for post-class review.

[0004] The aforementioned solutions mostly drive style changes directly through word-level or short-window features, lacking a unified representation constrained by both semantic intent and prosody. This leads to unstable segment boundaries that fluctuate with noise or speech rate, manifesting as frequent flickering or style jumps on the interface. Some solutions overlay multiple main styles and graphic elements in parallel to express more information, easily creating visual competition and distracting deaf students when reading the main subtitles. Other overlay annotations lack temporal and template constraints with the main subtitles, potentially obscuring text or conflicting with the main styles, reducing the continuity of real-time classroom reading. Furthermore, existing methods often bind style switching to subtitle line breaks, punctuation, or fixed time windows, making it difficult to correspond to actual expression units, and the problem of reading interruption is even more prominent in classroom scenarios.

[0005] Therefore, an enhanced subtitle method that can overcome the shortcomings of the prior art is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an enhanced subtitle method based on an expression code diffusion model. The core technical problem to be solved is: in the scenario of real-time transcription and display of classroom speech, how to generate a sequence of expression units containing expression codes, intensity, and boundaries based on the joint constraints of semantic intent and prosodic information, and form a subtitle style flow and overlay annotation mechanism that can be switched according to the boundary under the constraints of single main channel visual grammar, so as to map the expression level and key points to the interface rendering with a unified temporal rule without changing the transcribed content, thereby reducing reading interruptions for deaf students during classroom reading.

[0007] An enhanced captioning method based on an expression code diffusion model according to an embodiment of the present invention includes:

[0008] S1. Receive classroom audio stream and generate a transcribed text sequence to form audio-text pairs;

[0009] S2. Based on the speech-text pair, expressive code modeling is performed using the expressive code diffusion model to generate a sequence of expressive code triples. The expressive code triples include expressive code, expressive intensity, and expressive unit boundary. Under the constraints of intent and prosody, continuous speech is merged into expressive units, and the transcribed text fragments covered by the expressive unit boundary are defined.

[0010] S3. Based on the expression code triple sequence, construct the code template mapping of the single main channel visual syntax framework. Under the single main channel constraint, determine a unique main symbol template for each expression unit and bind the expression intensity to generate the main symbol style stream. The main symbol style stream corresponds to the boundary of the expression unit and serves as the basis for switching judgment.

[0011] S4. Determine the code changes of adjacent representation units based on the main symbol style flow and the representation code triplet sequence, and generate a template switching instruction set;

[0012] S5. Determine the set of preset trigger class expression codes based on the expression code triple sequence and generate subordinate annotation objects;

[0013] S6. Based on the transcribed text sequence, main symbol style stream, and template switching instruction set, synthesize the subtitle style stream and overlay the subordinate annotation objects;

[0014] S7. Output an enhanced subtitle frame sequence for interface presentation.

[0015] Optionally, S1 is as follows:

[0016] The system receives classroom audio streams, divides them into speech segments, and extracts prosodic strength, pauses, and intonation from the audio streams to generate a prosodic vector sequence, maintaining the continuity of prosodic strength, pauses, and intonation within the speech segments.

[0017] Speech recognition is performed on the classroom speech stream to generate a transcribed text sequence. Semantic intent and word phrase relationships are extracted from the transcribed text sequence to generate a semantic vector sequence. The transcribed text sequence is then segmented into text fragments according to continuous speech segments.

[0018] Synchronize the prosodic vector sequence and the transcribed text sequence according to the continuous speech segments, establish a one-to-one correspondence at each step, and bind the prosodic vector at each step with the semantic vector at each step to form a correspondence between speech and text pairs.

[0019] The prosodic vector sequence and the transcribed text sequence and their correspondence are taken as input and received by the intention encoder and prosodic encoder of the expression code diffusion model. The prosodic vector and semantic vector corresponding to each step in the speech-text pair are identified to form a speech-text pair.

[0020] Optionally, S2 is as follows:

[0021] The prosodic vector sequence and semantic vector sequence in the speech-text pair are input into the prosodic encoder and the intent encoder, respectively. The corresponding prosodic code and intent code are output at each step, and the two types of codes are used as input to the constraint fusion layer.

[0022] A constraint fusion layer is used to concatenate and linearly transform the intent encoding and prosodic encoding to generate a constraint vector sequence for each step. The constraint vector sequence is used as the conditional input of the expression code diffusion network and maintains the step correspondence with the speech-text pair.

[0023] Under the condition of constraint vector sequence, the expression code diffusion network is driven to iteratively optimize the fragment representation of continuous speech segments. It is sequentially passed through eight noise reduction blocks. In the conditional injection layer of each noise reduction block, the constraint vector of each step is added to the hidden state to maintain the linkage.

[0024] After iterative optimization, each step is decoded using the expression code output head, expression intensity output head, and expression unit boundary output head to obtain the expression code, expression intensity, and boundary scale vector, and the expression unit boundary is determined based on the boundary scale vector.

[0025] Code changes are determined based on the continuous change in expression intensity and the boundary condition threshold. When the change in expression intensity reaches the boundary condition threshold, the expression unit boundary is updated, and the continuous speech segment is divided into expression units according to the updated expression unit boundary.

[0026] Continuous discourse is grouped into expression units according to the expression unit boundaries, and the transcribed text fragments covered by the expression unit boundaries are defined. The expression codes and expression intensities obtained by the determination are respectively bound to the corresponding expression unit boundaries to generate expression code triples.

[0027] The generated expression code triple sequence is used as the input to the code template mapping, and the expression unit boundary is used as the switching criterion, so that the change of expression code is directly located to the change of main symbol template, and the expression unit control cycle result is provided for the subsequent main symbol style stream output.

[0028] Optionally, under the constraint vector sequence, the expression code diffusion network is driven to complete iterative optimization. After iterative optimization, the expression code output head, expression intensity output head, and expression unit boundary output head are used to decode each step to obtain the expression code, expression intensity, and boundary scale vector. In the process of determining the expression unit boundary based on the boundary scale vector, a joint boundary confidence formula is further introduced to jointly determine the boundary scale vector and expression intensity as the basis for determining the expression unit boundary.

[0029] ;

[0030] in, For step-level boundary confidence, The weighting coefficients for the differences in the boundary scale vectors. The weighting coefficients are used to express the intensity changes. The penalty weight coefficient for the minimum spacing constraint. and For steps The boundary scale vector takes two-dimensional values. and For steps and the intensity of expression in the previous step, To express the minimum spacing threshold of the cell boundary, in steps, The step index for the most recently confirmed expression unit boundary. To obtain the maximum function, which is used to form the normalized insufficient spacing penalty term, the following is... With boundary condition threshold When comparing, Time stamp step For boundary candidates, when the candidate spacing is less than The candidates added after removal are removed, resulting in an ordered and controlled set of expression unit boundary indices.

[0031] Optionally, S3 specifically refers to:

[0032] The expression code in the expression code triple sequence is input into the code template mapping, a master symbol template identifier is generated using a one-to-one binding method, and the initial values ​​of the style parameter vector are output to establish a unique mapping relationship between the expression code and the master symbol template identifier;

[0033] The expression intensity in the expression code triple sequence is used as the input of the intensity binding function. The adjustment coefficient of the style parameter vector output by the intensity binding function is used to adjust the style parameter vector. The style parameter vector is a six-dimensional vector containing color parameters, font weight parameters, outline parameters, background parameters, size parameters, and animation parameters. The binding with the expression intensity is completed within the constraints.

[0034] Using the boundary of the expression unit in the expression code triple sequence as the control period, the style segment of the main symbol style flow is generated according to the main symbol template identifier and the adjusted style parameter vector, so that each style segment corresponds one-to-one with the corresponding expression unit boundary, and carries the main symbol template identifier and style parameter vector in each style segment;

[0035] Input the main symbol style stream into the single main channel constraint layer. Based on the expression unit boundary, only one main symbol template is activated in the same time period. Output the constrained main symbol style stream. Output the constrained main symbol style segment with the expression unit as the control period and keep multiple main symbol templates from being presented in parallel.

[0036] The switching criteria are established based on the constrained main symbol style flow and the expression unit boundary. The main symbol template identifier and style parameter vector corresponding to each expression unit boundary are bound as the switching target, and the expression unit boundary is used as the sole benchmark for the switching criteria.

[0037] The constrained main symbol style flow is solidified at the boundary of the expression unit, so that the change of the expression code is directly located to the change of the main symbol template, and the expression unit control cycle of the main symbol style flow is formed, maintaining the linear output relationship of the main symbol style flow under the constraint of a single main channel;

[0038] The constrained main symbol style flow and the switching decision criteria are used as inputs for generating the template switching instruction set. This is then used to generate the template switching instruction set based on the code changes of adjacent expression units and to keep the execution relationship between the main symbol style flow and the expression unit boundary synchronized at the interface layer.

[0039] Optionally, the expression intensity in the expression code triple sequence is used as the input to the intensity binding function. The adjustment coefficients of the style parameter vector output by the intensity binding function are used to adjust the style parameter vector. The style parameter vector is a six-dimensional vector containing color parameters, font weight parameters, outline parameters, background parameters, size parameters, and animation parameters. The binding with the expression intensity is completed within the constraints. The intensity binding function is as follows:

[0040] ;

[0041] in, This is the adjusted style parameter vector. These are the initial values ​​for the style parameter vector. This is a six-dimensional upper bound vector, where each dimension represents the upper limit value allowed by the style parameter of that dimension. This is a six-dimensional lower bound vector, where each dimension represents the lower bound value allowed by the style parameter of that dimension. To perform dimension-by-dimensional multiplication, the two six-dimensional vectors are multiplied one by one along each dimension. The hyperbolic tangent function has an output range of... , The slope control parameter for the binding curve is a positive real number. The center point of intensity is taken as the value of... Inside, This is a range clipping function that applies to each dimension of the input six-dimensional vector. The cropping rules ensure that each dimension of the result is no less than [the specified value]. and not higher than The formula is based on As a single numerical input, the intensity is converted into continuous style adjustments within a limited range, without changing the overall appearance. Take values, and ensure that each style parameter is within range. Stable landing within constraints.

[0042] Optionally, S4 specifically refers to:

[0043] The master symbol style stream and the expression code triple sequence are mapped one-to-one according to the expression unit boundary. For each expression unit, the master symbol template identifier and style parameter vector are read, and the corresponding expression code and expression intensity are read to establish a comparison pair of adjacent expression units.

[0044] Code change determination is performed on adjacent expression units. When the main symbol template identifiers are not equal, template switching is determined. When the main symbol template identifiers are equal but the expression intensity is not equal, style parameter vector update is determined. The determination result is bound to the expression unit boundary of adjacent expression units as the switching basis.

[0045] Using the boundary of the expression unit of adjacent expression units as the switching moment, the target main symbol template identifier and the target style parameter vector at the switching moment are added to the template switching instruction set, and the corresponding execution position is determined with reference to the style segment of the main symbol style stream.

[0046] The template switching instruction set is organized in the order of expression unit boundaries to keep the template switching instruction set consistent with the expression unit control cycle of the main symbol style stream, and is used for the interface layer execution of the subsequent subtitle style stream synthesis unit.

[0047] Optionally, during the process of determining code changes for adjacent representation units and adding the target master symbol template identifier and target style parameter vector at the switching moment to the template switching instruction set, and organizing the template switching instruction set according to the order of representation unit boundaries, a difference measurement formula is further introduced to determine the difference in target style parameter vectors between adjacent representation units as the basis for organization:

[0048] ;

[0049] in, This measures the style difference between adjacent records; it is a dimensionless numerical value. This is the dimension index of the style parameter vector, with a value of to , for In the The value of dimension is a normalization style parameter. The threshold for style fine-tuning is a dimensionless non-negative number. To extract the maximum value, a function is used to extract the maximum value among the six-dimensional component differences, and the final output is based on the expression unit as the control period. This ensures a one-to-one correspondence between the template switching instruction set and the style segments of the main symbol style stream, and that these correspond to the style segments in the interface layer of the subsequent subtitle style stream compositing unit. Triggering execution ensures that template switching and style parameter vector updates are strictly driven by the boundaries of the expression unit, maintaining linear control and operability of classroom speech stream and transcribed text sequence in the visual presentation link.

[0050] Optional, S5 specifically includes:

[0051] The expression code triple sequence is input into the subordinate annotation object generation unit. The expression code and expression intensity are judged unit by unit. The preset trigger class expression code set is matched with the expression code, and the triggering result is determined according to the intensity triggering threshold.

[0052] Based on the trigger result, an annotation type is generated, and the boundary of the expression unit where the trigger occurs is used as the boundary of the associated expression unit. The corresponding main symbol template identifier is used as the associated main symbol template identifier, thus forming the association relationship of the subordinate annotation objects.

[0053] The annotation type and association relationship of the subordinate annotation object are set at the boundary of the expression unit to set the start time and end time. With the constraint of not changing the main symbol template identifier and style parameter vector, the subordinate annotation object does not interfere with the main symbol style flow.

[0054] The generated subordinate annotation objects are submitted to the subtitle style stream compositing unit for overlay on the constrained main symbol style stream, and are consistent with the execution order of the template switching instruction set.

[0055] Optional, S6 specifically includes:

[0056] Align the transcribed text sequence with the main symbol style stream according to the boundaries of the expression units, and establish a one-to-one correspondence between text fragments and style segments based on the switching time in the template switching instruction set;

[0057] Based on the target main symbol template identifier and target style parameter vector of the template switching instruction set, the style segment of the corresponding expression unit is synthesized, the text fragment of the transcribed text sequence is embedded into the style segment to form the initial subtitle style stream, and the output is controlled by the expression unit;

[0058] The subordinate annotation objects are superimposed onto the initial subtitle style stream according to their associated expression unit boundaries. The start and end times of the subordinate annotation objects are set without changing the constraints of the main symbol template identifier and style parameter vector, and they are bound to the corresponding expression unit boundaries.

[0059] The superimposed subtitle style streams are organized according to the order of the expression unit boundaries, so that the execution order of the subtitle style streams is consistent with that of the template switching instruction set, and corresponds one-to-one with the style segments of the main symbol style stream, forming a subtitle style stream for execution at the interface layer.

[0060] The beneficial effects of this invention are:

[0061] 1. This proposal presents an improved expression code modeling method based on joint intent and prosodic constraints. Step-level speech-text pairs are constructed after framing the classroom speech stream. Prosodic vector sequences and semantic vector sequences are input into the prosodic encoder and intent encoder, respectively. A constraint fusion layer forms a constraint vector sequence strictly aligned with the step index, which serves as the conditional input to the expression code diffusion network. After iterative noise reduction within segments, the diffusion network synchronously decodes the expression code output head, expression intensity output head, and expression unit boundary output head. Furthermore, by combining the joint boundary confidence formula, boundary scale changes, intensity changes, and minimum boundary spacing constraints are jointly determined, thereby obtaining controllable expression unit boundaries even with fluctuations in classroom speech rate and pauses. This design ensures that expression information no longer relies solely on fixed time windows or simple sentence segmentation rules, but forms a triple sequence of "expression code—intensity—boundary," with the boundary defining the covered transcribed text segments. This provides a stable control cycle for subsequent style generation, reducing unnecessary switching and reading interruptions in subtitle presentation, and better meeting the needs of hearing-impaired students for continuous reading and key point capture.

[0062] 2. This proposal introduces a novel enhanced subtitle generation mechanism based on a single-master-channel visual grammar framework. It maps the expression codes in the expression code triples to unique master symbol template identifiers in a one-to-one manner, eliminating the uncertainty of template selection under the same semantic meaning. Simultaneously, the expression intensity is input into an intensity binding function. Through hyperbolic tangent mapping and upper / lower bound clipping, the six-dimensional style parameters of color, weight, outline, background, size, and animation are continuously modulated with limited amplitude, ensuring that style changes remain consistent with expression intensity and do not exceed limits. Based on this, a single-master-channel constraint layer is introduced, limiting the activation of only one master symbol template within any expression unit time period and fixing the master symbol style segment according to the expression unit boundary. Then, a template switching instruction set is generated based on the template identifiers and intensity differences of adjacent expression units, and the fine-tuning of adjacent styles is merged or suppressed using a difference metric. This mechanism differs from existing approaches such as "multi-element parallel overlay" or "frequent word-level style changes." It strictly constrains the switching of the master visual grammar at the expression unit boundary, reducing the risk of parallel style competition and flickering at the interface layer, ensuring that classroom subtitles maintain clear hierarchy and temporal consistency in real-time presentation.

[0063] 3. This proposal suggests an enhanced method for overall subtitle synthesis and rendering. Without modifying the transcribed text content, it aligns text fragments, the main symbol style stream, and the template switching instruction set according to the expression unit boundaries, forming a directly executable subtitle style stream. Furthermore, for a preset set of trigger-type expression codes, it generates subordinate annotation objects based on intensity trigger thresholds, clarifying their annotation type, start and end times, and their relationship with the expression unit boundaries and the main symbol template identifier. This is then overlaid with the constraint of "not changing the main symbol template identifier and style parameter vector." Thus, expressive cues such as questions, emphasis, or reminders in the classroom can be expressed in the form of subordinate annotations without interfering with the reading of the main subtitles. Simultaneously, the switching of the main symbol style is still uniformly controlled by the boundaries, ensuring that the interface execution order is synchronized with the audio timeline. This overall chain incorporates expression modeling, template mapping, switching instructions, and overlay annotations into the same boundary-driven temporal system, facilitating the stable output of enhanced subtitle frame sequences in real-time classroom scenarios, thereby supporting hearing-impaired students in more coherently following the lecture structure and key information. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of an enhanced captioning method based on an expression code diffusion model proposed in this invention;

[0066] Figure 2 This is a flowchart of the speech-text pair construction process for an enhanced captioning method based on an expression code diffusion model proposed in this invention.

[0067] Figure 3 The flowchart for generating expression code triplet sequences using the expression code diffusion model of the enhanced captioning method based on the expression code diffusion model proposed in this invention is shown below.

[0068] Figure 4 The flowchart of the main symbol style generation process of the single-master-channel visual grammar framework for an enhanced captioning method based on an expression code diffusion model proposed in this invention is shown.

[0069] Figure 5 This is a flowchart of the template switching instruction set generation process for an enhanced subtitle method based on an expression code diffusion model proposed in this invention.

[0070] Figure 6 This is a flowchart illustrating the generation of subordinate annotation objects in an enhanced captioning method based on an expression code diffusion model proposed in this invention.

[0071] Figure 7 This is a schematic diagram of four consecutive frames of classroom footage and enhanced subtitles for an enhanced subtitle method based on an expression code diffusion model proposed in this invention.

[0072] Figure 8 This is a left-right comparison image of a normal subtitle and an enhanced subtitle, representing an enhanced subtitle method based on an expression code diffusion model proposed in this invention.

[0073] Figure 9 This is a magnified close-up image of the subtitle region in an enhanced subtitle method based on an expression code diffusion model proposed in this invention.

[0074] Figure 10 This is a simplified schematic diagram illustrating the linkage between the expression unit segmentation bar and the screen subtitles in an enhanced subtitle method based on the expression code diffusion model proposed in this invention.

[0075] Figure 11 This is a schematic diagram of the expression code diffusion model structure of the enhanced captioning method based on the expression code diffusion model proposed in this invention. Detailed Implementation

[0076] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0077] refer to Figures 1 to 11 An enhanced captioning method based on an expression code diffusion model, characterized by comprising:

[0078] S1. Receive classroom audio stream and generate a transcribed text sequence to form audio-text pairs;

[0079] S2. Based on the speech-text pair, expressive code modeling is performed using the expressive code diffusion model to generate a sequence of expressive code triples. The expressive code triples include expressive code, expressive intensity, and expressive unit boundary. Under the constraints of intent and prosody, continuous speech is merged into expressive units, and the transcribed text fragments covered by the expressive unit boundary are defined.

[0080] S3. Based on the expression code triple sequence, construct the code template mapping of the single main channel visual syntax framework. Under the single main channel constraint, determine a unique main symbol template for each expression unit and bind the expression intensity to generate the main symbol style stream. The main symbol style stream corresponds to the boundary of the expression unit and serves as the basis for switching judgment.

[0081] S4. Determine the code changes of adjacent representation units based on the main symbol style flow and the representation code triplet sequence, and generate a template switching instruction set;

[0082] S5. Determine the set of preset trigger class expression codes based on the expression code triple sequence and generate subordinate annotation objects;

[0083] S6. Based on the transcribed text sequence, main symbol style stream, and template switching instruction set, synthesize the subtitle style stream and overlay the subordinate annotation objects;

[0084] S7. Output an enhanced subtitle frame sequence for interface presentation.

[0085] In this embodiment, step S1 specifically includes:

[0086] Using classroom audio streams as input, the original audio signal is represented as a continuous time series. ,in The amplitude-time function of the classroom audio stream is given by the frame length. Frame segmentation is performed to obtain a step index sequence, with a silence threshold. With the threshold of downtime As a criterion, silence and pause detection are performed on the frame sequence to generate a set of speech continuity segment boundaries. And based on this, divide the discourse into continuous segments. Calculate frame energy within each consecutive utterance segment. With fundamental frequency trajectory ,in For step index The energy value of the corresponding frame. To obtain the fundamental frequency value using short-time autocorrelation and peak search, when And the duration is not less than The time is recorded as a pause and a pause flag is set. ,otherwise ;

[0087] For prosody vector generation, a feature mapping function is used. By combining the basic features, the energy is normalized to Normalize the fundamental frequency to ,in and Calculate the energy slope for the fundamental frequency range of the classroom speech stream. With fundamental frequency slope Calculate the length of the outage operation. For the current step onwards The number of steps, in terms of context window length For the number of steps, , , The features are concatenated within the window to form a context feature stack vector. Using linear mapping With bias right Perform the transformation to obtain each step Prosody Vector Sequence ,in It is a combination vector of prosodic strength, pauses and intonation, and maintains continuity within a continuous segment of speech;

[0088] Speech recognition is used to identify classroom speech streams and generate transcribed text sequences. ,in For text step indexing The corresponding word symbol carries a timestamp. transcribe the text sequence according to The text is segmented to obtain a set of text fragments, retaining the timestamp range of each fragment. Semantic vectors are generated using a semantic mapping function. The relationship between word symbols and word order is encoded, and the word symbols are input into the word embedding matrix to obtain the word embeddings. , dimension The positional code is obtained by linearly transforming the text step index. , dimension The clause boundary marker will be encoded as , dimension Encode the phrase relationships as , dimension After concatenating the above vectors, a linear transformation is applied. With bias The semantic vector is calculated. and set Dimensions To form step-level semantic vectors, word timestamps are used. Mapped to frame step index, using weights Indicator In step The percentage of overlapping time within the range is defined. ,in For words With steps Duration of time overlap For steps The covered set of words is used to obtain the weighted average for each step. 3D semantic vector sequence ;

[0089] Based on the set of discourse continuum boundaries For prosodic vector sequences With transcribed text sequence Synchronization is performed by mapping the timestamp range of text segments to a step index range, establishing a one-to-one correspondence between each step, and mapping the prosodic vector of each step. With each step semantic vector Binding is performed to form a correspondence between voice and text pairs, and the voice-text pair is defined as... In discourse continuum Maintain the continuity of the binding relationship within the text, and use boundaries when crossing consecutive discourse segments. Reconstruct the binding sequence of the next segment for the segmentation point, so that the paragraph-level temporal structure of the speech-text pair is consistent with the boundary of the classroom speech stream;

[0090] Prosody vector sequence Semantic vector sequence and the resulting voice-text pairs As input, it is received by the intent encoder and prosodic encoder of the expression code diffusion model, in order to index the steps. To align with the reference, each step The input intent encoder will input each step Input prosody encoder, and with To control the input period by segment and ensure that the two types of codes are received synchronously within a segment, under this input relationship, the outputs of the intent encoder and the prosodic encoder are concatenated in the constraint fusion layer and linearly transformed into a sequence of constraint vectors for each step, thereby forming the generation conditions of intent and prosodic constraints. This provides a stable input basis for the expression code diffusion network to generate expression code triplet sequences at the segment granularity.

[0091] In this embodiment, step S2 specifically includes:

[0092] Using speech-text pairs as input, the sequence of step-level prosodic vectors is represented as follows: , dimension Describing prosodic strength, pauses, and intonation, the step-level semantic vector sequence is represented as... , dimension Describing semantic intent and phrase relationships, the step index is represented as... To maintain consistency with the classroom audio stream framing, the step-level data is processed within segments using continuous speech segments as the control period. This ensures stable step-level correspondences within segments for encoding and fusion. The speech-text pair is defined as... ;

[0093] Will Input the prosodic encoder to obtain the step-level prosodic code. ,Will Input the intent encoder to obtain the step-level intent code. Using a constraint fusion layer and By performing step-by-step concatenation and linear transformation, a step-level constraint vector sequence is obtained. , dimension ,Will As a conditional input to the expression code diffusion network, and maintaining consistency with The one-to-one correspondence, with the discourse sequence as the control period, will The continuous input expression code diffusion network within a segment keeps the constraint vector within the segment linked to the hidden state;

[0094] In the constraint vector sequence Under certain conditions, a code diffusion network is driven to iteratively optimize fragment representations. The network consists of eight sequentially stacked denoising blocks, each containing two feedforward layers and one conditional injection layer. The hidden states are updated step-by-step, with the hidden state dimension being... The initial hidden state is determined by Through linear projection, the conditional injection layer of each noise reduction block is injected at each step. This is to maintain the joint constraints between the classroom speech stream and the transcribed text sequence, and to form a stable iterative path within a segment;

[0095] After iterative optimization, the three decoding heads of the output layer generate the results step by step. The expression code output head generates the code distribution and selects the expression code of each step as the term with the highest probability. The size of the code set is denoted as . The intensity output head generates scalar intensity. The scope is limited to to The expression unit boundary output head generates a boundary scale vector. The above three outputs are used for subsequent boundary determination. They are consistent with the step index and aligned with the control cycle of the speech continuum to ensure consistency within the segment.

[0096] To achieve controllable determination of expression unit boundaries under the constraints of intention and prosody, a joint boundary confidence formula is adopted, defining the step-level boundary confidence as:

[0097] ;

[0098] in, For step-level boundary confidence, The weighting coefficients for the differences in the boundary scale vectors. The weighting coefficients are used to express the intensity changes. The penalty weight coefficient for the minimum spacing constraint. and For steps The boundary scale vector takes two-dimensional values. and For steps and the intensity of expression in the previous step, To express the minimum spacing threshold of the cell boundary, in steps, The step index for the most recently confirmed expression unit boundary. To obtain the maximum function, which is used to form the normalized insufficient spacing penalty term, the following is... With boundary condition threshold When comparing, Time stamp step For boundary candidates, when the candidate spacing is less than The candidates added after removal are removed to obtain an ordered and controlled set of expression unit boundary indices;

[0099] Based on the updated set of expression unit boundary indices, the discourse continuum is divided into expression units. Each representation unit covers a continuous range of step indices and defines the corresponding transcribed text segment. For each representation unit... The step-level representation code distribution within the unit is selected by majority vote to obtain the unit-level representation code. The intensity of step-level expressions is weighted and averaged, with the weights denoted as . To obtain the unit-level expression intensity Unit-level expression code Unit-level expression intensity Boundary of expression units Bind them to form a code triple. And organized into a sequence of expression code triples in the natural order of boundary indices. ;

[0100] Express code triple sequence As input to the code template mapping, and using the expression unit boundary as the switching criterion, changes in the expression code directly locate changes in the main symbol template. In the output chain, only one expression unit is output as the control cycle. By maintaining a consistent order between the triplet sequence and the intra-segment step index, the expression code diffusion network provides a clear expression unit control cycle and an operable switching criterion for the generation of style segments in the main symbol style flow. Through the above implementation process, the expression code diffusion network synchronously generates expression codes, expression intensity, and expression unit boundaries under the constraints of intent and prosody. It also achieves controllable boundary updates at the paragraph level using a single calculation rule of joint boundary confidence, forming an expression code triplet sequence that can directly drive code template mapping. This is used for the robust generation of the main symbol style flow and template switching instruction set in classroom scenarios.

[0101] In this embodiment, step S3 specifically includes:

[0102] Take the sequence of triples representing the code as input, and denote each triple as... ,in For unit-level expression code, For unit-level expression intensity, The step index interval covered by the expression unit boundary is processed unit by unit according to the natural order of triples, with the expression unit as the control cycle, to ensure that the alignment between the main symbol style flow and the expression unit boundary is not broken.

[0103] Will Input the code template mapping library and obtain the main symbol template identifier using a one-to-one binding method. Initial values ​​of the style parameter vector ,definition It is a six-dimensional vector, with the dimensions listed in order as follows: , , , , , These represent the color parameter, font weight parameter, outline parameter, background parameter, size parameter, and animation parameter, respectively. The six-dimensional parameters use a normalized scale, and by convention... , , , , The range of values ​​is to , The normalized scale relative to the base font size has a value range of [value range missing]. to Through the above binding, a unique mapping relationship is established between the expression code and the main symbol template identifier, eliminating the uncertainty of template selection under the same expression code;

[0104] Will Input the strength binding function, output the adjusted style parameter vector:

[0105] ;

[0106] in, This is the adjusted style parameter vector. These are the initial values ​​for the style parameter vector. This is a six-dimensional upper bound vector, where each dimension represents the upper limit value allowed by the style parameter of that dimension. This is a six-dimensional lower bound vector, where each dimension represents the lower bound value allowed by the style parameter of that dimension. To perform dimension-by-dimensional multiplication, the two six-dimensional vectors are multiplied one by one along each dimension. The hyperbolic tangent function has an output range of... , The slope control parameter for the binding curve is a positive real number. The center point of intensity is taken as the value of... Inside, This is a range clipping function that applies to each dimension of the input six-dimensional vector. The cropping rules ensure that each dimension of the result is no less than [the specified value]. and not higher than The formula is based on As a single numerical input, the intensity is converted into continuous style adjustments within a limited range, without changing the overall appearance. Take values, and ensure that each style parameter is within range. Stable landing within constraints;

[0107] by To control the cycle, based on and The style segment that generates the main symbol style flow is denoted as style segment. , for each Carry the main symbol template identifier and style parameter vector, and press... The start and end indexes are located on the timeline to ensure a one-to-one correspondence between style segments and expression unit boundaries, and to address adjacent... and Adopting a non-overlapping stitching strategy, End step index and The starting step index is the splicing point to maintain the orderly output of style segments;

[0108] The sequentially generated master symbol style stream is input into the single master channel constraint layer, in each... Only one main symbol template identifier is activated within. When an external trigger causes concurrent activation requests, a priority overriding rule is used, with the current priority taking precedence. corresponding As a unique master symbol template, other activation requests are delayed until the next expression unit boundary to enter the decision queue, thereby eliminating parallel rendering. The single master channel constraint layer outputs the constrained master symbol style flow, maintaining a linear output relationship with the expression unit as the control cycle.

[0109] Based on the constrained main symbol style flow and Establish switching criteria for each The starting boundary determines the switching target, and and Bundle them as a target pair, and The starting step index is used as the switching trigger position for... arrive The transition, if Then, template switching is performed at the boundary. and Then, the style parameter vector is updated at the boundary. If both are the same, they remain unchanged. Cross-segment drift is eliminated through the boundary-driven unified rule.

[0110] The constrained main symbol style flow in Curing is carried out on the surface; curing refers to the process of curing on the surface. Keep all steps within the index and Without mid-process changes, the time series of the main symbol style stream is output through step-by-step index sampling, creating a one-to-one mapping between the main symbol style stream and the expression unit boundaries. Through a solidification strategy, changes in the expression code are directly located to changes in the main symbol template, and the expression unit is used as the sole control period, avoiding changes during the process. The display is unstable due to multiple internal switching issues.

[0111] The constrained main symbol style flow and the aforementioned switching determination criteria are sent to the template switching instruction set generation unit to generate input for the interface layer execution. Generate a candidate record, the target master symbol template of which is identified as follows: The target style parameter vector is Switch time location to The starting boundary is used to organize adjacent candidate records according to the order of expression units, and adjacent records are merged when there is template or pattern continuity. The output record sequence is used to drive the generation of subsequent template switching instruction sets. The above process is based on... As the sole driving source, relying on the one-to-one binding path provided by the code template mapping library, and through the limited amplitude adjustment of the strength binding function and the single template activation control of the single master channel constraint layer, a process is formed according to... The switching of the main symbol style stream provides a directly executable, time-sequential input for generating the template switching instruction set, ensuring the consistency and controllability of the classroom speech stream and the transcribed text sequence in the visual presentation link.

[0112] In this embodiment, step S4 specifically includes:

[0113] The main symbol style stream and the sequence of expression code triples are mapped one-to-one according to the boundaries of the expression units. Let the expression code triples be denoted as... ,in For unit-level expression code, For unit-level expression intensity, To represent the step index range covered by the unit boundary, let the style segment of the main symbol style flow be denoted as . ,in Main symbol template identifier, This is a six-dimensional style parameter vector, with the expression unit as the control period, and each... and Binding is performed so that the main symbol template identifier and style parameter vector are aligned with the expression code and expression intensity. Inner alignment, and with Output style segments for time positioning reference;

[0114] Adjacent expression units are paired, and denoted as... and read the corresponding style segment. and Used at the boundary Code change detection is performed at this point, and an intensity update threshold is set. This is used to suppress invalid updates caused by minor intensity fluctuations, and performs a judgment on adjacent expression units: when When the template is confirmed to be switched, and When it is confirmed that the style parameter vector is updated, and Without switching or updating, the judgment result will be compared with... Binding is used as the basis for switching, and the corresponding target main symbol template identifier and target style parameter vector are recorded to ensure that the judgment triggered by the interface layer at the boundary is unique;

[0115] Using the boundary of adjacent representation units as the switching time, the switching time step index is denoted as . When the template switch is successful, set the target master symbol template identifier to... Set the target style parameter vector to It also generates a switching record, and when the style parameter vector is updated, it retains the target main symbol template identifier. and with Record updates, using the style segment of the main symbol style flow. For reference, the execution location of the record switching is located at... At the beginning of the corresponding style block, ensure that the execution position is strictly consistent with the timeline of the style block;

[0116] The above switching records are organized according to the boundary order of the representation units into a template switching instruction set, denoted as _____. ,in For the first Each record contains a switching time step index, a target master symbol template identifier, and a target style parameter vector. Perform deduplication and merging: when adjacent records satisfy and When merging styles into the latter and removing the former, and when only style updates occur consecutively near the boundary, the style difference metric is calculated based on the difference metric formula. ,like Then retain the last update and remove the previous record:

[0117] ;

[0118] in, This measures the style difference between adjacent records; it is a dimensionless numerical value. This is the dimension index of the style parameter vector, with a value of to , for In the The value of dimension is a normalization style parameter. The threshold for style fine-tuning is a dimensionless non-negative number. To extract the maximum value, a function is used to extract the maximum value among the six-dimensional component differences, and the final output is based on the expression unit as the control period. This ensures a one-to-one correspondence between the template switching instruction set and the style segments of the main symbol style stream, and that these correspond to the style segments in the interface layer of the subsequent subtitle style stream compositing unit. Triggered execution ensures that template switching and style parameter vector updates are strictly driven by the boundaries of the expression unit, maintaining linear control and operability of the classroom speech stream and transcribed text sequence in the visual presentation chain.

[0119] In this embodiment, step S5 specifically includes:

[0120] Take the sequence of triples representing the code as input, and denote each triple as... ,in For unit-level expression code, For unit-level expression intensity, The step index interval covered by the expression unit boundary, with the expression unit as the control period, for each The determination is performed on a per-expression-unit basis, using a preset set of trigger class expression codes and a set of intensity trigger thresholds for matching and judgment. The preset set of trigger class expression codes is denoted as... It contains the expressive code that is allowed to trigger subordinate labeled objects, and the intensity trigger threshold uses a mapping table. It provides a directly usable numerical threshold for each expression code;

[0121] In the per-representation unit determination, the main symbol template identifier and style parameter vector of the current representation unit are read and denoted as follows: and Both of these originate from the style segment of the main symbol style flow. ,by and Based on binding, within the scope of expression units The system calculates and records triggers, and uses a combined rule formula of code matching and strength gating to determine triggers:

[0122] ;

[0123] in, If and only if ,otherwise ,when When the trigger is confirmed, the trigger result is "true". If the trigger is deemed invalid, the above determination is based on the matching of the expression code set and the comparison of the intensity with real numbers, ensuring that the triggering process relies only on directly computable underlying values ​​and is consistent with the control cycle of the expression unit. For the first Triggering score for each expression unit, Match the code to an indicator value, which is either zero or one. For the first The expression intensity of each expression unit, with a value range of [value range missing]. to , For expression code The corresponding intensity trigger threshold is a normalized value. This is a set of code for pre-defined trigger classes, and a set of code for offline configuration. For the first The expression code of each expression unit;

[0124] When the trigger is established, a label type is generated for the subordinate label object, denoted as... The annotation type is determined based on the mapping table from the expression code to the annotation type, so that triggers of the same category have a unified annotation semantic at the interface layer, and the boundary of the expression unit where the trigger is located is defined. This serves as the boundary of the associated expression unit, and identifies the main symbol template bound to that expression unit. As a primary symbol template identifier, it establishes the association relationship between subordinate annotation objects and includes the annotation type. , boundary of related expression unit Associated main symbol template identifier ;

[0125] Set the start and end times of subordinate annotation objects at the boundary of the expression unit, and denote the start time as... The termination time is recorded as To avoid changing the main symbol template identifier With style parameter vector To constrain this, subordinate labeled objects will be used as overlays within the time interval. Internal output, thus not changing the template selection and style parameter values ​​of the main symbol style flow. For triggering across expression units, a unit-by-unit independent setting method is adopted to avoid introducing new switching points within the expression unit and maintain the linear output relationship of the single main channel constraint.

[0126] The generated subordinate annotation objects are submitted to the subtitle style stream compositing unit. During the submission process, a record is made for each subordinate annotation object. Alignment is performed with reference to the execution order of the template switching instruction set. During interface layer compositing, the constrained main symbol style flow is used as the bottom layer, and subordinate annotation objects are superimposed in the order of the expression unit boundaries without changing the underlying main symbol template identifier and style parameter vector. For annotation types of the same type that appear consecutively in adjacent expression units, if their associated main symbol template identifiers are consecutive in time, they are rendered continuously during the compositing stage, but still based on the execution order of the template switching instruction set. It is an independent control region that does not cross the boundaries of expression units;

[0127] Through the above implementation process, the subordinate annotation object generation unit takes the expression code triple sequence as input, uses code matching and intensity gating to jointly determine the annotation type, and establishes a one-to-one association with the expression unit boundary and the main symbol template identifier. Without changing the main symbol style flow, the subordinate annotation objects are superimposed within the time interval of the expression unit, maintaining consistency with the execution order of the template switching instruction set, forming a temporal input that can be directly used for the subtitle style flow synthesis unit, ensuring the consistency and controllability of the classroom speech stream and the transcribed text sequence in the visual presentation link.

[0128] In this embodiment, step S6 specifically includes:

[0129] Align the transcribed text sequence with the main symbol style stream according to the expression unit boundaries, and read the template switching instruction set, denoted as . ,in For the first One switching record, To switch the step index corresponding to the time step, Identify the target main symbol template. Let the target style parameter vector be defined, with the expression unit boundary as the segmentation reference, and let the expression unit boundary be denoted as . , For the step index range, with Establish a one-to-one correspondence between switching moments and expression units, and transcribe the text sequence according to... The extracted text fragment is denoted as _____. , For the step index range A sequence of characters concatenated in an internal order, used to embed the corresponding style segment;

[0130] Based on the target master symbol template identifier and target style parameter vector of the template switching instruction set, the style segment of the corresponding expression unit is synthesized, and an initial subtitle style segment is constructed for each expression unit:

[0131] ;

[0132] in, This is the initial subtitle style segment quadruple. To represent the step index interval covered by the boundary of the expression unit, Identify the target main symbol template. The target style parameter vector is a six-dimensional normalized numerical value. For the text fragment to be presented, Outputting in the natural order of expression units forms an initial subtitle style stream with expression units as the control cycle. If adjacent... of and If consistent with the template switching instruction set, there will be no overlapping splicing, and the boundaries will be marked with... Position the execution at the starting position;

[0133] Subordinate annotation objects are superimposed onto the initial caption style stream according to the boundaries of their associated representation units. Any subordinate annotation object is denoted as... ,in For annotation type, To associate the boundaries of expression units, For the associated main symbol template identifier, To enable the step index corresponding to the start time, For the step index corresponding to the termination time, for all satisfying The subordinate annotation object, without changing and Under the constraints, The annotations within are displayed in Above, if the same Memory in multiple Then according to Stack in ascending order, maintaining the same order as and Conflict-free overlay, without introducing new template switching or style parameter vector updates;

[0134] The superimposed subtitle style stream is organized according to the order of the expression unit boundaries, and the final subtitle style segment is output, denoted as... ,in In order to be in The style segment record is superimposed on the subordinate annotation object. In order to be in The set of annotation types and their time intervals that are effective within the scope will according to The natural order of the text is organized into a subtitle style flow, along with a template switching instruction set. The execution order remains consistent and corresponds one-to-one with the style segments of the main symbol style flow. The interface layer follows the order of execution. middle Rendering scheduling is performed based on the start and end indexes. When a template switch or style update is triggered, the text fragment should be preserved. and , and subordinate labeled objects Consistency across the timeline;

[0135] In the above process, To transcribe the text sequence, no modifications to its content are introduced; only operations are performed according to... Fragmentation, Provides template switching instruction set and The direct value, Provides a collection of subordinate labeled objects , , The direct value, in Using the sole segmentation benchmark, it completes the one-to-one correspondence between text fragments and style segments, the generation and overlay of style segments, and the alignment with the order of template switching instruction sets, outputting a subtitle style stream for execution at the interface layer.

[0136] In this embodiment, step S7 specifically includes:

[0137] Taking the sequence of style segments from the subtitle style stream as input, let the style segment be denoted as ,in To represent the step index interval covered by the boundary of the expression unit, Identify the target main symbol template. For the target style parameter vector, For text fragments, In order to be in The set of annotation types and their time intervals that are effective within the specified time period, and the template switching instruction set are read. ,in To switch records, To switch the step index corresponding to the time step, To find the minimum value function, let the sampling time for each step be... Let the frame rate of the interface layer be... Used for timeline alignment;

[0138] Align the transcribed text sequence with the main symbol style flow according to the expression unit boundaries, for each Calculate start and end times, define and , To find the maximum value function, with To divide the time intervals, establish a one-to-one correspondence between text segments and style segments, and use... In As the starting point for segmentation, used to trigger template switching or style parameter vector updates, the alignment operation mentioned above only involves index reading and numerical multiplication, and does not change... and ;

[0139] according to and Synthesize the style segments corresponding to the expression units, and As a normalization style control variable, for text fragments Configure rendering settings: Set the main symbol color to Set word weight to Set the contour thickness to Set the background transparency or brightness to Set font size scaling to Set the animation progress or speed within a time range. Keep the above settings unchanged, and Embedded style segments output initial subtitle style stream fragments according to expression units as the control cycle;

[0140] Subordinate annotation objects are superimposed onto the initial caption style stream according to the boundaries of their associated representation units. Any subordinate annotation object is denoted as... ,in For annotation type, To associate the boundaries of expression units, For the associated main symbol template identifier, To enable the step index corresponding to the start time, For the step index corresponding to the termination time, for all satisfying The subordinate annotation object, without changing and Under the constraints, the time interval The annotations within are displayed in Above this, a rendering hierarchy is adopted that is higher than the background but lower than the main symbol text, without introducing new template switching or style parameter vector updates. Memory in multiple Then according to Stack in ascending order, maintaining consistency with the timeline of the expression units;

[0141] Organize and output the final subtitle style stream frame by frame, defining frame indices. , Let be the set of natural numbers, represent the non-negative integer index, and define the frame time as . For each In the interval Execution per frame: Loading and As the main symbol style setting, Rendering is done on the main symbol text layer, for all that satisfy... Subordinate annotation objects are overlaid on top of the main symbol text layer, without rendering other main symbol templates in parallel, and in... Moment app Switching records in Maintain a one-to-one correspondence between the style segments and the main symbol style flow;

[0142] All frames are organized into a subtitle style frame sequence according to the order of the expression unit boundaries, denoted as . ,in For at any time The rendered single-frame image, with As the sole segmentation benchmark, and The execution order remains consistent to ensure that... Trigger template switching or style parameter vector updates at the specified location, and in Overlapping subordinate labeled objects, ultimately with As the direct execution input of the interface layer, the output enhances the subtitle frame sequence, maintaining the linear control relationship between the classroom audio stream and the transcribed text sequence in the visual presentation link.

[0143] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An enhanced captioning method based on an expression code diffusion model, characterized in that, include: S1. Receive classroom audio stream and generate a transcribed text sequence to form audio-text pairs; S2. Based on the speech-text pair, expressive code modeling is performed using the expressive code diffusion model to generate a sequence of expressive code triples. The expressive code triples include expressive code, expressive intensity, and expressive unit boundary. Under the constraints of intent and prosody, continuous speech is merged into expressive units, and the transcribed text fragments covered by the expressive unit boundary are defined. S3. Based on the expression code triple sequence, construct the code template mapping of the single main channel visual syntax framework. Under the single main channel constraint, determine a unique main symbol template for each expression unit and bind the expression intensity. Generate the main symbol style stream. The main symbol style stream corresponds to the boundary of the expression unit and serves as the basis for switching judgment. S4. Determine the code changes of adjacent representation units based on the main symbol style flow and the representation code triplet sequence, and generate a template switching instruction set; S5. Determine the set of preset trigger class expression codes based on the expression code triple sequence and generate subordinate annotation objects; S6. Based on the transcribed text sequence, main symbol style stream, and template switching instruction set, synthesize the subtitle style stream and overlay the subordinate annotation objects; S7. Output an enhanced subtitle frame sequence for interface presentation.

2. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S1 specifically refers to: The system receives classroom audio streams, divides them into speech segments, and extracts prosodic strength, pauses, and intonation from the audio streams to generate a prosodic vector sequence, maintaining the continuity of prosodic strength, pauses, and intonation within the speech segments. Speech recognition is performed on the classroom speech stream to generate a transcribed text sequence. Semantic intent and word phrase relationships are extracted from the transcribed text sequence to generate a semantic vector sequence. The transcribed text sequence is then segmented into text fragments according to continuous speech segments. Synchronize the prosodic vector sequence and the transcribed text sequence according to the continuous speech segments, establish a one-to-one correspondence at each step, and bind the prosodic vector at each step with the semantic vector at each step to form a correspondence between speech and text pairs. The prosodic vector sequence and the transcribed text sequence and their correspondence are taken as input and received by the intention encoder and prosodic encoder of the expression code diffusion model. The prosodic vector and semantic vector corresponding to each step in the speech-text pair are identified to form a speech-text pair.

3. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S2 specifically refers to: The prosodic vector sequence and semantic vector sequence in the speech-text pair are input into the prosodic encoder and the intent encoder, respectively. The corresponding prosodic code and intent code are output at each step, and the two types of codes are used as input to the constraint fusion layer. A constraint fusion layer is used to concatenate and linearly transform the intent encoding and prosodic encoding to generate a constraint vector sequence for each step. The constraint vector sequence is used as the conditional input of the expression code diffusion network and maintains the step correspondence with the speech-text pair. Under the condition of constraint vector sequence, the expression code diffusion network is driven to iteratively optimize the fragment representation of continuous speech segments. The segments are sequentially passed through eight noise reduction blocks. In the conditional injection layer of each noise reduction block, the constraint vector of each step is added to the hidden state to maintain the linkage. After iterative optimization, each step is decoded using the expression code output head, expression intensity output head, and expression unit boundary output head to obtain the expression code, expression intensity, and boundary scale vector, and the expression unit boundary is determined based on the boundary scale vector. Code changes are determined based on the continuous change in expression intensity and the boundary condition threshold. When the change in expression intensity reaches the boundary condition threshold, the expression unit boundary is updated, and the continuous speech segment is divided into expression units according to the updated expression unit boundary. Continuous discourse is grouped into expression units according to the expression unit boundaries, and the transcribed text fragments covered by the expression unit boundaries are defined. The expression codes and expression intensities obtained by the determination are respectively bound to the corresponding expression unit boundaries to generate expression code triples. The generated expression code triple sequence is used as the input to the code template mapping, and the expression unit boundary is used as the switching criterion, so that the change of expression code is directly located to the change of main symbol template, and the expression unit control cycle result is provided for the subsequent main symbol style stream output.

4. The enhanced captioning method based on an expression code diffusion model according to claim 3, characterized in that, Under the constraint of the vector sequence, the expression code diffusion network is driven to complete iterative optimization. After iterative optimization, the expression code output head, expression intensity output head, and expression unit boundary output head are used to decode each step to obtain the expression code, expression intensity, and boundary scale vector. In the process of determining the expression unit boundary based on the boundary scale vector, a joint boundary confidence formula is further introduced to jointly determine the boundary scale vector and expression intensity as the basis for determining the expression unit boundary. ； in, For step-level boundary confidence, The weighting coefficients for the differences in the boundary scale vectors. The weighting coefficients are used to express the intensity changes. The penalty weight coefficient for the minimum spacing constraint. and For steps The boundary scale vector takes two-dimensional values. and For steps and the intensity of expression in the previous step, To express the minimum spacing threshold of the cell boundary, in steps, The step index for the most recently confirmed expression unit boundary. To obtain the maximum function, which is used to form the normalized insufficient spacing penalty term, the following is... With boundary condition threshold When comparing, Time stamp step For boundary candidates, when the candidate spacing is less than The candidates added after removal are removed, resulting in an ordered and controlled set of expression unit boundary indices.

5. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S3 specifically refers to: The expression code in the expression code triple sequence is input into the code template mapping, a master symbol template identifier is generated using a one-to-one binding method, and the initial values ​​of the style parameter vector are output to establish a unique mapping relationship between the expression code and the master symbol template identifier; The expression intensity in the expression code triple sequence is used as the input of the intensity binding function. The adjustment coefficient of the style parameter vector output by the intensity binding function is used to adjust the style parameter vector. The style parameter vector is a six-dimensional vector containing color parameters, font weight parameters, outline parameters, background parameters, size parameters, and animation parameters. The binding with the expression intensity is completed within the constraints. Using the boundary of the expression unit in the expression code triple sequence as the control period, the style segment of the main symbol style flow is generated according to the main symbol template identifier and the adjusted style parameter vector, so that each style segment corresponds one-to-one with the corresponding expression unit boundary, and carries the main symbol template identifier and style parameter vector in each style segment; The main symbol style stream is input into a single main channel constraint layer. Only one main symbol template is activated within the same time period according to the expression unit boundary. The constrained main symbol style stream is output. The constrained main symbol style segment is output with the expression unit as the control period and multiple main symbol templates are not presented in parallel. The switching criteria are established based on the constrained main symbol style flow and the expression unit boundary. The main symbol template identifier and style parameter vector corresponding to each expression unit boundary are bound as the switching target, and the expression unit boundary is used as the sole benchmark for the switching criteria. The constrained main symbol style flow is solidified at the boundary of the expression unit, so that the change of the expression code is directly located to the change of the main symbol template, and the expression unit control cycle of the main symbol style flow is formed, maintaining the linear output relationship of the main symbol style flow under the constraint of a single main channel; The constrained main symbol style flow and the switching decision criteria are used as inputs to generate the template switching instruction set. This is used to generate the template switching instruction set based on the code changes of adjacent expression units and to keep the execution relationship between the main symbol style flow and the expression unit boundary in the interface layer synchronized.

6. The enhanced captioning method based on an expression code diffusion model according to claim 5, characterized in that, The expression intensity in the expression code triple sequence is used as input to the intensity binding function. The intensity binding function outputs adjustment coefficients for the style parameter vector, which is then used to adjust the style parameter vector. The style parameter vector is six-dimensional, containing color parameters, font weight parameters, outline parameters, background parameters, size parameters, and animation parameters. Binding to the expression intensity is completed within constraints. The intensity binding function is specifically as follows: ； in, This is the adjusted style parameter vector. These are the initial values ​​for the style parameter vector. This is a six-dimensional upper bound vector, where each dimension represents the upper limit value allowed by the style parameter of that dimension. This is a six-dimensional lower bound vector, where each dimension represents the lower bound value allowed by the style parameter of that dimension. To perform dimension-by-dimensional multiplication, the two six-dimensional vectors are multiplied one by one along each dimension. The hyperbolic tangent function has an output range of... , The slope control parameter for the binding curve is a positive real number. The center point of intensity is taken as the value of... Inside, This is a range clipping function that applies to each dimension of the input six-dimensional vector. The cropping rules ensure that each dimension of the result is no less than [the specified value]. and not higher than The formula is based on As a single numerical input, the intensity is converted into continuous style adjustments within a limited range, without changing the overall appearance. Take values, and ensure that each style parameter is within range. Stable landing within constraints.

7. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S4 specifically refers to: The master symbol style stream and the expression code triple sequence are mapped one-to-one according to the expression unit boundary. For each expression unit, the master symbol template identifier and style parameter vector are read, and the corresponding expression code and expression intensity are read to establish a comparison pair of adjacent expression units. Code change determination is performed on adjacent expression units. When the main symbol template identifiers are not equal, template switching is determined. When the main symbol template identifiers are equal but the expression intensity is not equal, style parameter vector update is determined. The determination result is bound to the expression unit boundary of adjacent expression units as the switching basis. Using the boundary of the expression unit of adjacent expression units as the switching moment, the target master symbol template identifier and the target style parameter vector at the switching moment are added to the template switching instruction set, and the corresponding execution position is determined with reference to the style segment of the master symbol style stream. The template switching instruction set is organized in the order of expression unit boundaries to keep the template switching instruction set consistent with the expression unit control cycle of the main symbol style stream, and is used for the interface layer execution of the subsequent subtitle style stream synthesis unit.

8. The enhanced captioning method based on an expression code diffusion model according to claim 7, characterized in that, The code change determination is performed on adjacent representation units, and the target master symbol template identifier and target style parameter vector at the switching moment are added to the template switching instruction set. In the process of organizing the template switching instruction set according to the expression unit boundary, a difference measurement formula is further introduced to determine the difference of the target style parameter vector of adjacent representation units as the basis for organization: ； in, This measures the style difference between adjacent records; it is a dimensionless numerical value. This is the dimension index of the style parameter vector, with a value of to , for In the The value of dimension is a normalization style parameter. The threshold for style fine-tuning is a dimensionless non-negative number. To extract the maximum value, a function is used to extract the maximum value among the six-dimensional component differences, and the final output is based on the expression unit as the control period. This ensures a one-to-one correspondence between the template switching instruction set and the style segments of the main symbol style stream, and that these correspond to the style segments in the interface layer of the subsequent subtitle style stream compositing unit. Triggered execution ensures that template switching and style parameter vector updates are strictly driven by the boundaries of the expression unit, maintaining linear control and operability of classroom speech stream and transcribed text sequence in the visual presentation link.

9. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S5 specifically refers to: The expression code triple sequence is input into the subordinate annotation object generation unit. The expression code and expression intensity are judged on a unit-by-unit basis. The preset trigger class expression code set is matched with the expression code, and the triggering result is determined according to the intensity triggering threshold. Based on the trigger result, an annotation type is generated, and the boundary of the expression unit where the trigger occurs is used as the boundary of the associated expression unit. The corresponding main symbol template identifier is used as the associated main symbol template identifier, thus forming the association relationship of the subordinate annotation objects. The annotation type and association relationship of the subordinate annotation object are set at the boundary of the expression unit to set the start time and end time. With the constraint of not changing the main symbol template identifier and style parameter vector, the subordinate annotation object does not interfere with the main symbol style flow. The generated subordinate annotation objects are submitted to the subtitle style stream compositing unit for overlay on the constrained main symbol style stream, and are consistent with the execution order of the template switching instruction set.

10. The enhanced captioning method based on an expression code diffusion model according to claim 1, characterized in that, S6 specifically refers to: Align the transcribed text sequence with the main symbol style stream according to the boundaries of the expression units, and establish a one-to-one correspondence between text fragments and style segments based on the switching time in the template switching instruction set; Based on the target main symbol template identifier and target style parameter vector of the template switching instruction set, the style segment of the corresponding expression unit is synthesized, the text fragment of the transcribed text sequence is embedded into the style segment to form the initial subtitle style stream, and the output is controlled by the expression unit; The subordinate annotation objects are superimposed onto the initial subtitle style stream according to their associated expression unit boundaries. The start and end times of the subordinate annotation objects are set without changing the constraints of the main symbol template identifier and style parameter vector, and they are bound to the corresponding expression unit boundaries. The superimposed subtitle style streams are organized according to the order of the expression unit boundaries, so that the execution order of the subtitle style streams is consistent with that of the template switching instruction set, and corresponds one-to-one with the style segments of the main symbol style stream, forming a subtitle style stream for execution at the interface layer.

Images