A method for generating real-time dialogue subtitles for doctors and patients

Abstract

The application discloses a doctor-patient real-time dialogue subtitle generation method. The method comprises the following steps: obtaining audio to be generated subtitles and subtitle text generation parameters; according to the subtitle text generation parameters, the audio to be generated subtitles is windowed to obtain an acoustic feature sequence; the acoustic feature sequence is input into a streaming encoder to obtain acoustic coding; a double-branch decoding model is used to generate connection time sequence classification text and recurrent neural network text according to the acoustic coding and the subtitle text generation parameters; and the connection time sequence classification text and the recurrent neural network text are screened and combined according to the connection time sequence classification text, the recurrent neural network text and the subtitle text generation parameters to obtain generated subtitle text. The application takes into account real-time performance, readability and accuracy.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and speech recognition technology, and more specifically, to a method for generating captions for real-time doctor-patient dialogue. Background Technology

[0002] Streamed Automatic Speech Recognition (ASR) is a key supporting technology for online scenarios such as real-time doctor-patient dialogue captioning, conference transcription, and simultaneous interpretation. Currently, in the typical online scenario of real-time doctor-patient dialogue captioning, the following shortcomings still exist: I. The inherent contradiction between end-to-end latency and subtitle stability lies in the inherent conflict between real-time performance and readability. Doctor-patient dialogues often involve fluctuating speech rates, accent differences, background noise, rapid switching between multiple rounds of questions and answers, and continuous long sentences. However, streaming recognition can only utilize a portion of the audio evidence up to the current point in time, especially at the end of phrases, sentence boundaries, and in stages where semantics are not yet closed. The output hypothesis is highly sensitive to subsequent audio evidence, easily being replaced, rolled back, or supplemented as new evidence arrives. If the system adopts a more aggressive submission strategy to reduce latency, the already displayed subtitles will be frequently revised, causing visual jitter and affecting the continuity of reading and information reception efficiency in doctor-patient communication. Conversely, if a more conservative waiting strategy is adopted to suppress revisions, significant output lag will be introduced, weakening the interactive rhythm and on-site usability of real-time dialogue. Therefore, the system struggles to achieve a stable and consistent balance between low latency and low jitter.

[0003] Second, the key reasoning and submission mechanisms lack adaptive scheduling capabilities, making it difficult to implement differentiated control for different segments. Existing systems mostly use fixed window lengths, fixed right context values ​​(lookahead), and static submission boundary rules to constrain online reasoning and display behavior, supplemented by stabilization strategies based on indicators such as time, length, confidence, endpoint detection, or output consistency. These methods are simple to implement in engineering, but they often only allow post-processing constraints on the results using threshold rules, lacking explicit state variables and control signals that can characterize segment attributes and subtitle output risks, making it difficult to form an interpretable dynamic adjustment path. Since different segments in doctor-patient dialogues dynamically change in terms of pronunciation clarity, semantic certainty, and ambiguity, static configuration is prone to premature submission and frequent revisions on unstable segments, while it may cause unnecessary waiting and increase latency on relatively stable segments. Meanwhile, the system struggles to implement targeted control over critical situations in the subtitle layer, such as maintaining readability at sentence boundaries, suppressing premature submissions during semantically unclosed phases, controlling misalignment and rollback during round switching, and adopting a more cautious confirmation strategy when highly sensitive content such as proper names, numbers, and medical terms appears. As a result, the controllability and consistency of the subtitle generation process are insufficient under complex and dynamic conditions.

[0004] Third, the imperfect display layer output protocol and key content protection mechanism lead to uncontrollable subtitle update behavior and consistency risks. Special names, drug names, examination items, indicator values, dosage units, and time information in doctor-patient dialogue subtitles have high information value and high error correction costs. Some real-time subtitle systems directly and continuously refresh intermediate recognition results to the interface, or only provide coarse-grained, phased submissions, lacking layered output and constraints for draft and final subtitles, and lacking clear definitions of the scope of possible revisions and rollback boundaries. This allows areas already read by the user to be modified, causing significant reading interference and reducing comprehension efficiency. When subsequent decoding results conflict with submitted subtitles, the lack of consistency maintenance and conflict handling rules can easily lead to inconsistencies between subtitle display and downstream processing links. For example, functions relying on stable text, such as dialogue summary generation, keyword triggering, structured recording, and index archiving, may be affected. Existing stabilization strategies often do not distinguish between fragment types and information sensitivity, lacking confirmation and protection mechanisms for key entities. This makes the impact of key content being misidentified or frequently revised more significant, thereby weakening the credibility, usability, and consistency of subsequent recordings of the subtitles.

[0005] In summary, existing methods suffer from drawbacks such as the inability to balance latency and subtitle stability, the inability to adaptively adjust to the text being processed, and inconsistencies in the logic before and after the subtitles.

[0006] Existing technology discloses a training method, a speech intent recognition method, and an apparatus for a speech intent recognition model. The training method includes: acquiring a text sample and a first speech sample carrying semantic labels, wherein the content of the first speech sample corresponds to that of the text sample, and the semantic labels are the text semantic features of the text sample; using the first speech sample, pre-training the semantic extraction network in the speech intent recognition model to be trained to obtain a pre-trained speech intent recognition model, wherein the pre-trained speech intent recognition model includes a pre-trained semantic extraction network and an intent recognition network to be trained; acquiring a second speech sample carrying intent labels; and using the second speech sample, training the pre-trained speech intent recognition model to obtain a trained speech intent recognition model. This method is not optimized for the field of subtitle generation. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies, such as the inability to simultaneously address latency and subtitle stability, the inability to adaptively adjust to the text being processed, and the inconsistency in the logic before and after subtitles. It provides a method for generating subtitles for real-time doctor-patient dialogues, which combines real-time performance with readability and accuracy.

[0008] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows: A method for generating captions for real-time doctor-patient dialogue includes: S1: Obtain the audio and subtitle text generation parameters for the subtitles to be generated; S2: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is divided into windows to obtain an acoustic feature sequence; S3: Input the acoustic feature sequence into the streaming encoder to obtain the acoustic code; S4: Using a dual-branch decoding model, based on the acoustic coding and subtitle text generation parameters, connective temporal classification text and recurrent neural network text are generated respectively; S5: Based on the connection time-series classification text, the recurrent neural network text, and the subtitle text generation parameters, filter and combine the connection time-series classification text and the recurrent neural network text to obtain the generated subtitle text.

[0009] Further, the subtitle text generation parameters include the core text length and the total text length; in step S5, the generated subtitle text includes: S501: Based on the subtitle text generation parameters, the connection time-series classification text is divided into connection time-series classification core text and connection time-series classification tail text; the recurrent neural network text is divided into RNN core text and recurrent neural network tail text. S502: Calculate the similarity between the connection time-series classification core text and the recurrent neural network core text to obtain a first similarity; calculate the connection time-series classification core text confidence based on the connection time-series classification core text; S503: If the first similarity meets the first preset condition and the confidence of the connection time-series classification core text meets the second preset condition, then the connection time-series classification text will be used as the subtitle text; otherwise, the recurrent neural network text will be used as the subtitle text.

[0010] Furthermore, after step S5, the following steps are also included: S6: Generate a stable connection time-series classification text signal based on the connection time-series classification text and the connection time-series classification text from the previous time step; generate a recurrent neural network text confidence score based on the recurrent neural network text; generate a common prefix length based on the connection time-series classification text and the recurrent neural network text. S7: Based on the connection time-series classification text stability signal, recurrent neural network text confidence, and common prefix length, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

[0011] Furthermore, in step S7, the formula for calculating the core text length in the subtitle text generation parameters is as follows:

[0012] This represents the core text length, t represents the audio duration, and max represents the maximum value. Indicates the length of the common prefix. Indicates safety margin, This indicates a stable signal for connection-time classification text. This represents the text confidence level of a recurrent neural network.

[0013] Furthermore, the formula for calculating the stable signal of the connected temporal classification text is as follows:

[0014]

[0015]

[0016] This indicates a stable signal for connection-time classification text. This indicates the change in length at the tail, where t represents the audio timing. Indicates text length. Represents the longest common prefix function. This indicates that the text is linked according to a time sequence classification, and M represents the length of the trailing text. This represents the vocabulary list, where i represents the sequence number of the text at the end, and k represents the vocabulary list sequence number. Represents the average information entropy. This represents the probability that the i-th tail text is predicted to be the k-th word in the vocabulary; The formula for calculating text confidence in a recurrent neural network is as follows:

[0017] This represents the sequence concatenation amount in text from a recurrent neural network. Predicting sequences in text formed by recurrent neural networks The probability is given by L, where L represents the total number of sequences in the recurrent neural network text, and j represents the sequence number in the recurrent neural network text. Representing recurrent neural network text The result of concatenating the sequences.

[0018] Furthermore, after step S7, the following steps are also included: S8: Generate intent input features based on the acoustic coding, the common prefix length, the connection timing classification text stability signal, and the subtitle text generation parameters; S9: Based on the intent input features, use the intent prediction model to generate intent prediction results; S10: Based on the intention prediction result, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

[0019] Furthermore, the dual-branch decoding model includes: a connection-time classification branch and a recurrent neural network branch; wherein the connection-time classification branch outputs connection-time classification text, and the recurrent neural network branch outputs recurrent neural network text; The subtitle text generation parameters also include: the computational budget overhead of the recurrent neural network branch; Step S4 includes: S401: Obtain parameters of the computer system; S402: Allocate computing resources for connecting the temporal classification branch and the recurrent neural network branch based on the computational budget overhead of the recurrent neural network branch and the parameters of the current computer system; S403: Input the acoustic code into the connection time-series classification branch and the recurrent neural network branch respectively to obtain the connection time-series classification text and the recurrent neural network text.

[0020] Furthermore, in step S10, the calculation formula for the computational budget cost of the RNN branch of the subtitle text generation parameters is as follows:

[0021] This represents the computational budget cost of the recurrent neural network branch at audio time t, where t represents time. This represents the set of intents, where k represents the intent number. The budget adjustment factor representing the k-th type of intention. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention; In step S10, the formula for calculating the total text length of the subtitle text generation parameters is as follows:

[0022] Indicates the total text length. This represents the set of intents, where k represents the intent number. The moderating weight represents the k-th type of intent. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention.

[0023] A method for generating subtitles for long audio files, comprising: S01: Obtain the audio to be processed; subtitle text generation parameters; S02: Using subtitle text generation parameters, the audio to be processed is divided into a first audio to be processed and a second audio to be processed; wherein, the second audio to be processed includes at least one audio frame from the first audio to be processed; S03: Using the aforementioned method for generating subtitles for real-time dialogue between doctors and patients, subtitles are generated for the first audio to be processed and the second audio to be processed, respectively, to obtain the first subtitle text and the second subtitle text; S04: Take the text connecting the end of the time-series classification in the first subtitle text as the third text; take the text connecting the core of the time-series classification in the second subtitle text as the fourth text; calculate the similarity between the third text and the fourth text at the corresponding position of the third text to obtain the third similarity; S05: If the third similarity meets the third preset condition, then the connection time sequence classification core text in the first subtitle text is taken as the final text; otherwise, the recurrent neural network core text in the first subtitle text is taken as the final text.

[0024] A real-time doctor-patient dialogue caption generation system includes: Parameter acquisition module: Acquires the audio and subtitle text generation parameters for the subtitles to be generated; Sequence segmentation module: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is segmented into windows to obtain an acoustic feature sequence; Encoding module: Inputs the acoustic feature sequence into the streaming encoder to obtain acoustic coding; Text generation module: Using a dual-branch decoding model, based on the acoustic coding and subtitle text generation parameters, it generates connected temporal classification text and recurrent neural network text, respectively; Text filtering module: Based on the connection time-series classified text, the recurrent neural network text, and the subtitle text generation parameters, the module filters and combines the connection time-series classified text and the recurrent neural network text to obtain the generated subtitle text.

[0025] Compared with the prior art, the beneficial effects of the present invention are: This invention uses the connection-time classification text, the recurrent neural network text, and the subtitle text generation parameters to filter and combine the connection-time classification text and the recurrent neural network text to obtain the generated subtitle text. Because the connection-time classification text is generated quickly but with lower accuracy, while the recurrent neural network text is generated more slowly but with higher accuracy, this invention balances real-time performance, readability, and accuracy. Attached Figure Description

[0026] Figure 1 The flowchart is provided for a method for generating captions for real-time doctor-patient dialogue in Example 1.

[0027] Figure 2 This is a flowchart illustrating a method for generating real-time dialogue subtitles for doctors and patients, as provided in Example 1.

[0028] Figure 3 This is a schematic diagram of window division provided in Example 1.

[0029] Figure 4 This is a schematic diagram of a real-time dialogue caption generation system for doctors and patients provided in Example 1. Detailed Implementation

[0030] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] Example 1 like Figure 1 , Figure 2 As shown, a method for generating real-time dialogue subtitles for doctors and patients includes: S1: Obtain the audio and subtitle text generation parameters for the subtitles to be generated; S2: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is divided into windows to obtain an acoustic feature sequence; S3: Input the acoustic feature sequence into the streaming encoder to obtain the acoustic code; S4: Using a dual-branch decoding model, based on the acoustic coding and subtitle text generation parameters, connective temporal classification (CTC) text and recurrent neural network (RNN) text are generated respectively. S5: Based on the connection time-series classification text, the recurrent neural network text, and the subtitle text generation parameters, filter and combine the connection time-series classification text and the recurrent neural network text to obtain the generated subtitle text.

[0033] Further, the subtitle text generation parameters include the core text length and the total text length; in step S5, the generated subtitle text includes: S501: Based on the subtitle text generation parameters, the connection time-series classification text is divided into connection time-series classification core text and connection time-series classification tail text; the recurrent neural network text is divided into RNN core text and RNN tail text. S502: Calculate the similarity between the connection time-series classification core text and the recurrent neural network core text to obtain a first similarity; calculate the connection time-series classification core text confidence based on the connection time-series classification core text; S503: If the first similarity meets the first preset condition and the confidence of the connection time-series classification core text meets the second preset condition, then the connection time-series classification text will be used as the subtitle text; otherwise, the recurrent neural network text will be used as the subtitle text.

[0034] In one specific embodiment, at time t, the constructed window is input with an acoustic feature sequence. Input a streaming encoder and perform incremental inference to obtain the encoded representation sequence of the current window. The streaming encoder employs a state-preserving neural network architecture, including the Chunk Conformer or its equivalent variant. Its working mechanism involves the encoder processing the current window features... Read and reuse the internal state saved in the previous time step. This allows for the generation of the current representation through incremental computation without repeatedly calculating historical information, and the output of the updated state. For use in the next time step. This process satisfies:

[0035] in, For length is Frame-level acoustic feature sequences, For length is The encoded representation sequence, The encoder cache state includes, but is not limited to, self-attention key-value cache, convolution state, or recursive state, which is used to reuse historical context and reduce redundant computation.

[0036] To accommodate the "stable commit" and "dynamic rollback" mechanisms proposed in this invention, the system performs logical partitioning on the output encoded representation sequence H_t, dividing it into a "stable core segment" and a "reserved variable tail segment". Specifically, the system introduces a tail retention length parameter. ,in and and define boundary indexes Therefore,

[0037] Through this explicit partitioning, the system establishes clear commit boundaries for the subsequent decoding process, only for... Provides high-confidence commit support, while for Maintain open correctability to achieve underlying alignment with the draft / final version layered output protocol.

[0038] Furthermore, to support the "subtitle control intent prediction" in subsequent steps, the system can further stabilize the core segment. Feature extraction is performed to generate compact acoustic embedding vectors. In one implementation, the vector is obtained by performing a time-dimensional pooling operation on the core segment representation:

[0039] in This represents average pooling, weighted pooling, or attention pooling along the time dimension. The output is a fixed-dimensional vector used to represent acoustic cues such as pause patterns, energy fluctuations, and speech rate changes. This allows the system to obtain text-independent acoustic auxiliary features even when text decoding is not yet complete or the text content is ambiguous.

[0040] After the incremental encoding output described above, the system enters a dual-branch decoding process to generate real-time captions for the doctor-patient dialogue. The dual branches include a CTC branch and a CTC sub-branch. The CTC branch can employ an RNN-T decoding structure or its equivalent. The input to the CTC branch includes at least... or equivalent It is used to calculate the frame-level posterior distribution and generate draft caption hypotheses; the input of the RNN branch includes at least Along with historical tag information, which includes the text prefix from the previous moment and the corresponding prediction network state cache, candidate subtitles are continuously expanded during tag synchronization decoding. To illustrate the specific structure of the above data interface in a real-time subtitle scenario, the following description uses two consecutive window updates as an example. During the first update... The encoder input is the current window features. Compared with the initial cache state The encoder output is And calculate the boundary index. Optional output acoustic embedding At this time, the CTC branch receives... Draft subtitles as input and output In implementations employing beam search, the candidate set and its scores can be maintained as initial conditions for the next incremental search. RNN branch reception Initial text history (Empty prefix or start character) and prediction network initial state cache As input, output high-confidence captions It also outputs the updated predicted network cache state. For reuse in the next update. In the second update... The encoder input is the current window features. Compared to the cached state at the previous moment The encoder output is And calculate the boundary index. Optional output acoustic embedding At this time, the CTC branch receives... As input, and optionally with the candidate set and score maintained in the previous time step to complete the incremental update, the draft subtitle is output. And update the candidate set. The RNN branch receives... Previous text prefix (Depend on (or its candidate prefix) and predicted network cache state As input, continue tag synchronization expansion to output high-confidence captions. And update the predicted network cache state. The above method encodes the output. With boundary index In each update, the commit boundaries of the two-branch decoding and subtitle display layers are uniformly constrained, and the tail area retains a space for correction, thereby meeting the combined requirements of low latency and low revision for real-time dialogue subtitle generation between doctors and patients.

[0041] After completing the first After incremental encoding of the second window, the system performs a rolling update operation on the state, updating the internal state output at the current moment. Write to the cache store to overwrite or update the old state. This is the saved... In the When processing the secondary window, it is directly used as the left context. This allows for the retrieval and use of information, thereby creating a link for temporal continuity and context reuse across windows.

[0042] To strictly control the memory usage and computational complexity in real-time subtitle scenarios, the system performs the following checks before saving the state: Perform resource constraint processing. Specifically, the system applies preset historical coverage limit parameters. The cached state is truncated using truncation, compression, or first-in-first-out (FIFO) eviction policies, discarding outdated information that exceeds the effective receptive field. This update logic can be expressed as:

[0043] in, This represents the clipping operation on the state tensor in the time dimension. Through this step, the system ensures that the memory overhead and computation latency of the streaming encoder for single-step inference remain in the constant range (O(1)) when processing audio streams of arbitrary length, and do not increase linearly with the increase of audio duration.

[0044] In step S4, at time... The system utilizes the CTC (Connectionist Temporal Classification) decoding branch to analyze the acoustic features of the input. Frame-synchronized decoding is performed to generate an initial draft text with minimal latency. The CTC decoder first calculates the current acoustic features through a fully connected layer and a Softmax activation function. Belongs to the vocabulary Chinese characters or whitespace symbols (Blank, The posterior probability distribution of ).

[0045] To meet the requirement of real-time captions being displayed instantly, the system employs a greedy search strategy in one implementation. First, it determines the alignment label with the highest posterior probability for the current frame. :

[0046] Subsequently, through the mapping function For the deadline Alignment path sequence Perform normalization. Mapping function. Its function is to remove consecutive repeating characters and all whitespace characters from a sequence. This generates the final draft text sequence. :

[0047] Due to its non-autoregressive independent assumptions, the CTC branch can respond to acoustic changes in parallel and quickly, making it particularly suitable for capturing acoustic boundaries in speech and providing users with low-latency visual feedback (i.e., "fast on-screen").

[0048] The system utilizes the RNN-T (Recurrent Neural Network Transducer) decoding branch in parallel to decode the same acoustic feature. Tag-synchronized decoding is performed to generate a high-confidence final text. Unlike CTC, the RNN-T branch introduces a prediction network and a joint network to explicitly model the "acoustic-text" dependency.

[0049] Specifically, the prediction network is based on the non-empty text history generated in the previous time step. Generate language feature vectors The joint network will combine current acoustic features With language features Perform fusion and calculate the value of the next output label. Joint probability distribution:

[0050] Based on this probability distribution, the system selects the optimal symbol through beam search or greedy search. Update the current text assumption:

[0051] Because the RNN-T decoding branch effectively utilizes historical text context information, the text it generates is significantly superior to the CTC branch in terms of grammatical correctness and semantic coherence. Therefore, it is used as the main source for finalizing the subtitles (final draft) and correcting speech recognition errors in the draft text within the allowable delay range.

[0052] In step S5, the CTC decoding branch and the RNN-T decoding branch work in parallel to jointly generate real-time caption text. At each time step... The CTC decoding branch quickly provides draft subtitles, while the RNN-T decoding branch generates the final text based on deeper contextual information. The combination of these two approaches achieves a balance between low latency and high accuracy through parallel processing.

[0053] During real-time caption generation, the system dynamically selects the timing of text submission based on the outputs of CTC and RNN-T, combined with the current stability and consistency signals. When the draft text and the high-confidence text are highly consistent in the prefix and have high confidence, the system will prioritize submitting the draft text; if there is significant inconsistency at the end or the text is unstable, the system will delay submission and wait for more contextual information. The system controls the output timing according to the following formula:

[0054] in, This indicates that the CTC and RNN-T outputs have the same length in the prefix portion. and It is a threshold of consistency and confidence that determines when to submit the text.

[0055] Furthermore, after step S5, the following steps are also included: S6: Generate a stable connection time-series classification text signal based on the connection time-series classification text and the connection time-series classification text from the previous time step; generate a recurrent neural network text confidence score based on the recurrent neural network text; generate a common prefix length based on the connection time-series classification text and the recurrent neural network text. S7: Based on the connection time-series classification text stability signal, recurrent neural network text confidence, and common prefix length, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

[0056] In steps S6-S7, at time... The system is based on CTC text. With RNN text Perform a two-way comparison to generate a two-branch consistency signal. Based on this, the stable commit prefix length that can be safely solidified at the current moment is calculated. This step essentially constitutes the "arbitration mechanism" of streaming output, which aims to extract a definite consensus portion from uncertain streaming assumptions and strictly separate the "variable tail" from the "immutable history".

[0057] First, the system calculates the "hard consistency" metric for the outputs of the two branches, namely the length of the common prefix. By comparing the two text sequences, the length of the longest subsequence that perfectly matches each other starting from the first character is determined:

[0058] This metric reflects the degree of consensus between the CTC branch and the RNN-T branch on the speech content at the current moment.

[0059] Subsequently, to prevent erroneous submissions due to model overconfidence, the system introduced dynamic safety margins. right Make corrections to obtain the final stable commit prefix length. Unlike a fixed threshold, the safety margin of this invention... It is based on the first stability signal (such as) ) and second stability signal (such as The time-varying function is . Its computational logic satisfies the following constraints:

[0060] Specifically, when the first stability signal indicates severe tail shaking of the CTC (Central Traction Turbine) (Large) or second stability signal indicates low confidence in RNN-T ( When the value is small, the function automatically increases. This allows for a rollback of the commit boundary, reserving a longer buffer to accommodate potential corrections; conversely, when both branches are both consistent and highly confident, Approaching 0 enables aggressive commits.

[0061] Through the above mechanism, the output It not only integrates the semantic consensus of the two models, but also incorporates risk assessment of the current decoding state. This output will directly serve as a key input for subsequent steps, fundamentally defining the cutoff point between the draft and final versions, ensuring that the subtitles displayed to users possess both the low latency of streaming display and the high accuracy and jitter resistance of the final text.

[0062] Furthermore, in step S7, the formula for calculating the core text length in the subtitle text generation parameters is as follows:

[0063] This represents the core text length, t represents the audio duration, and max represents the maximum value. Indicates the length of the common prefix. Indicates safety margin, This indicates a stable CTC text signal. This represents the confidence level of the RNN text.

[0064] Furthermore, the formula for calculating the CTC text-stabilized signal is as follows:

[0065] Calculate the "tail change length". To quantify fluctuations at the text level:

[0066] The larger the value, the more severe the tail reversal.

[0067] Let the probability distribution of CTC output be... Then calculate the average information entropy of the current window tail region. :

[0068] The higher the value, the more ambiguous the model's recognition of the acoustic boundaries or content of the current speech. Even if the text has not changed, the system should still mark it as a potentially unstable state.

[0069] This indicates a stable CTC text signal. This indicates the change in length at the tail, where t represents the audio timing. Indicates text length. Represents the longest common prefix function. This indicates CTC text, where M represents the length of the trailing text. This represents the vocabulary list, where i represents the sequence number of the text at the end, and k represents the vocabulary list sequence number. Represents the average information entropy. This represents the probability that the i-th tail text is predicted to be the k-th word in the vocabulary; Because the RNN-T branch has stronger context modeling capabilities, its stability signal is mainly used to characterize the confidence level at the semantic level and the convergence of the decoding path.

[0070] The formula for calculating the confidence score of RNN text is as follows:

[0071] This represents the sequence concatenation amount in RNN text. Predicting sequences formed in RNN text The probability is given by L, where L represents the total number of sequences in the RNN text, and j represents the sequence number in the RNN text. Representing RNN text The result of concatenating the sequences.

[0072] RNN text confidence can also be combined with the beam search state during the decoding process to generate "candidate path dispersion" as an auxiliary signal. If the probability of the first-ranked hypothetical path in the beam search is much higher than that of the second-ranked path, the current output is determined to be in a highly stable state; conversely, if the probabilities of multiple candidate paths are similar, it indicates that there is semantic ambiguity and the submission priority should be reduced.

[0073] Furthermore, after step S7, the following steps are also included: S8: Generate intent input features based on the acoustic coding, the common prefix length, the connection timing classification text stability signal, and the subtitle text generation parameters; S9: Based on the intent input features, use the intent prediction model to generate intent prediction results; S10: Based on the intention prediction result, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

[0074] It should be noted that the "intention" mentioned in this invention does not refer to the speaker's semantic intention, but specifically refers to the adaptive scheduling decision category made by the streaming system in response to the current decoding state. Formally, let the set of intentions be... Each element This corresponds to a specific parameter adjustment strategy. The intent set Based on the control objectives of real-time captions, they can be divided into at least three function-oriented subsets. The first category is the delay-priority intent (or aggressive submission mode), denoted as... This intent is triggered when the system determines that the current speech clarity is high or the accumulated latency is close to the budget limit, instructing the system to reduce the length of the trailing text. And relax the consistency verification threshold to minimize waiting delay and achieve fast screen display; the second type is the stability-first intent (or conservative and robust mode), denoted as This intent is triggered when the system detects high uncertainty at the end of the text, a noisy acoustic environment, or a risk of rewrite, instructing the system to increase the length of the text at the end. And tighten the submission boundaries The first type sacrifices a small amount of real-time performance to obtain more future evidence in order to suppress visual jitter; the third type is content-sensitive intent (or entity protection mode), denoted as This intent is triggered when the system predicts that the current segment may contain names of people, places, numbers, or specific terms, instructing the system to force entry into a high-confidence verification process and optionally temporarily increasing the computational budget of the decoding branch. To ensure the accuracy of key information. The intent set. It can be dynamically expanded according to actual application scenarios. Its core function is to map complex streaming states into executable discrete control instructions, thereby achieving fine-grained closed-loop control of subtitle display behavior.

[0075] It should be noted that this invention addresses the technical contradiction of existing streaming recognition methods, which are prone to subtitle jitter during low-latency scenarios, while suppressing such jitter introduces additional output lag. This method introduces a decoding intent recognition mechanism based on the online inference framework of streaming framing and incremental coding. This reduces reliance on fixed window lengths, fixed future context values, and static threshold submission rules, enabling dynamic adaptive scheduling of the right context value (tail text length) and submission boundaries (subtitle text, final text). This results in more stable and accurate real-time subtitle text output within a given end-to-end latency budget.

[0076] To improve the controllability of subtitle output, the stabilization strategy of traditional confidence post-processing is abstracted into a closed-loop scheduling problem. Feature signals oriented towards decoding control are constructed and participate in model decision-making to achieve differentiated processing of different speech segments. Highly sensitive segments such as those with rewrite risk, sentence break boundaries, proper names, numbers, and medical terms are explicitly modeled as quantifiable intent signals. These signals, combined with decoding uncertainty and multi-branch hypothesis consistency information, drive the real-time updates of the right context (tail text length) and submission boundaries (subtitle text, final text) of the dynamic window. Draft and final texts adopt a layered output protocol, clearly defining the scope of rewrites and rollback boundaries. This allows stable segments to support rapid submission to control latency, while unstable segments and critical entity segments support delayed submission and introduce more speech evidence to suppress rewrite jitter and error propagation.

[0077] At any moment The system constructs a set of input features to drive the intent prediction model. This feature set aims to comprehensively characterize the current streaming decoding state, and its construction employs a multimodal feature fusion approach. Specifically, the input feature set... At least by acoustic characterization feature vector Decoding state feature vector and context state feature vector Composition. Among them, the acoustic representation feature vector. output acoustic code It is constructed by combining statistical measures of speech activity (such as volume dynamic range, speech rate change rate, and pause duration) to capture acoustic patterns such as hesitation, pauses, or fast speech; and decodes state feature vectors. The intermediate results extracted from steps S6-S7 can be represented as follows: ,in To change the length of the tail section, The length of the common prefix of the two branches. Information entropy of the tail output distribution; context state feature vector This includes the current window state parameters and the current right context value. The system constructs the final input using numerical values ​​and the historical intent prediction from the previous moment. In a preferred embodiment, the system uses a feature interaction network to construct the final input. Using a nonlinear mapping function Projecting the aforementioned heterogeneous features onto a unified semantic space, the fusion process can be described as follows:

[0078] in This indicates a concatenation or weighted summation operation. For the corresponding learnable projection matrix, This is the bias term. Through this multimodal input and fusion formula, the intention prediction model can still make accurate submission scheduling decisions based on acoustic cues (such as long pauses indicating sentence endings) and the uncertainty of the decoding state, even when the text has not yet been generated or has been generated incorrectly, thereby significantly improving the robustness of the system in complex scenarios.

[0079] The constructed multimodal input feature set Input pre-trained intent prediction model The system performs online inference to generate the intent prediction result for the current moment. The intent prediction model can employ a lightweight fully connected network, a recurrent neural network (RNN), or an attention mechanism network; its core function is to map the high-dimensional feature space to a discrete intent category space. In one implementation, the system first calculates the raw logits of the model output and then converts them into an intent probability vector using a Softmax normalization function. This vector represents the confidence distribution of the current decoding state belonging to each control intent category:

[0080] Subsequently, the system extracts the most probable intent label from the probability vector based on the maximum a posteriori probability principle. ,in For the first The probability of predicting class intent. To meet the low latency requirements of real-time captioning, the inference process is constrained by a preset computational budget to ensure that the forward propagation time of the model is controlled within the millisecond range, so as not to block the streaming operation of the main decoding link.

[0081] The system handles each type of intent in the intent set. Use independent sensitivity thresholds Perform gating decision and generate trigger signal components The following indicator function relationship is satisfied:

[0082] in This is an indicator function that outputs 1 (activation) when the condition is met, and 0 (silence) otherwise. This mechanism allows the system to set differentiated trigger thresholds for intents with different risk levels (e.g., setting a lower threshold for "entity protection" intents to improve recall, and a higher threshold for "aggressive submission" intents to ensure accuracy). In another implementation, the system uses a mutual exclusion triggering or priority arbitration mechanism to handle multi-intent conflicts. That is, when the probabilities of multiple intents simultaneously exceed the threshold, the intent with the highest priority is selected as the final trigger signal according to a preset priority lookup table, or a sparse weighted trigger vector is output, thereby achieving precise driving of subsequent control modules.

[0083] Furthermore, the dual-branch decoding model includes: a connection-time classification branch and a recurrent neural network branch; wherein the connection-time classification branch outputs connection-time classification text, and the recurrent neural network branch outputs recurrent neural network text; The subtitle text generation parameters also include: the computational budget overhead of the recurrent neural network branch; Step S4 includes: S401: Obtain parameters of the computer system; S402: Allocate computing resources for connecting the temporal classification branch and the recurrent neural network branch based on the computational budget overhead of the recurrent neural network branch and the parameters of the current computer system; S403: Input the acoustic code into the connection time-series classification branch and the recurrent neural network branch respectively to obtain the connection time-series classification text and the recurrent neural network text.

[0084] It should be noted that CTC decoding typically performs path search on the frame-by-frame probability distribution of the encoder output and obtains text hypotheses through greedy search or beam search. Its implementation is relatively simple and has low output latency. RNN-T decoding jointly models acoustic representations and historical label sequences through prediction networks and joint networks, possessing strong sequence modeling capabilities and output consistency. However, due to the need to maintain and progressively expand the candidate set, its engineering implementation complexity and computational overhead are relatively higher.

[0085] In one specific embodiment, the computational budget overhead of the RNN branch includes, but is not limited to, beam width, upper limit of the number of candidates, or maximum expansion steps.

[0086] Furthermore, in step S10, the calculation formula for the computational budget cost of the RNN branch of the subtitle text generation parameters is as follows:

[0087] This represents the computational budget cost of the RNN branch at audio time t, where t represents time. This represents the set of intents, where k represents the intent number. The budget adjustment factor representing the k-th type of intention. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention; The computational budget overhead is specifically quantified as the beamwidth or maximum number of active paths during the beam search process. The system employs elastic scaling rules in coordination with the right context.

[0088] The innovation of this mechanism lies in its "on-demand allocation" computing power scheduling strategy: when the system triggers an "entity protection" intent (such as detecting a person's name or numerical characteristics), a setting is made. A large positive value instructs the system to instantly increase the search breadth and depth to prevent critical information paths from being pruned early in the decoding process; conversely, when a "delay-first" intent is triggered or excessive device load is detected, the setting is... A negative value indicates that the system actively reduces the search space, concentrating limited computing power on high-probability paths. This intent-driven flexible computing mechanism not only ensures a high recognition rate for critical information but also effectively reduces the system's average power consumption and processing latency in non-critical segments, achieving a dual optimization of recognition accuracy and computational efficiency.

[0089] In step S10, the formula for calculating the total text length of the subtitle text generation parameters is as follows:

[0090] Indicates the total text length. This represents the set of intents, where k represents the intent number. The moderating weight represents the k-th type of intent. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention.

[0091] The update logic at the current moment follows a control law with saturation constraints. (Introduction) The innovation of the gain adjustment term lies in the realization of soft control, that is, the magnitude of parameter adjustment is positively correlated with the model's confidence in the current intention, thereby avoiding drastic parameter oscillations caused by false triggers; For the first The adjustment weight (or directional gain) of the intent type is set to a negative gain for delay-priority intents. To accelerate window contraction and screen loading, a positive gain is set for stability-priority intentions. This is to encourage the window to be expanded to obtain future evidence; The function then imposes physical constraints to ensure Always in the minimum context allowed by the system. With maximum delay budget Between these two points, the system achieves a "breathing window," automatically shrinking to approach the real-time limit when the speech is clear and continuous, and rapidly expanding to ensure recognition accuracy when the meaning is ambiguous or key entities appear.

[0092] A method for generating subtitles for long audio files, comprising: S01: Obtain the audio to be processed; subtitle text generation parameters; S02: Using subtitle text generation parameters, the audio to be processed is divided into a first audio to be processed and a second audio to be processed; wherein, the second audio to be processed includes at least one audio frame from the first audio to be processed; S03: Using any one of the methods for generating subtitles for real-time dialogue between doctors and patients, subtitles are generated for the first audio to be processed and the second audio to be processed, respectively, to obtain the first subtitle text and the second subtitle text; S04: Take the text connecting the end of the time-series classification in the first subtitle text as the third text; take the text connecting the core of the time-series classification in the second subtitle text as the fourth text; calculate the similarity between the third text and the fourth text at the corresponding position of the third text to obtain the third similarity; S05: If the third similarity meets the third preset condition, then the connection time sequence classification core text in the first subtitle text is taken as the final text; otherwise, the RNN core text in the first subtitle text is taken as the final text.

[0093] It should be noted that when the intermediate results generated by streaming decoding are continuously pushed to the subtitle interface, the intermediate assumptions of online recognition are modified as new speech evidence arrives. This can lead to issues such as rollback, replacement, and completion of subtitle content during display, causing subtitle revisions and visual jitter, which in turn affects readability and communication continuity in doctor-patient dialogue scenarios. To suppress these unstable output behaviors, existing technologies typically introduce submission strategies based on time, length, confidence, or endpoint detection (VAD), and statically bind online inference and subtitle display logic using fixed window lengths, fixed future contexts (lookahead), and static submission boundary rules. However, under dynamic conditions such as fluctuating speech rate, background noise, ellipsis, and dense occurrences of medical terminology, static thresholds and rules struggle to simultaneously achieve low latency and high stability, leaving room for further optimization in balancing latency control and stability in subtitle generation. This invention can adjust the core text length in real time based on the context. Furthermore, in step S02, the audio to be processed can be segmented based on the subtitle text generation parameters, thereby dynamically adjusting the pushed text and further balancing low latency and high stability of the subtitles.

[0094] In one specific embodiment, step S2, dividing the audio to be processed using subtitle text generation parameters, includes: performing online framing and windowing processing on the preprocessed input audio stream to divide the continuous speech signal into a short time frame sequence that scrolls over time. The frame length and frame shift for each frame are preset parameters, and the frame shift is configured to be less than the frame length to achieve temporal overlap between adjacent frames. Acoustic feature vectors are calculated based on the signal of each frame. And arrange them into an acoustic feature sequence in chronological order. The acoustic feature vectors include, but are not limited to, filter bank energy features, logarithmic spectrum features, Mel-frequency cepstral coefficient features, or equivalent variants thereof, and may optionally include first-order or second-order difference features. To adapt to streaming computing scenarios, the acoustic features can be normalized using online mean-variance normalization (CMVN) or incremental statistical normalization, i.e., updating the normalization parameters only using current and historical statistics to reduce distribution drift caused by different recording conditions. In one embodiment, the acoustic feature sequence... The output granularity is consistent with the subsequent windowing process, so that... It can be directly used as input to subsequent modules of continuously arriving chunks; in another embodiment, the acoustic feature sequence is first output at the frame level and written to a buffer, and then read from the buffer and assembled into chunks by subsequent steps according to preset addition length and overlap length. In this way, this step provides a standardized input interface for subsequent streaming windowing organization and streaming coding.

[0095] In step S02, at time... Construct a streaming window input that includes historical context and currently added information. The window overlap relationship and the division of newly added segments are as follows: Figure 3 As shown. The system first sets the number of new feature frames corresponding to the current window based on the step size of streaming processing and context requirements. and the number of overlapping feature frames is ,in and These correspond to the update step size and overlap duration in the time dimension, respectively. Next, the effective coverage interval of the current window in the feature sequence is determined based on the temporal progression relationship, with the [missing information] as the [missing information]. The newly added interval start frame index in the next update Based on, extract from to The continuous feature frames are used as the current processing window During this process, if the calculated starting index involves negative sequence subscripts, zero-padding or truncation is used to ensure the index's validity. The current window is thus constructed. Structurally, it consists of a length of Preceding overlapping feature frames and length is The system comprises newly added feature frames, with overlapping regions reusing tail information from the previous window to maintain the continuity of the historical context on the left. This overlapping windowing organization method enables smooth segmentation of continuous speech streams, effectively reducing boundary effects in streaming processing while introducing new information, and providing a stable input state for subsequent incremental coding.

[0096] In one specific embodiment, to reduce the overhead of redundant computations caused by window overlap or cross-window dependencies in streaming processing, the system establishes a left context caching mechanism to reuse historical window information. The left context caching includes at least an encoder-level cache. Optional features include feature-level caching. Among them, feature-level cache This is used to store the features of overlapping frames at the end of the previous window, allowing the next window to directly concatenate them to form a continuous input, avoiding repeated front-end feature extraction and preprocessing on overlapping frames. Encoder-level cache. This is used to store reusable states within the encoder, including but not limited to self-attention key-value caches, convolution states, or recursive states, for use in processing the first... Providing the necessary historical context for each window allows the encoder to obtain a continuous contextual representation without having to repeat forward computation on already processed historical segments.

[0097] In an implementable definition, let the streaming encoder be a function. Its input is the current window features. Compared to the state recorded at the previous moment The output is the encoded result. With the updated storage state ,satisfy

[0098] in, Represents the hidden representation sequence corresponding to the feature frames within the window. State As a carrier of historical context, it enables the encoder to utilize historical information from previous windows when processing the current window, thereby improving semantic continuity across windows and reducing redundant calculations.

[0099] Taking the generation of subtitles for a real-time doctor-patient dialogue as an example, the system extracts acoustic features according to a fixed frame shift and processes them online using overlapping windows. Assuming a window duration of 1.6 seconds and a window step size of 0.8 seconds, adjacent windows have an overlap region of 0.8 seconds. (First window...) The characteristics are Encoder output Afterwards, The cache is for the encoder-level left context. Second window. The input consists of the newly arrived 0.8s segment features and the overlapping segment at the end of the previous window. Without a caching mechanism, the encoder needs to perform a complete encoding and inference of the overlapping region again, resulting in redundant calculations and introducing unnecessary latency. Feature-level caching is employed. At this time, features of overlapping regions can be directly reused; encoder-level caching is employed. During the process, the encoder is processing When a new segment arrives, the cross-window context can be directly obtained using historical states, avoiding the need to re-infer from historical segments. This mechanism can improve the continuity and stability of caption generation in doctor-patient dialogues, reduce the risk of breaks and misidentification caused by window switching, and reduce computational overhead to meet real-time constraints.

[0100] Furthermore, to describe the control relationship of the effective length of the left context, the length of the encoder's visible historical context is defined as follows: This is determined by the cache capacity or policy. When the system needs to limit historical dependencies to control computational and storage overhead, it performs a truncation operation on the cache state.

[0101] in This indicates the pruning or compression of cached state, used to constrain the resource consumption of streaming processing and maintain controllable end-to-end latency.

[0102] To balance low-latency output and tail stability in real-time captioning scenarios, the system introduces a right context mechanism in the windowed processing chain. The right context value is... This means that, in its physical sense, the system's response to the current moment... Before making a final decision on the content, it is necessary to wait for and observe the future audio duration or the future number of chunks. This is achieved by introducing... The system can use future acoustic evidence to verify the recognition results at the current moment, thereby effectively suppressing subtitle rewrites and visual jitter caused by premature submission.

[0103] To ensure the system always meets the end-to-end latency budget The setting of the right context variable must be subject to strict real-time constraints. Define the total system latency. It consists of two parts: one part is the inherent processing delay caused by feature extraction, encoding, and decoding operations, denoted as The other part is the waiting delay introduced to obtain the right context, denoted as... The delay and The magnitudes are positively correlated. The system must satisfy the following delay constraint:

[0104] Based on the above constraints The upper limit of the feasible region is dynamically determined by the remaining time budget. In other words, the maximum allowed waiting time of the system cannot exceed the remaining space after deducting the current computation time from the total budget.

[0105] in The first as defined above A set of window frame indices. Correspondingly, the system can define "committable regions" and "reserved variable tail regions". Taking frame indices as an example, let the time boundary (or frame boundary) for safe commit be... Then the following condition is met:

[0106] in To preserve the tail length, space is reserved for subsequent corrections in the uncertain region of the tail. With the submission boundary At different granularities (frame domain vs. text domain): frame domain boundaries can be mapped to text prefix boundaries, or converted into a submittable text prefix length through alignment information.

[0107] In one implementation, the system dynamically calculates the maximum allowable value at the current moment based on the above inequality. Value. If the current computing load is light or the network condition is good (i.e. (If it's relatively small), the system can automatically increase it. To obtain more future context, thereby improving caption stability; if the system detects that processing latency has increased due to computational fluctuations, it will automatically reduce... Prioritize ensuring real-time character output. Adjustments will be made regardless of the intent-driven strategy employed in subsequent steps. All processes prioritize the aforementioned delay budget constraint to ensure that the end-to-end delay of the final output is always controlled within a certain range. Within the range.

[0108] During real-time caption generation, the system determines whether to submit the final text based on the output consistency signal, confidence signal, and uncertainty signal of the decoding branch. When the stability of the draft text or candidate text is insufficient, the system restores the stable text state of the previous moment through a rollback mechanism to avoid erroneous caption output due to instability at the end or decoding errors.

[0109] As described in step S05, the rollback mechanism determines whether it is triggered based on stability signals and dynamically adjusts the submission boundary. When text is unstable, the system rolls back the draft text to the previous stable point in time until the text reaches sufficient stability before submitting. The system determines whether to roll back using the following control formula:

[0110] in, Indicates the degree of variation at the end of the draft text. It is a stability threshold that determines whether to perform a rollback.

[0111] In one implementation, the submission boundary is... It is determined jointly by the dual-branch consistency signal and the intent triggering signal:

[0112] in, To stabilize the prefix length, The common prefix length is given by this formula. This formula indicates that when the consistency between the two branches of text is high and the uncertainty is low, the commit boundary increases to improve stability and reduce rewrites; conversely, when the uncertainty of the text increases, the commit boundary decreases, delaying commits to reduce errors. That is, in step S05, the final text length can be determined based on... The size is trimmed to a certain extent to improve stability.

[0113] In another implementation, submit boundaries It is also affected by intent triggering signals. If certain high-stability intents are triggered, the commit boundary will be extended forward, committing the text earlier; while if high-uncertainty intents (such as error correction / revert suppression intents) are triggered, the commit boundary will be delayed backward to preserve more tail area.

[0114] The control signal generated above , , The final output used for real-time caption generation. Based on these control signals, the system adjusts the right context variable, calculates the budget, and sets the commit boundary, thereby enabling real-time caption generation to output captions with low latency while ensuring high accuracy and stability of the caption content.

[0115] In one specific embodiment, the initial text is generated by the first decoding branch (CTC) and displayed to the user as a draft text for real-time captions. The final output is used to display stable, final text. Once the text enters the final output layer, it is no longer modified and is ultimately displayed in a defined caption text format. This mechanism effectively ensures the real-time nature and accuracy of the captions. The draft output provides real-time draft captions, while the final output provides confirmed final captions, avoiding erroneous rollbacks and revisions, and ensuring the consistency of caption content throughout the display process.

[0116] During real-time subtitle generation, computational delays, network fluctuations, or decoding errors may cause subtitle content to deviate from expectations. To ensure the stability and accuracy of the final subtitles, this invention introduces a rollback mechanism to correct unstable or unacceptable subtitle content. The rollback mechanism is based on rollback boundaries. Control when to trigger the rollback operation, and determine whether the draft text needs to be rolled back to the stable state of the previous moment based on the decoded stability signal.

[0117] When the length of the draft text generated by the first decoding branch (CTC) changes... The text confidence score of the second decoding branch (RNN-T) exceeds the preset threshold. When the text falls below a set threshold, the system triggers a rollback operation. During rollback, the system restores the text to its previous stable state and updates the commit boundaries. This ensures that unstable text is not submitted as the final output. The mechanism controls the stability of the subtitle output through a rollback operation, preventing erroneous subtitles from being displayed in real time.

[0118] Real-time caption generation is a continuous process. Each time a new audio segment or speech input arrives, the system performs a series of processing operations. First, the system receives new audio data through a streaming audio input interface and performs frame segmentation and feature extraction to generate new acoustic features. These acoustic features are then input into the streaming encoder to generate the coded representation for the current time step. And then perform further decoding processing.

[0119] Next, the system uses a two-branch decoding process, employing the first decoding branch (CTC) and the second decoding branch (RNN-T) to generate draft text and high-confidence text hypotheses. During this process, the system adjusts the right context variable based on the generated intent signal and stability signal. Calculate the budget and submission boundaries This is to ensure the real-time nature and accuracy of the subtitles.

[0120] Finally, the system adjusts the subtitle content in real time based on the updated context information to meet the stability and latency requirements of real-time subtitle generation. Once the stability requirements are met, the draft text is submitted to the final output layer, completing the final subtitle display.

[0121] like Figure 4 As shown, a real-time dialogue caption generation system for doctors and patients includes: Parameter acquisition module: Acquires the audio and subtitle text generation parameters for the subtitles to be generated; Sequence segmentation module: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is segmented into windows to obtain an acoustic feature sequence; Encoding module: Inputs the acoustic feature sequence into the streaming encoder to obtain acoustic coding; Text generation module: Using a dual-branch decoding model, CTC text and RNN text are generated respectively based on the acoustic coding and subtitle text generation parameters; Text filtering module: Based on the CTC text, the RNN text, and the subtitle text generation parameters, the CTC text and the RNN text are filtered and combined to obtain the generated subtitle text.

[0122] This invention addresses the caption generation scenario for real-time doctor-patient dialogues, proposing a streaming recognition method framework with caption output control as its core objective. This framework transforms the uncertainties in the online recognition process from post-process error correction into a process-controllable scheduling problem. Under a predetermined end-to-end delay budget, it uniformly constrains the timing of caption submission, the scope of possible revisions, and the rollback boundary, ensuring that caption output simultaneously meets the requirements of real-time presentation and reading stability. This reduces the frequency of replacement, rollback, and completion caused by end-sensitivity, thereby improving readability and information transmission efficiency in doctor-patient communication scenarios.

[0123] The innovation of this invention lies in constructing a control signal system oriented towards subtitle display decision-making, and thereby achieving segment-level adaptive scheduling. The system explicitly models the risk of rewrites, sentence segmentation boundaries, and highly sensitive content stages such as proper names, numbers, and medical terms as quantifiable subtitle control intent signals. Simultaneously, it introduces decoding uncertainty and hypothesis consistency information as decision-making criteria to drive real-time updates of future context (lookahead) and submission boundaries. For segments with incomplete semantic closure, high ambiguity, or dense key entities, the scheduling strategy increases evidence acquisition and delays submission to reduce error consolidation and the risk of frequent rewrites. For segments with clear pauses, definite semantics, and high consistency, the scheduling strategy advances submission and converges while waiting to control output lag. This mechanism enables the same system to implement differentiated control over different segments, avoiding premature submissions and unnecessary waiting issues caused by static windows and static thresholds in complex dialogue scenarios.

[0124] This invention further proposes a layered output protocol and boundary constraint mechanism for real-time subtitles to ensure controllable display layer behavior and consistency with downstream functions. The system outputs draft and final text in layers. Drafts are used for low-latency refresh and immediate feedback, while final text is used for stable submission and record fixing. The system limits the scope of revisions through submission and rollback boundaries, preventing unbounded modifications to the readable area already displayed, thereby suppressing visual jitter and reducing user cognitive burden. The aforementioned output control and closed-loop scheduling mechanism is compatible with various end-to-end decoding structures. The online decoder can use CTC, RNN-T, or other equivalent implementations to generate draft and final text. The decoding structure itself does not constitute a limitation of this invention. The key to this invention lies in the synergy of control signal modeling, dynamic scheduling strategies, and layered output protocols for medical and patient subtitle generation, significantly improving the reliability of key entity content and overall output consistency.

[0125] By combining an uncertainty-driven lookahead scheduler and a dual-branch decoding structure, high-accuracy real-time caption generation is achieved while maintaining low latency. The CTC decoding branch provides initial draft text, quickly responding to voice input to meet real-time requirements; the RNN-T decoding branch generates high-confidence final text through a deep learning model, ensuring the accuracy and consistency of the caption content. The system dynamically adjusts the right context (lookahead) and computational budget, precisely controlling the balance between latency and accuracy based on uncertainty and consistency signals during the decoding process. When the system detects high consistency and high confidence in the text content, it reduces the right context to accelerate submission; conversely, when it detects increased uncertainty, it delays submission and increases the right context to ensure the stability and accuracy of the final captions.

[0126] Meanwhile, the system further optimizes subtitle generation performance through a dynamic window scheduling mechanism and adaptive computing power. The dynamic window scheduling mechanism adaptively adjusts the right context variable based on the consistency and confidence signals of the CTC and RNN-T outputs, ensuring that unstable regions are adequately handled during subtitle generation. The adaptive computing power function dynamically adjusts computing resources according to device load, reducing computational demands and shortening latency when the load is high, and increasing the right context variable and computational budget when the load is low to improve subtitle accuracy. This innovative mechanism effectively improves the stability, accuracy, and efficiency of the system in real-time subtitle generation.

[0127] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating subtitles for real-time doctor-patient dialogue, characterized in that, include: S1: Obtain the audio and subtitle text generation parameters for the subtitles to be generated; S2: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is divided into windows to obtain an acoustic feature sequence; S3: Input the acoustic feature sequence into the streaming encoder to obtain the acoustic code; S4: Using a dual-branch decoding model, based on the acoustic coding and subtitle text generation parameters, connective temporal classification text and recurrent neural network text are generated respectively; S5: Based on the connection time-series classification text, the recurrent neural network text, and the subtitle text generation parameters, filter and combine the connection time-series classification text and the recurrent neural network text to obtain the generated subtitle text.

2. The method for generating real-time doctor-patient dialogue subtitles according to claim 1, characterized in that, The subtitle text generation parameters include the core text length and the total text length; in step S5, the generated subtitle text includes: S501: Based on the subtitle text generation parameters, the connection time sequence classification text is divided into connection time sequence classification core text and connection time sequence classification tail text; the recurrent neural network text is divided into recurrent neural network core text and recurrent neural network tail text. S502: Calculate the similarity between the connection time-series classification core text and the recurrent neural network core text to obtain a first similarity; calculate the connection time-series classification core text confidence based on the connection time-series classification core text; S503: If the first similarity meets the first preset condition and the confidence of the connection time-series classification core text meets the second preset condition, then the connection time-series classification text is used as the subtitle text; otherwise, the recurrent neural network text is used as the subtitle text.

3. The method for generating real-time doctor-patient dialogue subtitles according to claim 2, characterized in that, After step S5, the following is also included: S6: Generate a stable connection time-series classification text signal based on the connection time-series classification text and the connection time-series classification text from the previous time step; generate a recurrent neural network text confidence score based on the recurrent neural network text; generate a common prefix length based on the connection time-series classification text and the recurrent neural network text. S7: Based on the connection time-series classification text stability signal, recurrent neural network text confidence, and common prefix length, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

4. The method for generating real-time doctor-patient dialogue subtitles according to claim 3, characterized in that, In step S7, the formula for calculating the core text length in the subtitle text generation parameters is as follows: This represents the core text length, t represents the audio duration, and max represents the maximum value. Indicates the length of the common prefix. Indicates safety margin, This indicates a stable signal for connection-time classification text. This represents the text confidence level of a recurrent neural network.

5. A method for generating real-time doctor-patient dialogue subtitles according to claim 3 or 4, characterized in that, The formula for calculating the stable signal of connection-time classification text is as follows: This indicates a stable signal for connection-time classification text. This indicates the change in length at the tail, where t represents the audio timing. Indicates text length. Represents the longest common prefix function. This indicates that the text is linked according to a time sequence classification, and M represents the length of the trailing text. This represents the vocabulary list, where i represents the sequence number of the text at the end, and k represents the vocabulary list sequence number. Represents the average information entropy. This represents the probability that the i-th tail text is predicted to be the k-th word in the vocabulary; The formula for calculating text confidence in a recurrent neural network is as follows: This represents the sequence concatenation amount in text from a recurrent neural network. Predicting sequences in text formed by recurrent neural networks The probability is given by L, where L represents the total number of sequences in the recurrent neural network text, and j represents the sequence number in the recurrent neural network text. Representing recurrent neural network text The result of concatenating the sequences.

6. The method for generating real-time doctor-patient dialogue subtitles according to claim 3, characterized in that, After step S7, the following is also included: S8: Generate intent input features based on the acoustic coding, the common prefix length, the connection timing classification text stability signal, and the subtitle text generation parameters; S9: Based on the intent input features, use the intent prediction model to generate intent prediction results; S10: Based on the intention prediction result, adjust the subtitle text generation parameters to obtain new subtitle text generation parameters.

7. The method for generating real-time doctor-patient dialogue subtitles according to claim 6, characterized in that, The dual-branch decoding model includes: a connection-time classification branch and a recurrent neural network branch; wherein the connection-time classification branch outputs connection-time classification text, and the recurrent neural network branch outputs recurrent neural network text; The subtitle text generation parameters also include: the computational budget overhead of the recurrent neural network branch; Step S4 includes: S401: Obtain parameters of the computer system; S402: Allocate computing resources for connecting the temporal classification branch and the recurrent neural network branch based on the computational budget overhead of the recurrent neural network branch and the parameters of the current computer system; S403: Input the acoustic code into the connection time-series classification branch and the recurrent neural network branch respectively to obtain the connection time-series classification text and the recurrent neural network text.

8. The method for generating real-time dialogue subtitles for doctors and patients according to claim 7, characterized in that, In step S10, the calculation formula for the computational budget overhead of the recurrent neural network branch of the subtitle text generation parameters is as follows: This represents the computational budget cost of the recurrent neural network branch at audio time t, where t represents time. This represents the set of intents, where k represents the intent number. The budget adjustment factor representing the k-th type of intention. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention; In step S10, the formula for calculating the total text length of the subtitle text generation parameters is as follows: Indicates the total text length. This represents the set of intents, where k represents the intent number. The moderating weight represents the k-th type of intent. The binary trigger signal representing the k-th type of intent. This represents the predicted probability of the k-th type of intention.

9. A method for generating subtitles for long audio files, characterized in that, include: S01: Obtain the audio to be processed; Subtitle text generation parameters; S02: Using subtitle text generation parameters, the audio to be processed is divided into a first audio to be processed and a second audio to be processed; wherein, the second audio to be processed includes at least one audio frame from the first audio to be processed; S03: Using the method for generating subtitles for real-time dialogue between doctors and patients as described in any one of claims 2 to 7, subtitles are generated for the first audio to be processed and the second audio to be processed, respectively, to obtain the first subtitle text and the second subtitle text; S04: Take the text connecting the end of the time-series classification in the first subtitle text as the third text; take the text connecting the core of the time-series classification in the second subtitle text as the fourth text; calculate the similarity between the third text and the fourth text at the corresponding position of the third text to obtain the third similarity; S05: If the third similarity meets the third preset condition, then the connection time sequence classification core text in the first subtitle text is taken as the final text; otherwise, the recurrent neural network core text in the first subtitle text is taken as the final text.

10. A real-time doctor-patient dialogue subtitle generation system, applied to the subtitle generation method according to any one of claims 1 to 8, characterized in that, include: Parameter acquisition module: Acquires the audio and subtitle text generation parameters for the subtitles to be generated; Sequence segmentation module: Based on the subtitle text generation parameters, the audio of the subtitle to be generated is segmented into windows to obtain an acoustic feature sequence; Encoding module: Inputs the acoustic feature sequence into the streaming encoder to obtain acoustic coding; Text generation module: Using a dual-branch decoding model, based on the acoustic coding and subtitle text generation parameters, it generates connected temporal classification text and recurrent neural network text, respectively; Text filtering module: Based on the connection time-series classified text, the recurrent neural network text, and the subtitle text generation parameters, the module filters and combines the connection time-series classified text and the recurrent neural network text to obtain the generated subtitle text.

Images