Method and apparatus for interrupt identification and recovery of real-time audio-video interactive session

Abstract

The application discloses a kind of interruption identification and recovery method and device of real-time audio-video interactive session, it is related to real-time audio-video communication processing technical field.Method includes: in response to session interruption event, extract the multi-source state information before and after interruption from multiple service processes to build session recovery snapshot;It is converted into multi-source time sequence state vector, and interruption phase is identified based on vector;According to the identified reading question, answer question or safety recovery type interruption, generate recovery execution plan, and the question semantics, answer question timing, recording fragment time axis and voice processing context are recovered in coordination;After recovery is completed, calculate comprehensive consistency score, when meeting threshold, submit recovery result.The application unifies the coding of multi-source asynchronous state, recovers by hierarchical identification and phase binding, and is submitted with quantitative score gate, significantly improves the accuracy, continuity and system robustness of interruption recovery.

Description

Technical Field

[0001] This invention relates to the field of real-time audio and video communication processing technology, and in particular to a method and apparatus for identifying and resuming interruptions in real-time audio and video interactive sessions. Background Technology

[0002] Existing technologies have the following significant drawbacks when addressing the issue of abnormal interruptions in real-time audio and video interactive sessions such as online interviews and remote assessments: First, the recovery method relies on a single flag, making it difficult to accurately identify the interruption stage. Actual interruptions can occur at multiple stages, such as during question announcement, waiting for answers, answering, post-question buffering, or switching between questions. Network jitter, browser refresh, and other anomalies can lead to the loss of critical status events. Existing technologies rely solely on a single flag, such as "whether answering has started," for recovery decisions. This is highly susceptible to misjudgments due to missing or out-of-order events, such as misjudging an answering interruption as a question reading interruption.

[0003] Second, the restoration of the object is incomplete, leading to a disconnect in the context. Existing solutions mainly focus on restoring the network connection for real-time communication, but fail to simultaneously restore the semantic state of the questions, such as the current question, the round of answering, the answer timing status, the media recording status, and the voice processing context. This directly results in serious problems after restoration, such as repeated question playback, incorrect answer timing, and misalignment of recorded segments, disrupting the continuity and integrity of the conversation.

[0004] Third, there is a lack of effective handling of asynchronous state conflicts. Business states, cache states, and communication states are maintained asynchronously by different services, with varying refresh cycles and arrival orders. When a client reconnects and recovers, the server may still retain the old terminal state; or the question state may have progressed but recording has not been segmented, leading to inconsistencies in states. Existing technologies lack a mechanism to uniformly coordinate multi-source asynchronous states to suppress conflicts and duplicate triggering during the recovery process.

[0005] In summary, existing technologies are insufficient in terms of accuracy, comprehensiveness, and consistency in interruption recovery. There is an urgent need for a technical solution that can accurately identify the interruption phase of real-time audio and video and accordingly coordinate the recovery of multi-dimensional session context. Summary of the Invention

[0006] This invention aims to address the problems of single-source recovery criteria, one-sided recovery objects, and asynchronous state conflicts after real-time audio and video interaction session interruptions. It provides a method and apparatus for real-time audio and video interaction session interruption stage identification and collaborative recovery based on multi-source temporal state vectors. This invention models the interruption recovery problem as a technical closed loop consisting of state encoding, stage identification, path selection, and consistency gating to achieve highly accurate, highly continuous, and highly idempotent collaborative recovery.

[0007] One aspect of the present invention provides a method for identifying and resuming interruptions in real-time audio and video interactive sessions, comprising the following steps: In response to an interruption event in a real-time audio and video interaction session, multi-source state information within a preset time window before and after the interruption is extracted from various service processes required to maintain the session in order to construct a session recovery snapshot. The multi-source state information in the session recovery snapshot is converted into a multi-source temporal state vector, and the interruption stage to which the interruption event belongs is identified based on the multi-source temporal state vector; Based on the identified interruption phase, a recovery execution plan containing at least one recovery control field is generated, and based on the recovery execution plan, the context of multiple dimensions required for session recovery is collaboratively restored; Once the recovery execution plan is completed, the overall consistency score of this recovery operation is calculated, and the recovery result is submitted when the overall consistency score reaches or exceeds a preset threshold.

[0008] Another aspect of the present invention provides a device for identifying and resuming interruptions in real-time audio and video interactive sessions, comprising: The snapshot construction module is used to extract multi-source state information within a preset time window before and after the interruption from various service processes required to maintain the session in response to the interruption event of the real-time audio and video interaction session, so as to construct a session recovery snapshot. The interruption phase identification module is used to convert the multi-source state information in the session recovery snapshot into a multi-source temporal state vector, and identify the interruption phase to which the interruption event belongs based on the multi-source temporal state vector; The collaborative recovery module is used to generate a recovery execution plan containing at least one recovery control field based on the identified interruption stage, and to perform collaborative recovery of the context of multiple dimensions required for session recovery based on the recovery execution plan; The recovery submission module is used to calculate the overall consistency score of this recovery operation after the recovery execution plan is completed, and to submit the recovery result when the overall consistency score reaches or exceeds a preset threshold.

[0009] This invention provides a method and apparatus for interruption identification and recovery of real-time audio and video interactive sessions. By constructing a multi-source temporal state vector, it uniformly encodes asynchronous states scattered across multiple services into a determinable technical expression. Based on a hierarchical identification algorithm, it accurately determines the interruption stage, thereby driving multi-dimensional context-based collaborative recovery for different stages. Finally, a quantitative consistency score determines whether the recovery result has been submitted. Compared to existing technologies, this invention significantly improves the accuracy of interruption recovery, the continuity of session context, and the idempotency and traceability of the recovery process. Attached Figure Description

[0010] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a method for identifying and resuming interruptions in a real-time audio and video interactive session according to an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a real-time audio and video interactive session interruption identification and recovery device provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In the description of the embodiments of the present invention, the key technical terms to be used thereafter are first defined in order to facilitate understanding of the technical concept of the present invention.

[0013] Multi-source state information refers to data reflecting a certain dimension of the state of a session, which is independently generated and maintained by various service processes with different functions, such as business services, caching services, real-time communication services, recording services, automatic speech recognition services, text-to-speech services, and client state services, during the maintenance of a real-time audio and video interaction session.

[0014] Session recovery snapshot: refers to a set of multi-source state information collected by the system after detecting a session interruption event, which is based on the interruption time and involves tracing back and waiting for a certain time window.

[0015] Multi-source temporal state vector: This refers to the transformation of discrete, multi-source state information with varying formats from session recovery snapshots into a unified vectorized representation usable for numerical computation and logical decision-making, according to preset encoding rules and feature engineering methods. Each dimension of this vector corresponds to a key feature related to interruption phase identification, serving as a bridge connecting the original state data with the interruption phase identification algorithm.

[0016] Interruption Phase: This refers to the specific business phase in which a session is interrupted, and it serves as the fundamental basis for subsequent targeted recovery strategies. In this invention, the interruption phase is categorized at least into question-reading interruption, question-answering interruption, and security recovery interruption. Question-reading interruption refers to the interruption that occurs during the voice broadcast of the question; question-answering interruption refers to the interruption that occurs during the user's answering process; and security recovery interruption refers to a protective interruption established for system security in situations where it is difficult to clearly determine whether it falls into the first two categories, or where there is insufficient evidence or conflicting information.

[0017] Collaborative recovery: refers to not only restoring the network connection based on the identified interruption stage, but also synchronously and coordinately restoring the context state of multiple dimensions required for the session, including at least: topic semantic state (such as current topic identifier, topic stage), media link state (such as real-time communication room, publish-subscribe relationship), recording state (such as recording task, segment index) and speech processing state (such as automatic speech recognition context, broadcast offset).

[0018] Comprehensive Consistency Score: This is a comprehensive technical indicator used to quantitatively evaluate the quality of the recovery results after the recovery execution plan is completed. It is not a single judgment of business correctness, but rather a weighted calculation based on multiple technical dimensions such as semantic consistency of the title, consistency of the timeline, media integrity, and idempotency consistency. It serves as the decision-making gating basis for whether to finally submit the recovery results.

[0019] The core method of this invention will be described in detail and comprehensively below with reference to the accompanying drawings. It should be emphasized that the following description of the method principle, technical problems, and technical effects is based on the architecture and mechanism of the technical solution of this invention itself, and does not depend on any specific business application scenario.

[0020] This embodiment uses an online interview or online Q&A scenario as an example to elaborate on a method for identifying and collaboratively recovering interrupted stages in real-time audio and video interaction sessions based on multi-source temporal state vectors. Figure 1 As shown, the method mainly includes the following steps: Step S101: In response to an interruption event of the real-time audio and video interaction session, construct a session recovery snapshot.

[0021] Traditional methods typically acquire the current state instantly upon detecting an interruption. However, in distributed systems, network latency and processing time differences exist in the state callbacks and log recording of services such as business operations, real-time communication, and recording. If a critical state (such as the start of a quiz event) arrives only a few hundred milliseconds after the interruption detection, traditional methods will miss this information, leading to fundamental errors in subsequent stage identification. To address the technical problems of existing technologies, such as a single source of state acquisition and the inability to handle the loss or inconsistency of state information caused by asynchronous callbacks, this invention circumvents these shortcomings by constructing a acquisition window that spans both before and after the interruption.

[0022] This step specifically includes the following sub-steps: Step S1011: Determine the interruption time. When the system detects a preset interruption trigger condition, such as a heartbeat timeout between the client and server, a disconnection in a real-time communication room, or an interruption of the recording service callback, the system first records the exact time T0 when the interruption event occurs as a time reference. This reference is the anchor point for all subsequent time window calculations.

[0023] Step S1012: Extract multi-source status information within a first preset time period. Using the interruption time T0 as a baseline, a fixed first preset time period ΔT1 is traced back to form a historical data acquisition window [T0-ΔT1, T0]. The system retrieves all status information generated within this time window from all relevant service processes, including business services, caching services, real-time communication services, recording services, automatic speech recognition services, and client status services.

[0024] Step S1013: Extract asynchronous callback data arriving within the second preset duration. Using the interrupt time T0 as a reference, open a waiting window forward and continuously wait for a second preset duration ΔT2. During this period [T0, T0+ΔT2], the system continuously listens for and collects all delayed asynchronous callback data. This data represents the state events that occurred before T0 but failed to arrive in time due to network or processing delays.

[0025] Step S1014: Merge to form a session recovery snapshot. The system merges the two parts of multi-source status information collected in steps S1012 and S1013, and sorts them according to timestamps to finally form a structured session recovery snapshot with more complete information. This snapshot preferably includes one or more of the following fields: session identifier, user identifier, current question identifier, question stage, current round, text-to-speech broadcast offset, remaining answer time, most recent automatic speech recognition processing timestamp, automatic speech recognition activity score, real-time communication room identifier, publishing status, subscription status, disconnection reason, recording task identifier, recording segment index, recording segment interval, terminal identifier, client page visibility status, and snapshot generation timestamp.

[0026] Step S101 employs a dual-time-window acquisition strategy that combines backtracking and waiting to minimize the risk of inconsistent arrival times of state events in a distributed system. This provides a more reliable and comprehensive data foundation for subsequent stage identification than traditional single-point sampling, fundamentally improving the accuracy of recovery decisions.

[0027] Step S102: Convert the multi-source state information in the session recovery snapshot into a multi-source temporal state vector.

[0028] To address the technical challenge of heterogeneous and semantically fragmented multi-source state information, which cannot be directly utilized by a unified decision-making algorithm, this step uses pre-defined encoding rules and feature engineering to compress and map the heterogeneous state information into a standardized, multi-dimensional numerical vector, serving as the sole input to the subsequent interruption phase identification algorithm. This is equivalent to constructing a technical fingerprint for complex session states.

[0029] Specifically, the system iterates through all state fields in the session recovery snapshot generated in step S101, quantifies or symbolically encodes each key state, and generates a multi-source temporal state vector X, represented as X = {x1, x2, ..., x...}. n Each element x in this vector i Each of these represents a feature that is strongly correlated with the interruption phase judgment and has a clear physical meaning.

[0030] Preferably, the multi-source temporal state vector includes at least the following feature dimensions: x1: Current question stage code. It is a discrete numerical code for the business stages such as the broadcast stage, the waiting stage, the answering stage, the post-question buffering stage, and the inter-question switching stage.

[0031] x2: The presence of a start-of-question event. It is a Boolean value or a 0 / 1 code indicating whether an explicit start-of-question signal was captured within the time window of the session recovery snapshot.

[0032] x3: Text-to-speech offset percentage. It is a continuous value in the range [0,1], representing the progress of the current question's speech.

[0033] x4: Answer timer running status. It is a Boolean value or a 0 / 1 code, indicating whether the answer timer is in running counting state at the time of the interruption.

[0034] x5: Percentage of remaining time for answering questions. It is a continuous value in the range [0,1], representing the proportion of remaining time for answering questions.

[0035] x6: Automatic speech recognition activity level within the window before interruption. It is a continuous value or score used to measure the strength of valid speech input detected by the automatic speech recognition module in the short period before the interruption.

[0036] x7: Recording segment continuity metric. It is an encoded value used to evaluate whether the recording tasks and segment indices are continuous before and after an interruption, and whether there are gaps.

[0037] x8: Real-time communication disconnection reason code. It is a numerical code for different connection / disconnection reasons such as network timeout, client leaving, server kicking out, and unknown.

[0038] x9: Terminal switching flag. It is a Boolean value or a 0 / 1 code indicating whether the terminal initiating the recovery request is the same device as the active terminal before the interruption.

[0039] x 10 : Client-side page visibility status. It is an encoded value indicating whether the client-side page is visible in the foreground, invisible in the background, etc., when the connection is interrupted.

[0040] x 11 : Recent business event type. It encodes the most recent business event before the interruption, such as the completion of inter-question switching or the end of post-question buffering.

[0041] x 12 : Idempotent token matching status. It is a boolean value or a 0 / 1 code indicating whether the idempotent token carried by the current recovery request matches the one recorded by the server or is a completely new request.

[0042] Step S102 encodes the asynchronous state data, which is scattered across multiple services and has different formats, into a low-dimensional, highly representative, and machine-friendly time-series state vector. This vector shields the heterogeneity of the underlying data, allowing the upper-layer recovery decision logic to operate entirely without regard to the data source, relying solely on this standardized technical fingerprint for calculations and judgments. This greatly simplifies algorithm design and enhances the robustness and scalability of the system.

[0043] Step S103: Based on the multi-source timing state vector, identify the interrupt stage to which the interrupt event belongs.

[0044] This step aims to address the technical problem in existing technologies where recovery relies on a single basis and cannot accurately distinguish between different stages of interruption. It adopts a hierarchical judgment strategy that prioritizes strong deterministic evidence and supplements it with weak evidence fusion, and introduces a differential safety backoff mechanism to handle high uncertainty scenarios.

[0045] This step specifically includes the following sub-steps: Step S1031: Prioritize strong deterministic evidence.

[0046] The purpose of this priority level is to pursue certainty and interpretability in decision-making. After obtaining the multi-source temporal state vector X, the system first checks whether there are strong evidence rules that can definitively determine the outcome. These rules are based on the necessity and causal relationship of business logic. Once satisfied, the result can be directly output with high confidence without complex numerical calculations.

[0047] Specifically, the rules for prioritizing strongly certain evidence include: Rule 1: If there is a start event for answering questions and the answer timer is running, then it is directly determined as an interruption of answering questions (k2).

[0048] This is because the start of the question-answering event is a clear signal of entering the question-answering phase, and the running of the question-answering timer is conclusive evidence that the question-answering phase continues. The combination of these two characteristics constitutes the necessary and sufficient condition for determining an interruption in the question-answering process. Regardless of other characteristics, the fact that the interruption occurred during the question-answering process has been confirmed.

[0049] Rule 2: If there is no answer start event and the text-to-speech (question broadcast) offset ratio x3 is greater than the first threshold, then it is directly determined as a question reading interruption (k1).

[0050] This is because the absence of a "no answer start event" eliminates the possibility of an interruption in the answering process. If x3 is greater than the threshold, it means that the question has been broadcast for some time but the answering process has not yet begun; in this case, the interruption must have occurred during the question reading phase. The threshold is set to eliminate the ambiguous state where x3 is 0 or close to 0 at the very beginning of the question.

[0051] Rule 3: If the most recent business event type x 11 If the switching between questions is completed or the post-question buffering ends, it is directly determined as a safe recovery type interruption (k3).

[0052] This is because such events indicate that the session is in a safe gap where one topic has naturally ended and the next topic has not yet begun. The interruption at this point in time has a relatively clean context, and there is no need to worry about the complex state inside the topic when resuming. Therefore, it is directly classified as the safest and least conflict-prone safe recovery class.

[0053] If any of the above rules are met, the interruption phase identification process ends immediately, and the corresponding interruption phase is output directly.

[0054] Step S1032: When no strong deterministic evidence is found, perform a weak evidence fusion supplementary judgment.

[0055] When there is no clear, strong deterministic evidence in the multi-source time-series state vector X, it indicates that the system is in a gray area. At this time, a second-layer decision mechanism needs to be activated to make flexible inferences by integrating all available information.

[0056] Specifically, the candidate set of interruption stages is defined as K = {k1,k2,k3}, where k1 represents interruption during question reading, k2 represents interruption during question answering, and k3 represents interruption related to safe recovery. For each candidate stage k, its comprehensive score is calculated. The overall score Calculated using a pre-defined weighted summation model: : Represents the probability score of an interruption event belonging to candidate stage k.

[0057] : Represents the i-th eigenvalue in the multi-source temporal state vector X.

[0058] : for characteristics The mapping function transforms the original feature values ​​into numerical values ​​that are positively correlated with the score. For example, for the discrete feature encoding the current question stage, The function can be a lookup table, and the closer it is to the coding stage of the question, the higher the score.

[0059] : For stage k pairs of features The weight coefficients. These weights characterize the features. The importance of stage k is determined. For example, when determining an interruption in answering k2, the automatic speech recognition activity (x6) and the answer timer running status (x4) will have high weights; while when determining an interruption in reading k1, the broadcast offset percentage (x3) will have a higher weight. All weights are pre-set based on expert experience and optimized through experiments.

[0060] : This is the bias term for stage k, used to handle prior probabilities or inherent biases of the model.

[0061] The system calculates separately , , The candidate phase with the highest score is selected as the initial target interruption phase.

[0062] Step S1033: Difference safety rollback mechanism.

[0063] Even after weak evidence fusion, if the highest score is generated, there may still be situations where evidence is insufficient or scores across different categories are close. In such cases, forcibly adopting the highest score could carry high risks. This step aims to address this uncertainty risk.

[0064] Specifically, let the highest stage score be... The second highest score was Calculate the difference between the two, ΔS = ΔS is compared with a preset difference threshold δ.

[0065] If ΔS ≥ δ, it indicates that there is a significant difference between the highest score and the second highest score, and the evidence is relatively clear. Therefore, the candidate stage with the highest score selected in step S1032 is adopted as the final identified target interruption stage.

[0066] If ΔS < δ, it indicates that at least two stage scores are very close, the evidence is conflicting or insufficient, and the system lacks sufficient confidence in assigning any particular stage. In this case, the system does not forcibly determine it as a reading interruption or an answering interruption, but instead forcibly determines the final target interruption stage as a safety recovery interruption (k3).

[0067] The difference threshold δ in step S103 is a key risk threshold that automatically downgrades high uncertainty situations to safe recovery interruptions, effectively avoiding serious errors such as incorrect question continuation, timing errors, and incorrect recording due to stage misjudgment, and greatly improving the robustness and overall security of the system under abnormal edge conditions.

[0068] Step S104: Based on the identified interruption phase, generate a recovery execution plan and execute the phase-bound collaborative recovery.

[0069] This step aims to address the problems in existing technologies where recovery methods are singular, disconnected from control logic, and unable to specifically recover multi-dimensional contexts. By using the interruption stage identified in step S103 as an instruction, a recovery execution plan containing multiple control fields is driven, and corresponding bound recovery operations are performed on multiple dimensions of context, such as questions, timing, recording, and voice processing.

[0070] Specifically, the system first generates a structured recovery execution plan based on the target interruption phase. This plan preferably includes at least the following recovery control fields: target_question_id: Specifies the target question identifier that should be displayed after recovery.

[0071] target_phase: Specifies the phase of the problem that should be in after recovery.

[0072] tts_resume_mode: Specifies the text-to-speech recovery mode, such as replay from the beginning, resume from the safe point, or do nothing.

[0073] answer_timer_mode: Specifies the recovery mode of the answer timer, such as resume and continue timing, pause and wait for it to start, or do not start.

[0074] record_attach_mode: Specifies the attachment mode for the recorded segment, such as continuing the current question's timeline, creating a new broadcast segment range, or suspending it.

[0075] rtc_rejoin_mode: Specifies the rejoin mode for real-time communication, such as rejoining the original room or no operation required.

[0076] asr_resume_mode: Specifies the recovery mode for automatic speech recognition, such as resume and continue input context, reset context, suspend, etc.

[0077] commit_guard_mode: Specifies the mode of commit protection after recovery, such as disabling automatic commits or allowing commits.

[0078] Subsequently, based on the execution plan, the system performs collaborative recovery operations on contexts of different dimensions. The specific path is as follows: (1) If the target interruption stage is the interruption of reading the question. If a replay strategy is not used, the system retains the question identifier (target_question_id field) and restores the text-to-speech playback to a preset safe playback point (e.g., backtracking 2 seconds from the calculated interruption point) or starts directly from the question's starting point, without activating the answer timing. This operation solves the problem of semantic understanding difficulties that may occur when resuming playback from a breakpoint after a question reading interruption, ensuring the complete transmission of question information by backtracking to a certain safe playback point. Simultaneously, the restored recording segment is marked as the playback segment.

[0079] If a replay strategy is adopted, the system will keep the question identifier unchanged, but will generate a new broadcast segment and start the broadcast again.

[0080] (2) If the target interruption stage is the interruption of answering questions. The system needs to keep the question identifier (target_question_id field) and the number of rounds of answering unchanged, which is the basis for maintaining semantic continuity.

[0081] The system restores the remaining time for answering questions. Specifically, the answer timer can be set to the remaining value before the interruption and the timer can be started again. This solves the problem of the answer timer being lost or reset due to the interruption, ensuring the fairness and continuity of the answering time.

[0082] The system restores the automatic speech recognition input context. This ensures that the speech analysis before and after the interruption is coherent, avoiding the splitting of a sentence into two unrelated segments.

[0083] The system will continue to attach newly recorded media clips to the current topic's timeline through a mapping relationship. This mapping relationship is key to solving the problem of recording fragmentation and will be detailed in subsequent step S105.

[0084] Device isolation: The system binds the new terminal that initiates the recovery request as the target terminal, while actively sending instructions to the old terminal to revoke its streaming and recording / writing permissions, and marking any subsequent media packets sent by the old terminal as invalid sources. This technical action effectively solves the problem of contention between new and old terminals and the resulting state pollution during terminal switching scenarios.

[0085] (3) If the target interruption phase is a safe recovery interruption The system employs the most conservative strategy, keeping the current question from progressing, i.e., preventing automatic jumps to the next or previous question.

[0086] The system forcibly disables the automatic submission and automatic scoring functions to prevent erroneous results when the status is unclear.

[0087] The system triggers a question replay or displays a prompt on the front-end interface, awaiting user confirmation before returning control to the user or entering a preset safety routine. This solves the problem of the system unilaterally performing high-risk operations when the user's state is ambiguous.

[0088] Step S104 generates a recovery execution plan containing multi-dimensional control fields and binds different interruption stages with specific recovery action sequences, achieving refined and automated recovery strategies. The system no longer simply reconnects; instead, it acts like a precise navigator, calculating the optimal recovery path based on the stage of the interruption and automatically recovering all critical states such as questions, timing, recording, and voice, ultimately restoring the session to a logically consistent and data-complete continuous state.

[0089] Step S105: In the question interruption recovery, maintain the mapping relationship between the recorded segment and the question timeline.

[0090] This step is one of the core technical details of the question-answering interruption recovery path in step S104. It is specifically designed to solve the technical problem in existing technologies where interruptions cause recorded segments to scatter and fail to align with the question timeline, leading to subsequent analysis failures. This invention maintains a multi-dimensional mapping table, strictly linking each recorded segment belonging to the question-answering stage to a continuous timeline with the question as the dimension. Specifically, the system maintains a data structure to express the mapping relationship between recorded segments and the question timeline. Its key fields include: question identifier, answering round, recorded segment start index, recorded segment end index, and media time base. The question identifier and answering round uniquely locate a specific answering action; the recorded segment start / end index defines the range of continuous recorded segments belonging to that question and that round; the media time base defines a logical starting time point for the answering process of this question, and all timestamps within recorded segments under this question are offset from this base.

[0091] When a question-answering interruption occurs and is subsequently resumed, the system operates as follows: Based on the resumption plan, after confirming an interruption, the system searches for the mapping relationship between the target question and the question round. Newly generated recording segments will use the latest recording segment's end index plus one as their starting index, continuing to use the same media timeline. This seamlessly stitches all media segments before and after the interruption onto the same question timeline. Logically, it appears as if there was never an interruption.

[0092] Step S105 ensures the logical continuity of the recorded files by establishing and maintaining a precise mapping between questions, recorded segments, and the timeline. Regardless of how many times the session is interrupted, all recorded segments from the same question-answering process will be grouped together along the same timeline. This provides a solid data foundation for subsequent applications such as playback, transcription, and automatic scoring, completely resolving issues like playback frame skipping, transcription text misalignment, and analysis failures caused by segment fragmentation.

[0093] Step S106: Idempotency guarantee of the recovery process.

[0094] This step aims to address the problem of repeated execution of the same recovery task due to network retries, multiple interruptions, or system misjudgments, resulting in repeated state writing, resource waste, or state overwriting (i.e., state pollution). This is achieved by introducing a server-side deduplication mechanism based on idempotent tokens and recovery locks. Before the recovery process begins, the system queries a recovery processing record based on a session identifier and a question identifier. This record is preferably a lock structure, containing a recovery identifier, a recovery version number, and a lock expiration time.

[0095] When the system is ready to perform recovery, it first attempts to acquire the recovery lock. It checks if a processing record with a success status and a recovery identifier identical to that carried in the current request exists. If it does, it determines that this is a duplicate request, directly returns the already generated result, and avoids repeating the recovery operation. If it does not exist, it creates a new recovery lock, executes the recovery process, and updates the record status to success upon completion.

[0096] Step S106 fundamentally guarantees the idempotency of the recovery operation. Even if the upper-layer application or client retryes the recovery request multiple times due to network issues, the server will only perform the actual recovery operation once, effectively avoiding data corruption and resource waste caused by repeated execution, and ensuring the stability and eventual consistency of the system state.

[0097] Step S107: After the recovery execution plan is completed, calculate the overall consistency score of this recovery operation and submit the recovery based on the score result.

[0098] This step aims to address the technical problem of existing technologies lacking effective quality control after recovery, leading to erroneous recovery results being directly submitted as successful, thus triggering a series of subsequent error propagations. This is achieved by establishing a multi-dimensional, quantitative recovery quality assessment model, which serves as the sole gating factor for whether a recovery result is ultimately effective. The system does not equate restoring network connectivity with successful recovery; instead, it verifies and scores the recovery result from multiple key technical dimensions.

[0099] Specifically, a comprehensive consistency score C is calculated by weighted summation of sub-scores from the following four dimensions: Among them, the semantic consistency score of the question This is used to quantitatively assess the consistency between the core business state after recovery and the state before the interruption, and is calculated using the following formula: This represents the restored state vector, which may include a combination of question identifier and question stage. This represents the state vector in the snapshot before the interruption; It is a semantic distance function that measures the difference between the two. The range of values ​​for is [0, ... ]; It is the maximum semantic distance, and If restored correctly, and Completely identical, then The score is 1; the more inconsistent the score, the lower the score. This resolves the issue of incorrect questions or stages after restoration.

[0100] Among them, time axis consistency score This is used to quantitatively assess the deviation between the recovered timeline state and the expected theoretical value, and is calculated using the following formula: It refers to the actual recovery timeline position after recovery, such as the reciprocal of the remaining time for answering questions. It is the expected time point that the theory should restore it to. τ is the absolute value of the deviation, and τ is a time constant used to adjust the tolerance for deviation. The smaller the deviation, the closer the score is to 1; the larger the deviation, the faster the score decays. This solves the problem of timing errors after recovery.

[0101] Among them, media integrity score The following formula is used to quantitatively assess whether the recovered recorded media segments are continuous and complete: This indicates the total duration or number of bytes of all media segments that were not marked as invalid or missing, from the logical timeline starting point of the recorded segment to the current point after recovery. This indicates the total duration or number of bytes that should theoretically be recorded within the same time interval. A score closer to 1 indicates more complete media. This resolves the issue of missing or gapped recordings after recovery.

[0102] Among them, idempotency consistency score Used to assess whether this recovery was a successful, non-repeating operation. If the idempotency check in step S106 determines that this operation is a completely new execution and successfully acquires the recovery lock, then... If it is determined to be a blocked duplicate operation, then... The score is 0. This ensures that only the first successful recovery receives a perfect score.

[0103] In the above formula, λ1, λ2, λ3, and λ4 are preset weight coefficients for each dimension, satisfying λ1+λ2+λ3+λ4 = 1. The specific values ​​can be adjusted according to the business requirements for each indicator.

[0104] The system will calculate the overall consistency score Compare with a preset gating threshold θ: When C ≥ θ: This indicates that the recovery operation has met the quality requirements in all key dimensions, and the results are reliable. The system officially submits and persists all recovery results, and releases the recovery lock.

[0105] When C < θ: This indicates that the recovery quality is substandard and there is a significant problem. The system does not submit the recovery result, but instead triggers a preset degradation rollback process, such as retrying the recovery, notifying manual intervention, or marking the session as abnormal, thereby effectively preventing the spread of the error state.

[0106] Step S107 transforms the recovery process from an open-loop control to a closed-loop control. By introducing a quantified, multi-dimensional quality inspection checkpoint, the system gains a self-checking and error-correcting capability. The recovery operation is only considered complete when multiple technical indicators of the recovery result meet the standards. This not only ensures the correctness of the recovery but also greatly improves the robustness and reliability of the entire system, preventing the cascading amplification effect of erroneous recovery results on subsequent processes.

[0107] The following will use a specific application example to explain in detail the execution process and effect of the method of the present invention in a real scenario, so as to more intuitively understand the overall solution of the present invention.

[0108] In online interview scenarios, the system maintains one-on-one remote interview sessions, including business services, real-time communication services, recording services, and automatic speech recognition / text-to-speech services.

[0109] The interviewer is asking questions to the candidate in real time through the system, currently on question 15 (denoted as Q15). The system is broadcasting the question to the candidate via text-to-speech service, with a total broadcast time of 10 seconds. At the 4.5-second mark, network jitter occurs on the candidate's end, causing the real-time communication connection to be interrupted. At this point, the client detects the RTC room disconnection event and triggers the recovery process.

[0110] Step S1: Build a session recovery snapshot The system interruption time T0 is taken as the reference.

[0111] Backtracking Window (ΔT1 = 5 seconds): The system backtracks 5 seconds and retrieves the status from the business service: current question Q15, question_phase = broadcast phase, tts_offset = 4500ms, answer_timer_remaining = 60s, answer_timer_running = false. From the ASR service, asr_activity_score = 0.05 (extremely low). From the recording service, record_job_id = rec-007, record_segment_index = 3; this segment is marked as a broadcast segment.

[0112] Waiting window (ΔT2=1.5 seconds): Within this window, the system captures a delayed asr_activity_update callback, showing that the activity score is still 0.02, confirming that there is no valid voice input.

[0113] The system combines the above information to generate a structured session recovery snapshot.

[0114] Step S2: Construct multi-source temporal state vectors The system converts the session recovery snapshot into a multi-source temporal state vector X, with the following key features: x1 (stage encoding) = broadcast stage, x2 (answer start event) = 0, x3 (broadcast offset percentage) = 4500 / 10000 = 0.45, x4 (timer running) = 0, x6 (ASR activity) = 0.05.

[0115] Step S3: Identify the interruption phase First layer (strong deterministic evidence): The system first checks the rules and finds that there is no answer start event (x2=0) and the text-to-speech broadcast offset ratio x3=0.45 is greater than the first threshold (0.1). This rule is hit, and the system directly outputs the judgment result: the target interruption stage is the question reading interruption (k1). The process jumps to S4.

[0116] Step S4: Cooperative recovery bound to the execution phase The system generates a recovery execution plan, whose key control fields are: target_question_id=Q15, tts_resume_mode=continue playback from the safe broadcast point, answer_timer_mode=do not start, and record_attach_mode=create a new broadcast segment range.

[0117] The system will perform the following operations in collaboration: Problem semantic state restoration: Lock the current problem identifier to Q15.

[0118] TTS broadcast status restored: The system calculates the safe broadcast point as 4500ms-2000ms = 2500ms, and instructs the TTS service to resume broadcasting from the 2.5th second of Q15 to ensure that candidates can hear the questions continuously.

[0119] Answer timer status restoration: Maintain the answer timer at the initial 60 seconds, but do not start it, waiting for the answer start command.

[0120] RTC and Recording Status Recovery: Instruct the client to rejoin the original RTC room and rebuild the publish-subscribe link. Simultaneously, notify the recording service to create a new broadcast segment_4 for Q15 and record tts_resume_mode and a start offset of 2.5 seconds in the segment metadata.

[0121] Step S5: Consistency Scoring and Submission Once all recovery actions are completed, the system enters the consistency scoring phase.

[0122] The question after the interruption was Q15, which is the same as before the interruption, and the score was 1.0.

[0123] The TTS recovery point is 2.5 seconds. From the user's perspective, this is consistent with the context of the 4.5 seconds before the interruption. The timeline deviation is within an acceptable range, and the score calculated by the formula is 0.95.

[0124] The newly created segment_4 and the segment_3 before the interruption are logically connected to the broadcast timeline of Q15. After inspection, there is no missing data and the continuity is good, so the score is 1.0.

[0125] This was the first and only recovery request. The idempotent lock was successfully acquired, and the score is 1.0.

[0126] The weighted composite score C = 0.3 1.0 + 0.2 0.95 + 0.3 1.0 + 0.2 1.0 = 0.99.

[0127] The overall score C=0.99 is greater than the preset threshold θ=0.85, so the system determines that the recovery was successful and submits all status changes. The interviewer sees that the candidate automatically returns to the room after a brief offline period. After recovery, the system resumes the audio playback of the question from the 2.5-second mark of the current question, and the answer timer remains off, allowing the interview process to continue continuously and correctly.

[0128] This embodiment clearly demonstrates that the present invention does not simply involve reconnecting after an interruption. Instead, it achieves this through a closed loop: accurately identifying the interruption phase, generating a targeted plan, collaboratively restoring multi-dimensional states, and quantitatively verifying the quality of recovery. This ensures the professionalism, continuity, and data integrity of real-time audio and video interaction sessions even under complex and abnormal circumstances.

[0129] See Figure 2 Another embodiment of the present invention provides a real-time audio and video interaction session interruption identification and recovery device 200, including: a snapshot construction module 201, an interruption stage identification module 202, a collaborative recovery module 203, and a recovery submission module 204. This real-time audio and video interaction session interruption identification and recovery device 200 is capable of executing the real-time audio and video interaction session interruption identification and recovery method in the method embodiment.

[0130] Specifically, the real-time audio and video interactive session interruption identification and recovery device 200 includes: The snapshot construction module 201 is used to extract multi-source state information within a preset time window before and after the interruption from various service processes required to maintain the session in response to an interruption event of a real-time audio and video interaction session, so as to construct a session recovery snapshot. Interruption phase identification module 202 is used to convert the multi-source state information in the session recovery snapshot into a multi-source temporal state vector, and identify the interruption phase to which the interruption event belongs based on the multi-source temporal state vector; The collaborative recovery module 203 is used to generate a recovery execution plan containing at least one recovery control field based on the identified interruption stage, and to perform collaborative recovery of the context of multiple dimensions required for session recovery based on the recovery execution plan; The recovery submission module 204 is used to calculate the overall consistency score of this recovery operation after the recovery execution plan is completed, and submit the recovery result when the overall consistency score reaches or exceeds a preset threshold.

[0131] It should be noted that the technical solutions corresponding to the real-time audio and video interactive session interruption identification and recovery device 200 provided in this embodiment, which can be used to execute various method embodiments, have similar implementation principles and technical effects to the methods, and will not be repeated here.

[0132] Figure 3 This is a schematic diagram of an electronic device 300 according to another embodiment of the present invention. The electronic device 300 is used to implement the method for interruption identification and recovery of real-time audio and video interactive sessions in the method embodiment. The electronic device 300 in the embodiments of the present invention may include, but is not limited to, smartphones, tablet computers, PCs, laptops, servers, etc. Figure 3 The electronic device 300 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0133] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 305. An input / output (I / O) interface 304 is also connected to the bus 305.

[0134] Typically, the following devices can be connected to I / O interface 304: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0135] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to the specific combination of the above-described technical features, but should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for identifying and resuming interruptions in real-time audio and video interactive sessions, characterized in that, Includes the following steps: In response to an interruption event in a real-time audio and video interaction session, multi-source state information within a preset time window before and after the interruption is extracted from various service processes required to maintain the session in order to construct a session recovery snapshot. The multi-source state information in the session recovery snapshot is converted into a multi-source temporal state vector. Priority judgment is made based on strong deterministic evidence in the multi-source temporal state vector, including: if a question start event exists and the question timer has started, it is determined as a question answering interruption; if no question start event exists and the proportion of currently played questions is greater than a first threshold, it is determined as a question reading interruption; if the most recent business event type indicates that the question switching is complete or the post-question buffer has ended, it is determined as a security recovery interruption; if there is no strong deterministic evidence in the multi-source temporal state vector, the score of each candidate interruption stage is calculated based on the mapping function and stage weight of each feature vector in the multi-source temporal state vector, and the candidate stage with the highest score is selected as the target interruption stage; when the difference between the highest score and the second highest score is less than a preset difference threshold, the target interruption stage is determined as a security recovery interruption. Based on the identified interruption phase, a recovery execution plan containing at least one recovery control field is generated, and based on the recovery execution plan, the context of multiple dimensions required for session recovery is collaboratively restored; Once the recovery execution plan is completed, the overall consistency score of this recovery operation is calculated, and the recovery result is submitted when the overall consistency score reaches or exceeds a preset threshold.

2. The method for interruption identification and recovery of real-time audio and video interactive sessions according to claim 1, characterized in that, According to the recovery execution plan, the steps for collaboratively recovering the context of multiple dimensions required for session recovery include: If the target interruption stage is the interruption of reading the question and no replay strategy is adopted, the question identifier remains unchanged, the question voice broadcast status is restored to the preset safe broadcast point or the question start point, and the answer timer is not started. If the target interruption stage is a reading interruption and a replay strategy is adopted, a new broadcast segment interval is generated while keeping the question identifier unchanged; If the target interruption stage is the interruption of answering questions, then the question identifier and the round of answering remain unchanged, the remaining time for answering questions is restored, the automatic speech recognition input context is restored, the mapping relationship between the recorded segment and the current question timeline is continued, and the new terminal that initiates the restoration request is bound as the target terminal of the current session. An instruction is sent to the old terminal before the interruption to revoke its streaming publishing permission and recording writing permission, and the media packets subsequently sent by the old terminal are marked as invalid sources. If the target interruption is a safety recovery interruption, then the current question will not be progressed, automatic submission and automatic scoring will be disabled, and a replay will be executed or the user will be asked to confirm.

3. The method for interruption identification and recovery of real-time audio and video interactive sessions according to claim 1, characterized in that, The steps for calculating the overall consistency score of this recovery operation include: The semantic consistency score of the question is calculated according to the following formula. : ； in, This represents the recovered state vector; This represents the state vector in the snapshot before the interruption; This represents a semantic distance function that measures the difference between two things. The range of values ​​for is [0, ... ]; This represents the maximum semantic distance, and ; The timeline consistency score is calculated using the following formula. : in, It is the actual timeline position after recovery; It is the expected point in time that the theory should return to; τ is the absolute value of the deviation; τ is the time constant used to adjust the tolerance for deviation. The media integrity score is calculated using the following formula. : ； in, This indicates the total duration or number of bytes of all media that were not marked as invalid or missing, from the logical timeline starting point of the recorded segment to the current point after recovery. This indicates the total duration or number of bytes that should theoretically be recorded within the same time interval; If the current recovery operation is not repeated, the idempotent consistency score will be... The value is 1; if the current recovery operation is repeated, the idempotent consistency score is... The value is 0; The semantic consistency score, temporal consistency score, media integrity score, and idempotency consistency score of the given topic are weighted and summed to obtain the overall consistency score: ； Where λ1, λ2, λ3, λ4 are weighting coefficients, which satisfy λ1+λ2+λ3+λ4 = 1.

4. The method for interruption identification and recovery of real-time audio and video interactive sessions according to claim 1, characterized in that, The step of extracting multi-source state information within a preset time window before and after the interruption from various service processes required to maintain the session, in order to construct a session recovery snapshot, includes: When an interrupt event is detected, the interrupt time is determined; Based on the interruption time, a first preset duration is traced back as the first time window, and multi-source status information generated within the first time window is obtained from each service process; Based on the interruption time, wait for a second preset time as a second time window, and obtain the asynchronous callback data that arrives late within the second time window; The multi-source state information obtained in the first time window and the asynchronous callback data obtained in the second time window are merged to form the session recovery snapshot.

5. The method for interruption identification and recovery of real-time audio and video interactive sessions according to claim 1, characterized in that, Also includes: Before the recovery process begins, query the recovery processing records based on the session identifier and the question identifier. If a successful processing record with the same recovery identifier exists, return the generated result directly to avoid repeating the recovery operation.

6. A device for identifying and resuming interruptions in real-time audio and video interactive sessions, characterized in that, include: The snapshot construction module is used to extract multi-source state information within a preset time window before and after the interruption from various service processes required to maintain the session in response to the interruption event of the real-time audio and video interaction session, so as to construct a session recovery snapshot. The interruption phase identification module is used to convert the multi-source state information in the session recovery snapshot into a multi-source temporal state vector, and to make priority judgments based on strong deterministic evidence in the multi-source temporal state vector, including: if there is a question start event and the question timer has been started, it is determined to be a question answering interruption; if there is no question start event and the proportion of the current question that has been broadcast is greater than a first threshold, it is determined to be a question reading interruption; if the most recent business event type indicates that the question switching is completed or the question buffering ends, it is determined to be a security recovery type interruption; if there is no strong deterministic evidence in the multi-source temporal state vector, the score of each candidate interruption phase is calculated based on the mapping function and phase weight of each feature vector in the multi-source temporal state vector, and the candidate phase with the highest score is selected as the target interruption phase; when the difference between the highest score and the second highest score is less than a preset difference threshold, the target interruption phase is determined to be a security recovery type interruption; The collaborative recovery module is used to generate a recovery execution plan containing at least one recovery control field based on the identified interruption stage, and to perform collaborative recovery of the context of multiple dimensions required for session recovery based on the recovery execution plan; The recovery submission module is used to calculate the overall consistency score of this recovery operation after the recovery execution plan is completed, and to submit the recovery result when the overall consistency score reaches or exceeds a preset threshold.

7. The device for interruption identification and recovery of real-time audio and video interactive sessions according to claim 6, characterized in that, The collaborative recovery module is further used for: If the target interruption stage is the interruption of reading the question and no replay strategy is adopted, the question identifier remains unchanged, the question voice broadcast status is restored to the preset safe broadcast point or the question start point, and the answer timer is not started. If the target interruption stage is a reading interruption and a replay strategy is adopted, a new broadcast segment interval is generated while keeping the question identifier unchanged; If the target interruption stage is the interruption of answering questions, then the question identifier and the round of answering remain unchanged, the remaining time for answering questions is restored, the automatic speech recognition input context is restored, the mapping relationship between the recorded segment and the current question timeline is continued, and the new terminal that initiates the restoration request is bound as the target terminal of the current session. An instruction is sent to the old terminal before the interruption to revoke its streaming publishing permission and recording writing permission, and the media packets subsequently sent by the old terminal are marked as invalid sources. If the target interruption is a safety recovery interruption, then the current question will not be progressed, automatic submission and automatic scoring will be disabled, and a replay will be executed or the user will be asked to confirm.

8. The device for interruption identification and recovery of real-time audio and video interactive sessions according to claim 6, characterized in that: The recovery submission module is further used for: The semantic consistency score of the question is calculated according to the following formula. : ； in, This represents the recovered state vector; This represents the state vector in the snapshot before the interruption; This represents a semantic distance function that measures the difference between two things. The range of values ​​for is [0, ... ]; This represents the maximum semantic distance, and ; The timeline consistency score is calculated using the following formula. : in, It is the actual timeline position after recovery; It is the expected point in time that the theory should return to; τ is the absolute value of the deviation; τ is the time constant used to adjust the tolerance for deviation. The media integrity score is calculated using the following formula. : ； in, This indicates the total duration or number of bytes of all media that were not marked as invalid or missing, from the logical timeline starting point of the recorded segment to the current point after recovery. This indicates the total duration or number of bytes that should theoretically be recorded within the same time interval; If the current recovery operation is not repeated, the idempotent consistency score will be... The value is 1; if the current recovery operation is repeated, the idempotent consistency score is... The value is 0; The semantic consistency score, temporal consistency score, media integrity score, and idempotency consistency score of the given topic are weighted and summed to obtain the overall consistency score: ； Where λ1, λ2, λ3, λ4 are weighting coefficients, which satisfy λ1+λ2+λ3+λ4 = 1; The snapshot construction module is further configured to: determine the interruption time when an interruption event is detected; based on the interruption time, backtrack for a first preset duration as a first time window, and obtain multi-source status information generated within the first time window from each service process; based on the interruption time, wait for a second preset duration as a second time window, and obtain asynchronous callback data that arrives late within the second time window; merge the multi-source status information obtained within the first time window and the asynchronous callback data obtained within the second time window to form the session recovery snapshot; It also includes a deduplication module, which queries the recovery processing records based on the session identifier and the question identifier before the recovery process begins. If a successful processing record with the same recovery identifier exists, the generated result is returned directly to avoid repeating the recovery operation.

Images