A multi-modal time-series flow active control method and device based on semantic anchor points

Abstract

The application discloses a kind of based on semantic anchor point's multimodal time sequence flow active control method and device, receive at least two modal data including video, audio and text time sequence flow, each modal data is aligned to uniform time axis and extracts modal variation feature, context continuity feature and historical interaction correlation feature;According to the feature detection candidate event boundary, determine semantic logic turning point and generate the semantic anchor point containing timestamp, context abstract, associated relationship and control hook, and then construct anchor point relationship diagram;In the process of time sequence flow play, according to user state, the importance of current semantic anchor point and the penalty of breaking determines whether to initiate active voice interrogation, and according to the semantics and confidence of user voice response, from the continuation, backtrack to target semantic anchor point and jump to branch semantic anchor point to determine control action;User state and anchor point relationship diagram are updated after the control action is executed.

Description

Technical Field

[0001] This invention relates to the field of multimodal time-series data processing and intelligent control technology for streaming media, specifically to a method and apparatus for active control of multimodal time-series streams based on semantic anchors. Background Technology

[0002] With the development of applications such as online video, live audio streaming, remote collaboration, intelligent conferencing, industrial monitoring, content review, and interactive content distribution, continuous time-series streams have become an important carrier for information display, human-computer interaction, and process control. Compared with static data, time-series streams have characteristics such as continuity, strong time dependence, multimodal concurrency, and dynamic changes in context. Their control process involves not only playback progress but also semantic changes, user feedback, equipment status, and external control constraints.

[0003] Existing time-series streaming systems typically handle acquisition, encoding, transmission, caching, playback, transcoding, and retrieval, and can locate segments using fixed timestamps, manual chapter breaks, subtitle segmentation, or keyword indexing. However, most of these methods still treat the time-series stream as a unidirectional output data sequence, requiring users to control it through pause, drag, fast forward, rewind, or input commands. The system struggles to proactively identify semantic and logical turning points during playback, such as theme changes, causal relationship shifts, changes in operational states, or the occurrence of key events. It also struggles to transform these turning points into control nodes that can be used for rewinding, jumping, and proactive interaction.

[0004] While existing multimodal processing technologies can perform visual recognition, speech recognition, subtitle parsing, and semantic analysis separately, the results of each modality are usually independent, lacking fusion state modeling under a unified timeline. Meanwhile, some systems can record user pauses, replays, and historical operations, but fail to uniformly apply response latency, state uncertainty, anchor point importance, and interruption costs to proactive interaction triggering, easily leading to frequent interruptions or failure to intervene in a timely manner. Furthermore, traditional backtracking and jumps are usually executed based on absolute time positions, making it difficult to guarantee that the target position corresponds to a complete semantic starting point. Therefore, there is an urgent need for a time-series proactive control scheme that can integrate multimodal semantic changes, automatically construct semantic anchor points, proactively initiate interactions based on user state, and execute closed-loop control based on response results. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal time-series stream active control method and apparatus based on semantic anchors, which solves the problems of existing time-series stream control relying on manual operation, difficulty in identifying semantic logic turning points and actively executing semantic continuation, backtracking or branch jump in combination with user state.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a multimodal temporal stream active control method based on semantic anchors, comprising the following steps:

[0008] Receive time-series streaming media including at least two modalities of data, including video, audio and text, align each modal data to a unified timeline according to the media timestamp, and extract modal change features, contextual continuity features and historical interaction correlation features between adjacent time windows;

[0009] Based on the features detected candidate event boundaries, anchor point scores for the candidate event boundaries are calculated, and semantic logic inflection points are determined when the anchor point scores satisfy the trigger threshold, minimum time interval, and smoothness stability conditions.

[0010] Generate semantic anchors containing timestamps, context summaries, associations, and control hooks for the semantic logical turning points, and construct an anchor relationship graph based on the sequential relationship, backtracking relationship, branching relationship, or semantic association relationship between the semantic anchors;

[0011] During the playback of time-series streaming media, the user status is updated based on user interaction feedback, and the user status, the importance of the current semantic anchor point, and the interruption penalty are combined to determine whether to trigger an active voice questioning.

[0012] When an active voice challenge is triggered, the challenge content is generated based on the current semantic anchor point, the user's voice response is identified, and the response semantics and response confidence are determined. Based on the response semantics, response confidence, anchor point relationship graph, and user status, the target control action is determined from resume playback, backtracking to the target semantic anchor point, and jumping to the branch semantic anchor point. The target control action is executed, and the user status and anchor point relationship graph are updated based on the execution result.

[0013] As a preferred embodiment of the first aspect of the present invention, aligning the modal data to a unified timeline according to media timestamps includes:

[0014] Media segment objects are generated according to a fixed time window or an adaptive time window adjusted according to the degree of change in the time-series streaming media; video data is sampled, scaled, denoised and standardized; audio data is framed, denoised and endpoint detected; text data is segmented, cleaned, marked and timestamped; and the processed modal data are bound to the same time window.

[0015] As a preferred technical solution of the first aspect of the present invention, the anchor point score is determined based on the intensity of visual change, the intensity of audio change, the intensity of text semantic change, the context coherence correction term, and the historical interaction relevance correction term; the candidate event boundary is determined as a semantic logic turning point only when the anchor point score reaches the anchor point trigger threshold, the time interval between the candidate event boundary and the previous effective semantic anchor point reaches the minimum time interval, and the anchor point score after smoothing reaches the stability threshold.

[0016] As a preferred embodiment of the first aspect of the present invention, the semantic anchor further includes an anchor number, an anchor score, an anchor category, a feature summary, adjacent anchors, and a jump relationship; the control hook is associated with a player control interface, a streaming media server control interface, or an edge control interface so that the backtracking action or branch jump action is executed according to the media position corresponding to the target semantic anchor.

[0017] As a preferred technical solution of the first aspect of the present invention, the user interaction feedback includes at least one of the following: pause operation, replay operation, fast forward operation, rewind operation, voice response delay, low confidence feedback, and historical control results; cognitive load, state uncertainty, and interaction intent intensity are determined based on the user interaction feedback, and an active questioning trigger score is calculated by combining the importance of the current semantic anchor point and the interruption penalty; active voice questioning is triggered when the active questioning trigger score reaches a preset trigger threshold.

[0018] As a preferred technical solution of the first aspect of the present invention, the response confidence is determined based on the automatic speech recognition confidence, natural language understanding confidence, the consistency between the user's speech response and the current semantic anchor context, and noise penalty;

[0019] When the confidence level of the response is lower than the preset response threshold, the branch jump action or other high-risk control action is not executed directly. Instead, at least one of the following is executed: re-questioning, clarification confirmation, narrowing the range of candidate control actions, maintaining continuous broadcast, short pause, returning to the previous stable semantic anchor point, or recording the pending confirmation state.

[0020] As a preferred embodiment of the first aspect of the present invention, the determination of the target control action includes:

[0021] For each candidate control action, the matching degree between the user's response intent and the candidate control action, the relevance between the target semantic anchor point pointed to by the candidate control action and the current context, the consistency between the candidate control action and the historical control trajectory, and the control risk of the candidate control action are determined. Based on the matching degree, relevance, consistency and control risk, the comprehensive evaluation value of each candidate control action is calculated, and the candidate control action with the largest comprehensive evaluation value is determined as the target control action.

[0022] In a second aspect, the present invention provides a multimodal temporal stream active control device based on semantic anchors, for implementing the first aspect, comprising:

[0023] A streaming media access unit is used to receive time-series streaming media that includes at least two modalities of data;

[0024] The multimodal processing unit is used to align the data of each modality to a unified time axis and extract modality change features, contextual continuity features, and historical interaction correlation features.

[0025] The semantic anchor detection unit is used to detect candidate event boundaries and determine semantic logic inflection points when the anchor scores of candidate event boundaries meet the trigger threshold, minimum time interval and smoothness stability conditions.

[0026] Anchor instrumentation and indexing units are used to generate semantic anchors containing timestamps, context summaries, associations, and control hooks, and to construct an anchor relationship graph;

[0027] The user status assessment unit is used to update the user status based on user interaction feedback, and to determine whether to trigger an active voice questioning by combining the user status, the importance of the current semantic anchor point, and the interruption penalty.

[0028] An active voice interaction unit is used to generate question content, collect user voice responses, and determine response semantics and response confidence.

[0029] The control decision and execution unit is used to determine and execute resume, backtrack, or branch jump actions based on response semantics, response confidence, anchor point relationship diagram, and user status.

[0030] The log and status update unit is used to update the user status and anchor point relationship graph based on the execution results.

[0031] As a preferred technical solution of the second aspect of the present invention, the anchor point instrumentation and indexing unit is configured to: in real-time monitoring mode, move the time window according to the playback progress of the time-series streaming media and continuously generate semantic anchor points; or, in offline pre-scanning mode, traverse the complete time-series streaming media before playback and pre-establish an anchor point library and an anchor point relationship graph; the anchor point library is indexed according to at least one of media source, timestamp, anchor point category, anchor point score, context summary and jump relationship.

[0032] As a preferred technical solution of the second aspect of the present invention, the log and status update unit is used to record the source semantic anchor, the target semantic anchor, the reason for the active challenge, the response confidence, the control action, the execution result and the abnormal event, and feeds back the execution result to the user status evaluation unit and the anchor instrumentation and indexing unit to update the backtracking relationship, branching relationship or semantic association relationship between the user status and the semantic anchor.

[0033] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0034] This invention maps multimodal data, including video, audio, and text, onto a single timeline and identifies semantic logical turning points by combining modality change features, contextual continuity, and historical interaction relevance. This generates semantic anchors with timestamps, contextual summaries, associations, and control hooks, transforming media backtracking and navigation from relying on fixed time positions to execution based on semantic nodes. Furthermore, by combining user state, anchor importance, and interruption penalties to determine the timing of proactive voice challenges, and selecting continuation, backtracking, or branching actions based on user response semantics, response confidence, and anchor relationship graphs, this invention reduces invalid interruptions while improving the matching degree between control actions and the current semantic context. Simultaneously, by updating user state and anchor relationship graphs after control actions are executed, subsequent challenge triggering and control decisions can be continuously adjusted using historical feedback, thereby improving the real-time performance, accuracy, stability, and interpretability of proactive control of multimodal time-series streams. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0036] Figure 1 This is a flowchart illustrating the overall workflow of the multimodal temporal stream semantic anchor active control method of the present invention.

[0037] Figure 2 This is a flowchart of the semantic anchor point detection and instrumentation process of this invention;

[0038] Figure 3 This is a flowchart of the active voice questioning and flow control decision-making process of the present invention. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure may be practiced with one or more specific details omitted, or methods, components, steps, etc. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0041] This invention preprocesses, aligns, and extracts features from real-time or offline multimodal time-series streams to identify semantic logic turning points and generate semantic anchors. During playback, it determines whether to initiate an active voice challenge based on user status, anchor importance, and interruption penalty. Then, it executes playback continuation, backtracking, or branch jump based on user voice response, anchor relationship graph, and control constraints, thereby forming a closed-loop active control process.

[0042] Example 1

[0043] like Figure 1-3 As shown, this invention provides a multimodal temporal stream active control method based on semantic anchors, comprising the following steps:

[0044] S1, receive time-series streaming media input, receive real-time or offline input time-series streaming media, read video frames, audio segments, subtitles or text metadata, media timestamps, playback status and control context in the time-series streaming media, and encapsulate the read data into a unified media segment object according to the time window.

[0045] The media segment object includes a segment identifier, a source identifier, a start timestamp, an end timestamp, a video reference, an audio reference, and a text reference.

[0046] The time window can be a fixed-length time window or an adaptive time window that is dynamically adjusted according to the changes in the time-series streaming media. For real-time input time-series streaming media, the time window moves with the playback progress; for offline input time-series streaming media, each media segment can be read in chronological order before the actual playback.

[0047] As one possible implementation, the length of a fixed time window can be set according to the content density and real-time requirements of the time-series streaming media. When using an adaptive time window, the time window can be shortened when the modal change of adjacent time windows increases and extended when the modal change decreases, so as to balance the accuracy of inflection point detection and processing efficiency.

[0048] S2 performs multimodal data acquisition and preprocessing, preprocessing video frames, audio segments, and subtitles or text metadata in the media segment object respectively.

[0049] The process includes sampling, scaling, denoising, and standardizing video frames; framing, denoising, endpoint detection, and acoustic feature preparation for audio segments; and segmenting, cleaning, tagging, and timestamp correction for subtitles or text metadata.

[0050] Preprocessed visual, audio, and text data are bound to a unified time window, enabling data from different modalities to establish a correspondence based on their respective media timestamps, thereby reducing time misalignment between different modalities.

[0051] When there is a discrepancy between the original timestamp of the subtitle or text metadata and the timestamp of the audio segment, the text timestamp can be corrected based on the speech transcription result, the start and end times of the subtitle, or the continuity relationship of adjacent text segments. When one modality data is missing in a certain time window, the remaining modality data can be retained and the modality missing identifier can be recorded for subsequent feature fusion weight adjustment.

[0052] S3 performs temporal alignment and feature extraction, using a fixed time window or an adaptive time window as the processing unit, mapping visual data, audio data, and text data to the same time axis, and extracting visual change features, audio change features, and text semantic change features respectively.

[0053] The visual change features include at least one of shot change, image embedding difference, key target change, and layout change; the audio change features include at least one of energy change, speech rate change, pause change, timbre change, and speaker change; the text semantic change features include at least one of subtitle change, transcribed text change, keyword change, topic vector change, and contextual summary change.

[0054] Further extract contextual continuity and historical interaction relevance, and combine visual change features, audio change features, text semantic change features, contextual continuity and historical interaction relevance into a feature package object.

[0055] The feature package object is used to carry visual change vectors, acoustic change vectors, textual semantic vectors, and fused feature vectors. The preprocessing, temporal alignment, and feature extraction of multimodal data provide a unified data foundation for subsequent identification of semantic logical turning points.

[0056] Among them, contextual continuity is used to characterize the degree of continuity between the current time window and the previous time window in terms of topic, event, object, or operation status; historical interaction relevance is used to characterize the degree of association between the current media content and the user's previous pauses, playback, fast forward, rewind, voice response, or control results.

[0057] S4, identify semantic logical turning points, generate candidate event boundaries based on visual changes, audio changes, and text semantic changes between adjacent time windows, and for the candidate event boundaries at time t, calculate the anchor point score according to formula (1) by comprehensively considering the intensity of visual changes, the intensity of audio changes, the intensity of text or semantic modal changes, the contextual coherence correction term, and the historical interaction relevance correction term:

[0058] (1)

[0059] In the formula, The anchor point score represents the boundary of the candidate event at time ttt. , , These represent the intensity of change in visual, audio, and text / semantic modalities, respectively. Indicates a context coherence correction item. This indicates a historical interaction correlation correction term. , , , and For configurable weights, For bias terms, This is the normalized mapping function.

[0060] To avoid misjudgment of semantic anchors due to short-term noise, the anchor score is smoothed, and minimum interval and stability criteria are introduced. The anchor indicator value is determined according to the formula:

[0061] (2)

[0062] In the formula, Indicates time Anchor point indication value at; This indicates the threshold for triggering anchor point scoring; Indicates the timestamp corresponding to the previous valid semantic anchor; Indicates the minimum time interval between adjacent valid semantic anchors; This represents the anchor point score after smoothing. Indicates the stability threshold. This indicates that all the judgment conditions are met simultaneously.

[0063] When the anchor point indicator value is 1, the corresponding candidate event boundary is determined as a valid semantic logic turning point; when the anchor point indicator value is 0, no semantic anchor point is generated, and monitoring of subsequent time windows continues.

[0064] The above anchor point scoring trigger threshold, minimum time interval and stability threshold can be pre-configured according to the time-series streaming media type and semantic segment density, or can be adjusted according to historical false detection results, but do not change the evaluation relationship defined by formula (1) and formula (2).

[0065] S5. Generate and insert semantic anchors. Generate semantic anchor records for valid semantic logic turning points and write the semantic anchor records into the anchor library or control table.

[0066] The semantic anchor record includes anchor number, timestamp, anchor score, context summary, feature summary, anchor category, adjacent anchors, associated anchors, backtracking relationship, branching relationship, and control hook.

[0067] Among them, the context summary is used to represent the media content within a preset time range before and after the corresponding semantic anchor; the feature summary is used to represent the visual features, audio features, and text semantic features on which the semantic anchor is generated; and the control hook is used to associate the semantic anchor with the control interface of the player, streaming media server, or edge control component.

[0068] An anchor point relationship graph is constructed based on the temporal order, semantic association, and media branching relationships among multiple semantic anchor points. Nodes in the anchor point relationship graph correspond to semantic anchor points, and edges between nodes are used to represent sequential relationships, backtracking relationships, branching relationships, or semantic association relationships.

[0069] By inserting anchor points at semantic logic inflection points, subsequent backtracking or jump control no longer depends solely on absolute time position, but can be executed based on semantic control nodes with contextual meaning and relationships.

[0070] When generating context summaries, subtitles, transcribed text, visual targets, and audio events within a preset time range before and after the semantic anchor can be selected for summarization. When generating control hooks, anchor timestamps, media source identifiers, and target player interface parameters can be bound together, enabling the target semantic anchor to be directly converted into executable positioning instructions.

[0071] S6 monitors user status and interaction feedback. During the playback of time-series streaming media, it continuously reads the current playback progress, current anchor point status, user interaction logs, and optional status signals, and updates the user status model.

[0072] The user interaction log includes at least one of the following: user pause operation, replay operation, fast forward operation, rewind operation, voice response delay, low confidence feedback, and historical control results.

[0073] Based on the user interaction logs, response latency, historical control results, and optional state signals, the user's cognitive load, state uncertainty, and interaction intent are estimated, and a user state object is generated.

[0074] The user state object includes cognitive load estimation, interaction intent strength, response latency, historical control results, and system confidence.

[0075] For example, when users continuously pause or replay near the same semantic anchor point, cognitive load estimation and interaction intent strength can be improved; when historical control actions do not receive clear feedback, or when the user's voice response remains in a low-confidence state, state uncertainty can be improved.

[0076] S7 determines whether an active voice challenge is triggered. Based on the current user state, the importance of the current semantic anchor, and the interruption cost of the active challenge, the active challenge trigger score is calculated according to the formula:

[0077] (3)

[0078] In the formula, This represents an estimate of cognitive load or state stress. This indicates uncertainty in the state; Indicates the strength of the interaction intent; Indicates the importance of the current semantic anchor; This indicates that the penalty item has been interrupted. , , , and These represent the weights of the corresponding evaluation items; Indicates the bias term; This represents the normalized mapping function.

[0079] When the score for proactive questioning does not reach the preset trigger threshold, the current playback status is maintained, and the user status and interaction feedback are monitored.

[0080] When the score for proactive questioning reaches the preset trigger threshold, the process of proactive voice interaction begins.

[0081] In deployment environments with high real-time control requirements, the weight corresponding to the importance of the current semantic anchor can be increased; in deployment environments with high continuous playback requirements, the weight corresponding to the interruption penalty can be increased. Therefore, proactive voice questioning is not triggered at every semantic anchor, but is initiated when the user state, anchor importance, and interruption cost all meet the triggering conditions.

[0082] The importance of the current semantic anchor can be determined based on the anchor score, anchor category, context summary, and the anchor's connection in the anchor relationship graph; the interruption penalty can be determined based on the current media playback continuity requirement, the time interval since the last challenge, and the frequency of historical challenges.

[0083] S8, initiate an active voice challenge and recognize the response, generate the active challenge content based on the context summary of the current semantic anchor, the anchor category, the associated anchor and the user status, and output the active challenge content through voice broadcast.

[0084] The active questioning content is used to confirm whether to maintain the current playback, return to the previous semantic position, or jump to the associated branch semantic anchor point.

[0085] Collect the user's voice response to the proactive inquiry, perform automatic speech recognition on the voice response to obtain the response text and automatic speech recognition confidence score; then perform natural language understanding on the response text to obtain intent label, control target and natural language understanding confidence score.

[0086] The response confidence is calculated according to the formula based on the confidence scores of automatic speech recognition, natural language understanding, the consistency between the response semantics and the current anchor context, and noise penalty:

[0087] (4)

[0088] In the formula, Indicates the confidence level of automatic speech recognition; Indicates the confidence level of natural language understanding; This indicates the consistency between the response semantics and the current anchor context; Indicates the noise penalty term; , , , These represent the weights of the corresponding evaluation items.

[0089] When the response confidence level reaches the preset response threshold, the identified intent label and control target will be sent to the control decision process.

[0090] When the response confidence level is lower than the preset response threshold, high-risk control actions are not executed directly. Instead, re-questioning, clarification confirmation, or downgrading of default control are performed.

[0091] The default downgrade control includes at least one of the following: continuous broadcast, brief pause, returning to the previous stable semantic anchor point, or recording the pending confirmation state. The specific method is determined based on the deployment configuration and control security constraints.

[0092] High-risk control actions include branch jumps that may significantly change the playback position or cause the device to deviate from the current semantic context, as well as other control actions determined by the deployment configuration. By employing degraded default controls in low-confidence states, erroneous jumps caused by speech recognition errors or semantic understanding errors can be avoided.

[0093] S9 generates control decisions by establishing a set of candidate control actions based on the semantic understanding results of the voice response, the current semantic anchor, the anchor relationship graph, the user state, and control constraints.

[0094] The candidate control action set includes at least a replay action, a backtracking action to the target semantic anchor point, and a branch jump action to the branch semantic anchor point.

[0095] The resume playback action applies when the user's response indicates that the current playback flow does not need to be changed, or when the user's proactive voice inquiry is not triggered.

[0096] Backtracking actions are suitable for situations where the user's response indicates a need to return to a previous semantic position, the current context is incomplete, or the system detects an uncertain state.

[0097] Branch jump actions are suitable when the user response explicitly points to a related branch, alternative path, or target semantic anchor.

[0098] For each candidate control action in the candidate control action set, the target control action is determined according to the formula, taking into account the matching degree between the response intent and the candidate control action, the relevance between the target semantic anchor and the current context, the consistency between the candidate control action and the historical control trajectory, and the control risk or interruption cost:

[0099] (5)

[0100] In the formula, Represents the set of candidate control actions; This indicates the degree of matching between the response intent and the candidate control action; This indicates the correlation between the target semantic anchor point corresponding to the candidate control action and the current context; This indicates the consistency between candidate control actions and historical control trajectories; This indicates the control risks or interruption costs that may arise from executing a candidate control action; , , , These represent the weights of the corresponding evaluation items; This indicates a specific target control action.

[0101] The comprehensive evaluation value of each candidate control action is calculated by formula (5), and the candidate control action with the largest comprehensive evaluation value is determined as the target control action, instead of directly executing control based on a single keyword in the voice response.

[0102] S10 executes media control and updates the status, invoking the local player, browser player, streaming media server, or edge control interface according to the target control action.

[0103] When the target control action is a resume playback action, maintain the current playback state; when the target control action is a backtracking action, switch the playback position to the timestamp corresponding to the target semantic anchor point; when the target control action is a branch jump action, switch the playback position to the target branch semantic anchor point in the anchor point relationship graph.

[0104] After the control action is completed, record the source anchor point, target anchor point, triggering reason, response confidence, execution time, execution result and exception information of this control action, and write the control execution result, user feedback, response confidence, error status and anchor point change back to the user state model.

[0105] Update the user state vector according to the formula and represent the anchor point relationship graph:

[0106] (6)

[0107] In the formula, Represents the current user state vector; This represents the updated user state vector; Indicates the update step size; This represents the state update function; Indicates the target control action that has been executed; Indicates the result of user response or control; Indicates an abnormal event; This represents a diagram showing the relationship between anchor points. Represents a set of semantic anchor points; It indicates the sequential relationship, backtracking relationship, branching relationship, or semantic association relationship between semantic anchors.

[0108] After completing the status update, the process returns to the playback monitoring process, continuing to execute user status monitoring, proactive voice questioning judgment, and media control, thus forming a closed-loop proactive control process. By continuously maintaining the user status model and anchor point relationship graph, the stability of semantic anchor point selection and control action decisions can be improved during multi-round interactions.

[0109] In real-time streaming media input scenarios, media segments are read using a sliding time window of fixed or adaptive length. After the streaming media is received, the video frames, audio segments, and text metadata in the current time window are processed synchronously, and the feature differences between the current time window and the previous time window are calculated.

[0110] When the boundary of a candidate event meets the semantic anchor point determination criteria, a semantic anchor point is immediately generated and written to the anchor point library or control table.

[0111] When the playback progress reaches the vicinity of the corresponding semantic anchor, the current user state is read. If the state uncertainty is low and the interruption penalty is high, the current playback is maintained; if the semantic anchor is of high importance, the user pauses or repeats operations, or historical feedback indicates that there may be a need for control, an active voice questioning is initiated.

[0112] After obtaining the user's voice response, determine whether to continue playback, rewind, or branch jump based on the response semantics and response confidence, and execute the corresponding media control actions.

[0113] For example, during a meeting video playback, when the visual image switches from the reporting page to the data analysis page, the speaker changes, and the subtitle topic vector shows a significant difference, the system can identify the corresponding position as a candidate event boundary. When the candidate event boundary meets the judgment conditions of equations (1) and (2), it generates a semantic anchor point. If the user pauses continuously or replays repeatedly after the semantic anchor point, and the active questioning trigger score reaches the preset trigger threshold, the system initiates a voice questioning whether to return to the starting position of the data analysis section, and performs a resume playback or rewind action based on the user's response.

[0114] In offline time-series streaming media scenarios, before the actual playback, the complete media content is traversed, and multimodal feature extraction, candidate event boundary detection, and anchor point scoring are performed sequentially, while an anchor point library and anchor point relationship graph are pre-built.

[0115] The anchor library can be searched by media source, timestamp, anchor category, anchor score, context summary, and jump relationship.

[0116] During playback, when the playback progress reaches the vicinity of the corresponding semantic anchor point, it is not necessary to recalculate all multimodal features. Instead, the pre-built anchor point context and relationships are directly read to determine the user state and make control decisions, thereby reducing the real-time computing pressure during playback.

[0117] When a user's speech response is affected by environmental noise, sentence breaks, overlapping speech, or recognition model errors, automatic speech recognition and natural language understanding may output low-confidence results.

[0118] Calculate the response confidence score before executing media control. When the response confidence score is lower than the preset response threshold, do not directly execute branch jump or large-span backtracking. Instead, generate secondary confirmation content based on the context of the current semantic anchor, or execute re-challenge, narrow down the range of candidate control actions, maintain continuous playback, or return to the previous stable semantic anchor.

[0119] When multiple consecutive response failures occur, the abnormal state is recorded and maintained for continuous playback to reduce the impact of error control on the continuity of time-series streaming media.

[0120] When a user's voice response indicates a need to switch to an associated location, subsequent path, or alternative media segment, candidate branch semantic anchors are retrieved in the anchor relationship graph.

[0121] Each candidate branch semantic anchor has a timestamp, context summary, branch weight, association type, and control constraints. The target branch semantic anchor is selected based on intent matching, anchor relevance, historical control consistency, and risk cost.

[0122] After switching the playback position to the target branch semantic anchor point, record the source anchor point, target anchor point, triggering reason, response confidence and execution result, and use this record to update the user state model and anchor point relationship graph.

[0123] Example 2

[0124] like Figure 1-3 As shown, the present invention provides a multimodal time-series active control device based on semantic anchors, including a streaming media access unit, a multimodal processing unit, a semantic anchor detection unit, an anchor instrumentation and indexing unit, a user state evaluation unit, an active voice interaction unit, a control decision and execution unit, and a log and state update unit.

[0125] The streaming media access unit is used to receive real-time or offline input time-series streaming media, read video frames, audio segments, text metadata, and playback status, and generate a unified media segment object according to a time window. The media segment object includes a segment identifier, a source identifier, a start timestamp, an end timestamp, a video reference, an audio reference, and a text reference.

[0126] The multimodal processing unit is connected to the streaming media access unit and is used to preprocess, align, and extract features from visual, audio, and text modalities, and output a unified multimodal feature package.

[0127] The multimodal processing unit performs sampling, scaling, denoising, and standardization on video frames, performs frame segmentation, noise reduction, endpoint detection, and acoustic feature preparation on audio segments, and performs sentence segmentation, cleaning, tagging, and timestamp correction on subtitles or text metadata.

[0128] The multimodal feature package includes visual change vectors, acoustic change vectors, text semantic vectors, and fused feature vectors.

[0129] The semantic anchor detection unit is connected to the multimodal processing unit and is used to generate candidate event boundaries based on visual changes, audio changes and text semantic changes between adjacent time windows, and to calculate the anchor score of the candidate event boundaries according to formula (1).

[0130] The semantic anchor detection unit is also used to judge the candidate event boundary according to formula (2), combined with the anchor trigger threshold, minimum time interval, smoothness score and stability condition, and determine the candidate event boundary that meets the condition as the semantic logic turning point.

[0131] The anchor instrumentation and indexing unit is connected to the semantic anchor detection unit and is used to generate semantic anchor identifiers, timestamps, context summaries, association features, jump relationships and control hooks for semantic logical turning points, and write the generated semantic anchors into the anchor library or control table.

[0132] The anchor instrumentation and indexing unit is also used to establish and maintain an anchor relationship graph based on the temporal order, semantic relevance, and branching relationship between semantic anchors.

[0133] The nodes in the anchor point relationship graph represent semantic anchor points, and the edges between nodes represent sequential relationships, backtracking relationships, branching relationships, or semantic association relationships.

[0134] The user state assessment unit is used to estimate the user's cognitive load, state uncertainty, and interaction intent based on user interaction logs, response latency, repetitive operations, historical control results, and optional state signals, and to generate a user state object.

[0135] The user status assessment unit is also used to obtain the importance of the current semantic anchor and the interruption penalty item, and to calculate the active questioning trigger score according to formula (3).

[0136] When the active questioning trigger score does not reach the preset trigger threshold, the user state evaluation unit outputs a continuous broadcast command; when the active questioning trigger score reaches the preset trigger threshold, the user state evaluation unit outputs interaction trigger information to the active voice interaction unit.

[0137] The active voice interaction unit is connected to the user state evaluation unit and the anchor instrumentation and indexing unit, respectively. When the active questioning triggering condition is met, it generates active questioning content based on the context summary of the current semantic anchor, the anchor category, the associated anchor and the user state, and outputs the active questioning content through voice broadcast.

[0138] The active voice interaction unit is also used to collect user voice responses, perform automatic speech recognition and natural language understanding on the user voice responses, and obtain response text, intent labels, control targets, automatic speech recognition confidence scores and natural language understanding confidence scores.

[0139] The active voice interaction unit calculates the response confidence based on the confidence of automatic speech recognition, the confidence of natural language understanding, the contextual consistency and the noise penalty according to formula (4).

[0140] When the response confidence level is insufficient, the active voice interaction unit performs re-questioning, clarification confirmation, or downgrades the default control; when the response confidence level meets the requirements, the response semantic result is sent to the control decision and execution unit.

[0141] The control decision and execution unit is connected to the active voice interaction unit, the anchor instrumentation and indexing unit, and the user state evaluation unit, respectively, and is used to construct a set of candidate control actions based on response semantics, current semantic anchors, anchor relationship diagrams, user state, and control constraints.

[0142] The candidate control action set includes at least a replay action, a backtracking action to the target semantic anchor point, and a branch jump action to the branch semantic anchor point.

[0143] The control decision and execution unit determines the target control action from the set of candidate control actions according to formula (5), based on the matching degree between the response intention and the candidate control action, the correlation between the target semantic anchor and the current context, the consistency between the candidate control action and the historical control trajectory, and the control risk or interruption cost.

[0144] The control decision and execution unit calls the local player, browser player, streaming media server or edge control component through the media control interface to perform playback continuation, backtrack to the target semantic anchor point or jump to the branch semantic anchor point.

[0145] The log and status update unit is connected to the control decision and execution unit, the user status evaluation unit, and the anchor instrumentation and indexing unit, respectively, and is used to record interaction logs, control logs, exception logs, and anchor change logs.

[0146] The log and status update unit is also used to write the user's voice response, response confidence, target control action, control execution result, error status and abnormal event back to the user state model, and update the user state vector and anchor point relationship diagram according to formula (6).

[0147] The updated user state object and anchor point relationship diagram are fed back to the user state evaluation unit and the control decision and execution unit for subsequent proactive questioning triggering and control action selection.

[0148] The streaming media access unit, multimodal processing unit, semantic anchor detection unit, anchor instrumentation and indexing unit, user state evaluation unit, active voice interaction unit, control decision and execution unit, and log and state update unit in the above device work together to realize multimodal time-series stream perception, semantic anchor detection, active voice questioning, response semantic understanding, media control execution, and state update.

[0149] The device in this embodiment can be implemented using microservices, middleware, player plugins, or an edge-cloud collaborative architecture. The various units can exchange media segment objects, feature packet objects, anchor record objects, user state objects, voice response objects, and control action objects via message queues, event buses, RESTful APIs, gRPC, or in-process calls.

[0150] Automatic speech recognition, natural language understanding, multimodal feature extraction, and media control interfaces can utilize local models, cloud services, or hardware acceleration modules, depending on the deployment environment.

[0151] The device can be deployed as a standalone server, an edge device, a player plugin, a browser extension, a streaming media control middleware, or a cloud-based intelligent control service.

[0152] When deployed locally, streaming media access, semantic anchor detection, and media control execution can be completed locally, while automatic speech recognition and natural language understanding can be performed using local models or remote services. When deployed in the cloud, media segment objects and user state objects can be sent to the cloud through an encrypted channel, and the generated control actions are returned to the media execution end through a controlled interface.

[0153] The units can also be connected using an event-driven approach. When a media segment object is generated, a segment arrival event is published; when a semantic anchor is generated, an anchor update event is published; when the active challenge trigger condition is met, an interaction request event is published; and when a control action is completed, a control feedback event is published, thereby supporting asynchronous processing, exception retries, and functional unit replacement.

[0154] Multimodal feature extraction can employ models capable of outputting visual, audio, or textual features; semantic anchor scoring can be implemented using linear models, tree models, neural network models, or a combination of rules and models; user status assessment can employ statistical rules, machine learning models, or reinforcement learning strategies; and anchor libraries can be implemented using relational databases, key-value databases, vector databases, graph databases, or in-memory indexes.

[0155] As long as the alternative method still executes the closed-loop process of "multimodal perception - semantic anchor instrumentation - state-triggered active questioning - response-driven flow control", it belongs to the implementation method of the technical solution of the present invention.

[0156] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multimodal temporal stream active control method based on semantic anchors, characterized in that, Includes the following steps: Receive time-series streaming media including at least two modalities of data, including video, audio and text, align each modal data to a unified timeline according to the media timestamp, and extract modal change features, contextual continuity features and historical interaction correlation features between adjacent time windows; Based on the features detected candidate event boundaries, anchor point scores for the candidate event boundaries are calculated, and semantic logic inflection points are determined when the anchor point scores satisfy the trigger threshold, minimum time interval, and smoothness stability conditions. Generate semantic anchors containing timestamps, context summaries, associations, and control hooks for the semantic logical turning points, and construct an anchor relationship graph based on the sequential relationship, backtracking relationship, branching relationship, or semantic association relationship between the semantic anchors; During the playback of time-series streaming media, the user status is updated based on user interaction feedback, and the user status, the importance of the current semantic anchor point, and the interruption penalty are combined to determine whether to trigger an active voice questioning. When an active voice challenge is triggered, the challenge content is generated based on the current semantic anchor, the user's voice response is identified, and the response semantics and response confidence are determined. Based on the response semantics, response confidence, anchor point relationship graph, and user state, the target control action is determined from resume playback, backtracking to the target semantic anchor point, and jumping to the branch semantic anchor point. The target control action is executed, and the user state and anchor point relationship graph are updated based on the execution result.

2. The multimodal temporal stream active control method based on semantic anchors according to claim 1, characterized in that, The step of aligning the modal data to a unified timeline according to the media timestamps includes: Media segment objects are generated according to a fixed time window or an adaptive time window adjusted according to the degree of change in the time-series streaming media; video data is sampled, scaled, denoised and standardized; audio data is framed, denoised and endpoint detected; text data is segmented, cleaned, marked and timestamped; and the processed modal data are bound to the same time window.

3. The multimodal temporal stream active control method based on semantic anchors according to claim 1, characterized in that, The anchor point score is determined based on the intensity of visual changes, the intensity of audio changes, the intensity of text semantic changes, the contextual coherence correction item, and the historical interaction relevance correction item. The candidate event boundary is determined as a semantic logic turning point only when the anchor point score reaches the anchor point trigger threshold, the time interval between the candidate event boundary and the previous effective semantic anchor point reaches the minimum time interval, and the smoothed anchor point score reaches the stability threshold.

4. The multimodal temporal stream active control method based on semantic anchors according to claim 3, characterized in that, The semantic anchor also includes anchor number, anchor score, anchor category, feature summary, adjacent anchors and jump relationship; the control hook is associated with the player control interface, streaming media server control interface or edge control interface so that backtracking action or branch jump action is executed according to the media position corresponding to the target semantic anchor.

5. The multimodal temporal stream active control method based on semantic anchors according to claim 1, characterized in that, The user interaction feedback includes at least one of the following: pause operation, replay operation, fast forward operation, rewind operation, voice response delay, low confidence feedback, and historical control results; based on the user interaction feedback, cognitive load, state uncertainty, and interaction intent intensity are determined, and an active questioning trigger score is calculated by combining the importance of the current semantic anchor point and the interruption penalty; when the active questioning trigger score reaches a preset trigger threshold, an active voice questioning is triggered.

6. The multimodal temporal stream active control method based on semantic anchors according to claim 5, characterized in that, The response confidence is determined based on automatic speech recognition confidence, natural language understanding confidence, consistency between the user's speech response and the current semantic anchor context, and noise penalty; When the confidence level of the response is lower than the preset response threshold, the branch jump action or other high-risk control action is not executed directly. Instead, at least one of the following is executed: re-questioning, clarification confirmation, narrowing the range of candidate control actions, maintaining continuous broadcast, short pause, returning to the previous stable semantic anchor point, or recording the pending confirmation state.

7. The multimodal temporal stream active control method based on semantic anchors according to claim 6, characterized in that, The determination of the target control action includes: For each candidate control action, the matching degree between the user's response intent and the candidate control action, the relevance between the target semantic anchor point pointed to by the candidate control action and the current context, the consistency between the candidate control action and the historical control trajectory, and the control risk of the candidate control action are determined. Based on the matching degree, relevance, consistency and control risk, the comprehensive evaluation value of each candidate control action is calculated, and the candidate control action with the largest comprehensive evaluation value is determined as the target control action.

8. A multimodal temporal stream active control device based on semantic anchors, used to implement the multimodal temporal stream active control method based on semantic anchors as described in any one of claims 1-7, characterized in that, include: A streaming media access unit is used to receive time-series streaming media that includes at least two modalities of data; The multimodal processing unit is used to align the data of each modality to a unified time axis and extract modality change features, contextual continuity features, and historical interaction correlation features. The semantic anchor detection unit is used to detect candidate event boundaries and determine semantic logic inflection points when the anchor scores of candidate event boundaries meet the trigger threshold, minimum time interval and smoothness stability conditions. Anchor instrumentation and indexing units are used to generate semantic anchors containing timestamps, context summaries, associations, and control hooks, and to construct an anchor relationship graph; The user status assessment unit is used to update the user status based on user interaction feedback, and to determine whether to trigger an active voice questioning by combining the user status, the importance of the current semantic anchor point, and the interruption penalty. An active voice interaction unit is used to generate question content, collect user voice responses, and determine response semantics and response confidence. The control decision and execution unit is used to determine and execute resume, backtrack, or branch jump actions based on response semantics, response confidence, anchor point relationship diagram, and user status. The log and status update unit is used to update the user status and anchor point relationship graph based on the execution results.

9. The multimodal temporal stream active control device based on semantic anchors according to claim 8, characterized in that, The anchor point instrumentation and indexing unit is configured to: in real-time monitoring mode, move the time window according to the playback progress of the time-series streaming media and continuously generate semantic anchor points; or, in offline pre-scanning mode, traverse the complete time-series streaming media before playback and pre-build an anchor point library and anchor point relationship graph; the anchor point library is indexed according to at least one of media source, timestamp, anchor point category, anchor point score, context summary and jump relationship.

10. The multimodal temporal stream active control device based on semantic anchors according to claim 9, characterized in that, The log and status update unit is used to record source semantic anchors, target semantic anchors, active challenge triggering reasons, response confidence, control actions, execution results and abnormal events, and feeds back the execution results to the user status evaluation unit and the anchor instrumentation and indexing unit to update the backtracking relationship, branching relationship or semantic association relationship between user status and semantic anchors.

Images