An intelligent interaction method and an interaction screen system supporting multi-modal input and output

Abstract

The application relates to an intelligent interaction method and an interaction screen system supporting multi-modal input and output, aiming to improve the intelligent processing capability of voice, text and visual data in complex interaction scenarios. Through millisecond-level voice and text segmentation, endpoint positioning, semantic label generation and multi-modal time alignment, dynamic extraction and priority scheduling of semantic anchor points are realized. Combined with high-confidence control flow, attention mask generation and fine-grained visual modeling, semantic distillation and cross-modal contrast learning are adopted to effectively optimize the consistency of visual features and semantics. Based on real-time network status and interaction demand, the processing granularity and sampling frequency are dynamically adjusted to ensure the adaptability and synchronization of semantic understanding. The technical scheme improves the accuracy, real-time performance and robustness of multi-modal interaction, and is suitable for intelligent voice assistants, human-computer interaction systems and the like.

Description

Technical Field

[0001] This invention relates to the field of multimodal semantic understanding and intelligent interaction technology, and in particular to an intelligent interaction method and interactive screen system that supports multimodal input and output. Background Technology

[0002] Current multimodal semantic understanding and intelligent cloud interaction technologies have made significant progress in the fusion and understanding of multimodal data such as speech, images, and text. Existing mainstream solutions generally employ two technical approaches: on the one hand, deep neural networks are used to project features from multiple modalities into a unified embedding space, achieving global alignment and fusion across modalities; on the other hand, sequence modeling structures (such as multi-level Transformers and cross-modal attention mechanisms) are used to process temporal features from different sources sequentially or in parallel, thereby improving the system's ability to understand complex multimodal scenarios.

[0003] In practical applications, multimodal modeling schemes that follow a unified vector space alignment approach often rely on general-purpose or specialized large-scale vision-language models. They achieve high-level semantic alignment through full data concatenation or encoding, utilizing end-to-end large-parameter models. Representative technologies of this type mainly focus on deep integration of machine vision and speech understanding, end-to-end cross-modal dialogue systems, and knowledge graph-based multimodal reasoning, and are widely used in intelligent interaction scenarios such as smart offices, remote education, smart healthcare, and smart cities. However, current technologies face several prominent problems: First, the feature spaces of different modalities (such as speech, image, and text) are highly heterogeneous, with significant differences in spatiotemporal distribution, representation density, data dimensionality, and arrival latency. Directly adopting global concatenation or unified encoding alignment will lead to the following problems: huge consumption of computational resources, high computational overhead of the full cross-modal attention matrix, and increased difficulty in controlling semantic consistency between information from different modalities, often resulting in semantic dilution or misalignment.

[0004] Secondly, existing technologies generally employ serial or synchronous processing paradigms, performing indiscriminate parallel processing on the raw signal data of all modalities. This is easily limited by the processing speed of high-latency modalities (such as image uploading and encoding), affecting the overall system's initial response and interactive smoothness. Especially in weak network environments, large-volume modal data such as images often experience stuttering due to bandwidth limitations, causing the dominant interactive streams (such as real-time voice and command text interaction) to lose their original timeliness advantage.

[0005] Secondly, cross-modal temporal alignment is challenging. Existing methods, such as those employing knowledge graphs, global reasoning, or adversarial completion, typically require lengthy pre-construction or stacked feature processing, increasing both system development and maintenance complexity, as well as the hardware burden on the terminal side and cloud computing bottlenecks. For high-concurrency user scenarios, existing technologies struggle to simultaneously balance timeliness, energy efficiency, and system response robustness.

[0006] Development trends show that the industry has placed higher demands on the real-time response capabilities and intelligent collaboration levels of multimodal intelligent cloud interaction systems, and there is an urgent need to improve the efficiency of multimodal fusion and timing collaboration strategies, especially in scenarios that are highly sensitive to latency, such as smart office, remote blended teaching, and mobile interactive terminals.

[0007] The shortcomings of existing technologies are mainly reflected in the following aspects: The heterogeneity of multimodal inputs leads to high computational complexity in direct fusion, making semantic alignment difficult and failing to meet the requirements of low latency and high efficiency. Traditional global feature concatenation and unified encoding paradigms are difficult to implement in scenarios with limited computing resources and bandwidth, and are especially unsuitable for real-time high-concurrency services.

[0008] The inability to effectively avoid the dragging problem of high-latency modalities on low-latency dominant interaction flows results in large delays in the first character of the overall interaction and large fluctuations in response, which seriously affects the user experience.

[0009] The lack of a time-driven collaborative mechanism centered on the evolution of user intent prevents the asynchronous incremental evolution of semantic understanding and non-blocking output, thus limiting the system's ability to adapt to dynamic interaction scenarios.

[0010] It is difficult to achieve efficient resource allocation and semantic collaboration by adaptively adjusting the modal processing rhythm based on network bandwidth in an edge-cloud collaborative architecture.

[0011] To address the aforementioned issues, there is an urgent need for an intelligent semantic understanding method that integrates asynchronous streaming, multimodal heterogeneity, and multi-temporal collaboration. This method should not only dynamically adapt to the latency characteristics of different modalities, effectively reducing the computational load of global fusion, but also perform temporal-driven distillation of multimodal representations based on the actual progress of each user intent, achieving low-latency, highly robust cross-modal understanding and intelligent interaction. This technological breakthrough has significant practical implications for improving the response speed, understanding accuracy, and business intelligence level of multimodal cloud-based intelligent interaction systems in real-world business scenarios. Summary of the Invention

[0012] This application provides an intelligent interaction method and interactive screen system that supports multimodal input and output, aiming to solve one of the problems or problems of the prior art mentioned in the background.

[0013] This application provides an intelligent interaction method that supports multimodal input and output, specifically including: Acquire speech stream data and text input data, perform semantic segmentation processing, and generate intent label sequences.

[0014] Based on the intent tag sequence, a semantic anchor sequence is constructed, and the semantic anchor sequence is input into the anchor scheduler as a priority scheduling object.

[0015] An image frame sequence is acquired, and asynchronous feature extraction is performed on the image frame sequence using the temporal information in the anchor point sequence to generate an attention mask to constrain the image encoder to focus on key regions.

[0016] Based on the attention mask constraint of the image encoder's computation focus, the local visual representation of the high-delay modality is divided into multiple micro-time windows to output a set of local visual representations corresponding to each micro-time window.

[0017] The intent label sequence is input into the distillation buffer, and the output of each micro-time window in the local visual representation set is compared and learned online with the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory.

[0018] Based on the visual feature trajectory and the intent label sequence, a semantic consistency index is calculated. It is then determined whether the index has reached a preset alignment verification threshold, and an anchor point fusion result marked as distilled or an incremental update request marked as incomplete is generated.

[0019] If it is determined that the anchor point fusion result of the distilled state has been generated, the anchor point fusion result of the distilled state is sent to the downstream task to perform non-blocking semantic evolution. If it is determined that the incremental update request of the incomplete state has been generated, new local visual representation sets are continuously received for iterative alignment.

[0020] This application provides an intelligent interactive screen system that supports multimodal input and output, specifically including: The semantic segmentation module is used to acquire speech stream data and text input data and perform semantic segmentation processing to generate intent label sequences. Anchor scheduling module is used to construct a semantic anchor sequence based on the intent tag sequence, and input the semantic anchor sequence as a priority scheduling object into the anchor scheduler; The attention mask module is used to acquire an image frame sequence, use the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence, and generate an attention mask to constrain the image encoder to focus on key regions. The micro-time window module is used to divide the local visual representation of the high-delay modality into multiple micro-time windows according to the computational focus of the image encoder constrained by the attention mask, so as to output a set of local visual representations corresponding to each micro-time window. The contrastive learning module is used to input the intent label sequence into the distillation buffer and perform online contrastive learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory. The consistency judgment module is used to calculate the semantic consistency index based on the visual feature trajectory and the intent label sequence, determine whether the index reaches the preset alignment verification threshold, and generate an anchor point fusion result marked as distilled or an incremental update request marked as incomplete. The task distribution module is used to send the anchor point fusion result of the distilled state to the downstream task to perform non-blocking semantic evolution if it is determined that the anchor point fusion result of the distilled state has been generated, and to continuously receive new local visual representation sets for iterative alignment if it is determined that the incremental update request of the incomplete state has been generated.

[0021] This application provides an intelligent interaction method and interactive screen system that supports multimodal input and output, which has the following beneficial effects: (1) By deploying a lightweight intent screening model at the perception end, a semantic anchor sequence with timestamps and confidence is generated. The key intent information in the original multimodal input is extracted and labeled in milliseconds in the form of structured slots, action verbs and core entities. This effectively avoids the need for real-time uploading and parallel processing of full image or video data in traditional multimodal fusion, and significantly reduces the front-end computing load and network transmission overhead. The semantic anchor sequence enters the cloud scheduling system as an independent control flow, driving the dynamic allocation of subsequent processing resources. This enables the system to ensure that the first response delay of voice-dominated interaction is controlled within hundreds of milliseconds in weak network environments such as 4G, which greatly improves the real-time performance and smooth experience of user interaction. It is especially suitable for high-concurrency scenarios that are highly sensitive to response sensitivity, such as remote education and smart office.

[0022] (2) The proposed “anchor-guided attention mask” mechanism and “distillation buffer” collaborative architecture realize the paradigm shift of the visual modeling process from global fine-grained encoding to local focused activation: the auxiliary channel dynamically generates attention constraint signals based on semantic anchors, triggers the extraction of deep visual features of key regions only in the corresponding time interval, and maintains low-power shallow buffers in other time periods, avoiding the redundant computation caused by the continuous operation of high-complexity image encoders in traditional schemes; at the same time, an online contrastive learning mechanism is introduced at the micro-time window granularity, using speech / text anchors as teacher signals to guide the semantic convergence path of local visual representations, and realizing the progressive distillation of cross-modal knowledge through gradient truncation and soft label transfer strategies, which not only avoids the high computational power consumption caused by full computation of cross-modal attention matrix, Transformer stacking or multi-stage adversarial completion, but also ensures the consistency and integrity of the multimodal semantic evolution process, significantly improving the resource utilization and inference efficiency of the system under limited computing power conditions.

[0023] (3) A non-blocking semantic evolution mechanism is adopted. Once a semantic anchor point completes multimodal alignment verification, it can be output to the downstream task module in advance without waiting for all data to be processed. This achieves the streaming response capability of "partial results available first", effectively overcoming the task blocking and latency accumulation problems caused by the reliance on global feature synchronization in traditional end-to-end models. The entire processing flow uses the evolution of user intent as a natural metronome, reconstructing cross-modal semantic alignment from a static spatial mapping to a dynamic temporal distillation process. This not only enhances the system's robustness to asynchronous and incomplete inputs, but also supports adaptive improvement of image semantic backfilling accuracy with changes in network bandwidth, possessing good elastic expansion capability and environmental adaptability. This technical path completely avoids the dependencies of knowledge graph pre-construction, global feature splicing, and complex parameter tuning, and constructs an efficient, lightweight, and interpretable closed-loop adaptive multimodal understanding system. It provides solid technical support for application scenarios such as campus football that require efficient connection of resources inside and outside the school, diverse content, and accurate response, and achieves synergistic optimization of recommendation relevance and diversity, system response speed, and semantic integrity. Attached Figure Description

[0024] Figure 1 This is the main flowchart of an intelligent interaction method that supports multimodal input and output.

[0025] Figure 2 This is a sub-flowchart of an intelligent interaction method that supports multimodal input and output.

[0026] Figure 3 This is another sub-flowchart of an intelligent interaction method that supports multimodal input and output. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0028] The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0029] like Figure 1 As shown, this application provides an intelligent interaction method that supports multimodal input and output, specifically including: S1: Acquire speech stream data and text input data, perform semantic segmentation processing, and generate intent label sequences.

[0030] S2: Based on the intent label sequence, construct the semantic anchor sequence and input the semantic anchor sequence as the priority scheduling object into the anchor scheduler.

[0031] S3: Obtain the image frame sequence, utilize the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence, and generate an attention mask to constrain the image encoder to focus on key regions.

[0032] S4: Based on the attention mask constraint of the image encoder's computation focus, the local visual representation of the high-delay modality is divided into multiple micro-time windows to output the set of local visual representations corresponding to each micro-time window.

[0033] S5: Input the intent label sequence into the distillation buffer, and perform online comparative learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory.

[0034] S6: Calculate the semantic consistency index based on the visual feature trajectory and intent label sequence, determine whether the index has reached the preset alignment verification threshold, and generate anchor point fusion results marked as distilled or incremental update requests marked as incomplete.

[0035] S7: If it is determined that the anchor point fusion result of the distilled state has been generated, the anchor point fusion result of the distilled state is sent to the downstream task to perform non-blocking semantic evolution. If it is determined that an incremental update request of the incomplete state has been generated, new local visual representation sets are continuously received for iterative alignment.

[0036] Step S1: Acquire speech stream data and text input data, perform semantic segmentation processing, and generate an intent label sequence. Specifically, this includes: S1.1: Acquire raw speech stream data from the microphone array and raw text input data from the keyboard or touch interface, perform pre-emphasis filtering on the raw speech stream data to enhance high-frequency components, and perform character encoding standardization on the raw text input data to output a standard speech signal frame sequence optimized for signal-to-noise ratio and a standardized text character stream with a unified encoding format.

[0037] The process acquires raw speech stream data from a microphone array and raw text input data from a keyboard or touch interface. Pre-emphasis filtering is performed on the raw speech stream data to enhance high-frequency components, and character encoding standardization is performed on the raw text input data to output a standard speech signal frame sequence optimized for signal-to-noise ratio and a standardized text character stream with a unified encoding format. Sub-step S1.1 serves as the data preprocessing entry point for the multimodal semantic understanding process, aiming to eliminate the physical differences and noise interference from heterogeneous input sources, providing a high-fidelity, standardized basic data stream for subsequent millisecond-level semantic segmentation. For the raw speech stream data, the pre-emphasis filter in the digital signal processing module is first invoked, setting the pre-emphasis coefficient to 0.97. A high-pass filter operation is performed on the raw PCM audio data with a sampling rate of 16kHz. By enhancing the energy of the high-frequency bands, the high-frequency attenuation of the speech signal caused by glottal radiation during pronunciation is compensated, generating a spectrally balanced pre-emphasized speech signal. The discrete-time signal value after pre-emphasis is calculated using the following formula: in, Indicates the first The pre-emphasized signal amplitude at each sampling point Indicates the first The original signal amplitude at each sampling point The pre-emphasis coefficient is set between 0.9 and 1.0, with 0.97 being preferred to balance high-frequency enhancement and noise suppression. Subsequently, the pre-emphasized speech signal is subjected to frame-by-frame windowing processing. A Hamming window function with a frame length of 25ms and a frame shift of 10ms is selected for sliding truncation, which transforms the continuous speech stream into a set of overlapping short-time stationary signal frames, effectively suppressing spectral leakage and preserving local time-frequency characteristics.

[0038] For the original text input data, a character encoding standardization engine is simultaneously activated to detect the encoding format of the input text. If it is identified as non-UTF-8 encoding, a conversion operation is performed to uniformly convert it to UTF-8 encoding format to eliminate the risk of garbled characters. The converted text stream is then subjected to Unicode normalization processing, replacing combined character sequences with single pre-combined characters according to the NFC (Normalization to Canonical Composition) standard to ensure the consistency of characters with the same semantic meaning at the binary level.

[0039] Further full-width / half-width character conversion is performed, mapping full-width letters, numbers, and punctuation marks to their corresponding half-width characters. Simultaneously, leading and trailing whitespace and invisible control characters are removed to generate a clean, standardized text character stream. Finally, amplitude normalization is applied to the standard speech signal frame sequence, compressing its dynamic range to the [-1, 1] interval to prevent subsequent quantization error accumulation. The standardized text character stream undergoes length verification, removing redundant trailing characters that exceed the maximum token limit for a single interaction.

[0040] Through the above parallel preprocessing mechanism, the original heterogeneous multimodal data obtained in the previous step is transformed into a standard speech signal frame sequence with optimized signal-to-noise ratio and a standardized text character stream with a unified encoding format. This achieves standardized alignment of multimodal input data at the physical and logical layers, significantly reducing the computational complexity and false recognition rate in the subsequent endpoint detection and semantic parsing stages.

[0041] S1.2: Based on the standard speech signal frame sequence, the sliding window short-time energy zero-crossing rate joint detection algorithm is used to perform endpoint localization processing, and based on the standard text character stream, the punctuation marks and semantic pause rules are used to perform logical segmentation processing to generate a set of discrete speech segments and a set of discrete text segments with clear start and end boundary marks.

[0042] The input data for this step is the standard speech signal frame sequence with optimized signal-to-noise ratio (SNR) and the standardized text character stream with unified encoding format, output from the previous step S1.1. For the standard speech signal frame sequence, a sliding time window with a length of 20 milliseconds and a frame shift of 10 milliseconds is constructed, and the speech sampling points within the window are processed frame by frame. The short-time energy value of the speech signal within the current sliding window is calculated, and the energy feature parameter reflecting the speech intensity is obtained by summing the squares of the amplitudes of all sampling points within the window.

[0043] The system synchronously calculates the number of symbol changes between adjacent sampling points within the current sliding window and statistically analyzes the zero-crossing rate to characterize the frequency distribution and voiced / unvoiced properties of the speech signal. A dynamic dual-threshold determination mechanism is implemented, setting the short-time energy threshold to three times the root mean square energy of the background noise and the zero-crossing rate threshold to 1.5 times the average zero-crossing rate of high-frequency noise. When the short-time energy value of three consecutive sliding windows is lower than the energy threshold and the zero-crossing rate is higher than the zero-crossing rate threshold, it is determined as the start point of a silence or noise segment; when the short-time energy value of three consecutive sliding windows is higher than the energy threshold and the zero-crossing rate is lower than the zero-crossing rate threshold, it is determined as the start point of a valid speech segment.

[0044] Based on the above start and end point determination results, endpoint detection logic is executed to remove invalid silent segments at the beginning and end, extract discrete speech segments containing complete semantic information, and mark each segment with a start and end timestamp accurate to the millisecond level. For the normalized text character stream, a punctuation index table based on regular expressions is established to identify the position coordinates of separators such as periods, question marks, exclamation marks, and commas in the character stream. Combining semantic pause rules in natural language processing, the lexical density and syntactic structural integrity between adjacent punctuation marks are analyzed to determine whether there are abnormal sentence breaks or missing punctuation due to speech recognition errors. For long text sequences with missing punctuation, a lightweight syntactic dependency parsing model is used to predict potential semantic boundaries and insert virtual segmentation markers.

[0045] Based on the identified real punctuation positions and predicted virtual segmentation markers, the continuous stream of canonical text characters is divided into several independent sets of discrete text segments. Each discrete text segment is assigned a logical temporal identifier corresponding to its position in the original input stream, ensuring the traceability of text and speech segments in the temporal dimension.

[0046] By employing a sliding window short-time energy zero-crossing rate joint detection algorithm and a punctuation semantic pause rule, continuous unstructured multimodal inputs are transformed into a set of discrete speech segments and a set of discrete text segments with clear start and end boundary markers. This achieves preliminary structured alignment of multimodal data at the temporal level, providing standardized input units for subsequent lightweight streaming speech recognition and named entity recognition, and significantly reducing the computational redundancy of subsequent semantic analysis modules.

[0047] S1.3: Call a lightweight streaming speech recognition model to perform phoneme decoding on the set of discrete speech segments to generate candidate transcribed text, and merge the candidate transcribed text with the set of discrete text segments into a domain-adaptive named entity recognition model to perform entity extraction and action verb matching to output a list of raw semantic units carrying preliminary semantic labels.

[0048] The system receives the set of discrete speech segments and the set of discrete text segments with clearly defined start and end boundaries from step S1.2 as input data objects. For each frame of audio signal in the discrete speech segment set, a streaming speech recognition model deployed on a lightweight computing node at the edge or in the cloud performs forward inference. This streaming speech recognition model adopts an encoder-decoder structure based on the Transformer architecture and introduces a causal convolution module to limit the leakage of future information and ensure real-time performance. Internally, the model maintains a dynamically expanding attention state cache, which is gradually updated with the incoming new audio frames.

[0049] The optimal path is found in the phoneme probability distribution space using a beam search algorithm, mapping the acoustic feature sequence to the corresponding phoneme sequence. Dictionary lookup and language model re-scoring are then performed on the generated phoneme sequence to eliminate homonyms and generate candidate transcribed text with timestamp alignment information. Simultaneously, word segmentation is performed on the discrete text fragment set, using a sub-word unit partitioning strategy based on BPE (Byte-Pair Encoding) to decompose the normalized text character stream into a fine-grained semantic token sequence.

[0050] The generated candidate transcribed text and the decomposed text token sequence are merged in both spatiotemporal dimensions to construct a unified multimodal text input stream. This multimodal text input stream is then fed into a domain-adaptive named entity recognition model. This model is pre-tuned on a specific vertical domain corpus and exhibits high sensitivity to technical terms, personal names, place names, and specific action verbs. The model first extracts contextual features of the text sequence through a bidirectional long short-term memory network to capture long-distance dependencies.

[0051] Subsequently, a conditional random field layer is used to predict labels for each token, identifying entity boundaries and category attributes. For action verb matching, a domain-specific verb ontology is constructed, and the semantic similarity between the extracted verbs and standard action prototypes in the ontology is calculated. A cosine similarity metric is used to filter action verbs with confidence scores higher than a preset threshold, and these are mapped to standardized intent action labels. The identified core entities, standardized action verbs, and their start and end position indices in the original input stream are then structurally encapsulated.

[0052] Through the above processing method, the discrete fragments from the previous step are transformed into a list of raw semantic units carrying preliminary semantic labels, entity types, and action attributes, realizing the initial transformation from unstructured media data to structured semantic information, and providing standardized semantic primitives for subsequent multimodal temporal alignment.

[0053] S1.4: Based on the temporal overlap relationship in the original semantic unit list, a dynamic time warping algorithm is used to perform multimodal temporal alignment processing. The confidence score and intent slot filling value of each semantic unit are calculated according to the alignment result to generate an intent tag sequence containing a precise timestamp interval, confidence value, intent slot field and core entity field.

[0054] Obtain the list of raw semantic units carrying preliminary semantic labels from the output of the preceding step S1.3. The list includes candidate transcribed text sequences corresponding to discrete speech segments and their timestamp intervals, as well as entity extraction results and action verb matching labels corresponding to discrete text segments. Perform temporal feature extraction operations on the speech modal data and text modal data in the raw semantic unit list to construct speech feature time series and text feature time series respectively. The speech feature time series is generated based on the start and end timestamps and duration of each speech segment, while the text feature time series is generated based on the input completion timestamp and semantic pause interval of each text segment.

[0055] A nonlinear mapping relationship between speech feature time series and text feature time series is constructed using a dynamic time warping algorithm. The optimal path that minimizes the global alignment cost is found by calculating the cumulative distance matrix between the two sequences on the time axis. Specifically, the speech sequence length is defined as N, and the text sequence length as M. An N×M distance matrix D is constructed, where element d(i,j) represents the local distance between the i-th unit of the speech sequence and the j-th unit of the text sequence. This local distance is calculated based on a weighted average of the timestamp overlap rate and semantic label similarity. The cumulative distance matrix C is calculated using a recursive formula, as follows: in, The th in the cumulative distance matrix Line 1 The value of the column, For local distance, The function selects the minimum cumulative distance among three adjacent predecessor nodes as the baseline value of the current node to ensure path monotonicity constraints. Starting from the bottom right corner C(N,M) of the matrix, it traces backward to the top left corner C(1,1), determining the optimal pairing relationship between speech units and text units based on the backtracking path to resolve temporal misalignment issues caused by differences in speaking speed or input delay. Based on the determined optimal pairing relationship, fusion processing is performed on multimodal semantic units with temporal overlap, and the confidence score of each fused unit is calculated. This score is jointly determined by the speech recognition confidence, text input determinism, and the temporal alignment tightness of the two, calculated using the following formula: in, To calculate the overall confidence score, For speech recognition confidence, For text input determinism, For time overlap normalization coefficients, , , These are preset weight coefficients, satisfying α+β+γ=1. Intent slots are filled based on the fused semantic information, mapping action verbs, core entities, and modifiers to a predefined intent framework. If a slot is missing in one modality but has a high-confidence counterpart in another, cross-modal completion is performed. An intent tag sequence is generated, containing a timestamp interval accurate to milliseconds, quantized confidence values, complete intent slot fields, and core entity fields.

[0056] By performing multimodal temporal alignment processing and calculating confidence scores and slot filling values ​​through a dynamic time warping algorithm, the original semantic unit list from the previous step is transformed into an intent tag sequence with spatiotemporal consistency and high credibility. This achieves precise synchronization and logical complementarity of multimodal semantics in the time dimension, providing a standardized data foundation for the subsequent construction of semantic anchor sequence with independent control flow attributes.

[0057] For example, in a smart meeting scenario, a user simultaneously inputs the voice command "Please check the sales report for last quarter" via microphone and generates a text command by clicking the "Q3Sales" button on the touchscreen. S1.3 outputs a list of raw semantic units, where the voice units contain the transcribed text "View," "last quarter," and "sales report," with corresponding timestamp intervals of [100ms, 300ms], [350ms, 600ms], and [650ms, 900ms], respectively, and recognition confidence scores of 0.92, 0.88, and 0.95; the text units contain the entities "Q3" and "Sales," with input timestamps of [200ms] and [210ms], respectively, and input determinism of 1.0 for both. The voice feature time series V={v1,v2,v3} and the text feature time series T={t1,t2} are constructed. Calculate the local distance matrix d(i,j). For example, d(1,1) is calculated based on the semantic similarity (low) and temporal overlap (none, as the text is an instantaneous event, the window needs to be expanded to 50ms before and after) of "View" and "Q3", resulting in a larger value; d(2,1) is calculated based on the high semantic similarity and temporal proximity of "Last Quarter" and "Q3". Assume the weight coefficients α=0.4, β=0.4, and γ=0.2. After dynamic time warping and backtracking, the optimal pairing is determined as follows: voice unit v2 (Last Quarter) is aligned with text unit t1 (Q3), voice unit v3 (Sales Report) is aligned with text unit t2 (Sales), and voice unit v1 (View) has no direct text correspondence but is retained as an action verb. Calculate the confidence score of the fusion unit: For the v2-t1 pair, P_voice=0.88, P_text=1.0, Overlap=0.8 (calculated based on the extended window overlap), then S_21=0.40.88+0.41.0+0.20.8=0.352+0.4+0.16=0.912. For the v3-t2 pair, P_voice=0.95, P_text=1.0, Overlap=0.75, then S_32=0.40.95+0.41.0+0.20.75=0.38+0.4+0.15=0.93. The final generated intent label sequence is as follows: the intent action is "View", the time interval is [100ms, 900ms], and the core entity slots are filled with {Time: "Q3 / Last Quarter", confidence level: 0.912, interval: [350ms, 600ms]} and {Object: "Sales / Sales Report", confidence level: 0.93, interval: [650ms, 900ms]}. This sequence eliminates the ambiguity of a single modality and significantly improves the accuracy and timeliness of semantic understanding.

[0058] Step S2: Based on the intent label sequence, construct a semantic anchor sequence and input the semantic anchor sequence as the priority scheduling object into the anchor scheduler. Specifically, this includes: S2.1: Obtain the intent tag sequence containing timestamps, confidence scores, intent slots, and core entities. Perform continuous temporal interval mapping on the discrete timestamps in the intent tag sequence to generate a set of multimodal semantic occurrence temporal intervals with clear start and end boundaries.

[0059] It should be noted that the semantic anchor sequence is a structured intent label sequence after temporal mapping and priority weighting, used to drive the temporal alignment of subsequent visual feature extraction. The anchor scheduler is a task scheduling unit based on a hybrid mechanism of time-slice round-robin and priority preemption, used to dynamically allocate computing resources according to the weight and temporal information of the semantic anchors.

[0060] The system receives an intent tag sequence from the output of the preceding step S1, containing timestamps, confidence levels, intent slots, and core entities. The timestamps in this sequence are represented as discrete sampling points or segment start markers. The system parses the discrete timestamp data in the intent tag sequence, extracting the original values ​​of the start time t_start and end time t_end for each semantic unit. For the millisecond-level fragmented timestamps generated by the speech modality, a temporal continuity check is performed to identify whether the time gap between adjacent semantic units exceeds a preset silence threshold. If the detected time gap is less than the preset threshold, it is determined to be a natural continuation of the same semantic event. A merging operation is then performed on adjacent discrete timestamps, logically connecting the end time of the previous segment with the start time of the next segment.

[0061] For overlapping timestamp intervals, a maximum confidence priority principle is adopted, retaining time periods with higher confidence scores as effective semantic coverage intervals and eliminating redundant timestamps with low confidence. Based on the merged time boundaries, a continuous time axis mapping relationship is constructed, mapping discrete intent labels to a continuous time dimension to form closed time intervals [T_i, T_j]. Boundary smoothing processing is performed on the generated time intervals, introducing a small time buffer Δt to compensate for clock synchronization errors of the multimodal acquisition equipment, ensuring that the start and end boundaries of the time intervals can completely cover the corresponding speech waveforms and text segments.

[0062] Through the above continuous temporal interval mapping process, the discrete and fragmented timestamps in the previous step are transformed into a set of multimodal semantic occurrence temporal intervals with clear start and end boundaries, realizing the transformation from point-like time markers to segment-like semantic temporal domains, and providing a precise time reference for the spatiotemporal alignment of subsequent semantic anchors.

[0063] S2.2: Based on the set of time intervals for the occurrence of multimodal semantics and the corresponding intent slots, perform semantic anchor encapsulation operations to strongly bind and fuse time dimension information and intent dimension information to generate original semantic anchor data units carrying complete context information.

[0064] S2.3: Utilize the confidence parameter in the original semantic anchor data unit to perform priority weighting calculation on the original semantic anchor data unit to generate a weighted semantic anchor sequence with dynamic weight attributes, thereby establishing the execution order of each anchor in concurrent processing.

[0065] Obtain the raw semantic anchor data units with complete contextual information generated in the previous steps, and extract the confidence scores, intent slot fill rates, and core entity matching degrees contained therein as the initial parameter set for priority calculation. Perform normalization processing on the extracted confidence scores, mapping them to the standard range of 0 to 1 to eliminate the dimensional influence caused by the differences in the output probability distributions of different modality recognition models.

[0066] The intent clarity coefficient is calculated based on the intent slot fill rate. When the slot fill rate reaches a preset high threshold, a higher intent clarity weight is assigned, reflecting that the semantic fragment has a stronger business logic orientation. Semantic richness is evaluated in conjunction with core entity matching degree. By querying the association density of entities in the domain knowledge graph, the information gain value of an entity in a specific interaction scenario is determined. A dynamic priority weighting function is constructed to linearly combine the normalized confidence score, intent clarity coefficient, and semantic richness, as expressed by the formula: in For final priority scoring, To normalize the confidence level, The intention clarity coefficient. For semantic richness, , , These are the corresponding dynamically adjusted weight coefficients. The values ​​of weight coefficients w1, w2, and w3 are adjusted in real time based on the current system load. In high-load scenarios, the confidence weight w1 is increased to prioritize high-reliability data, while in low-load scenarios, the intent clarity weight w2 is increased to improve the understanding depth of complex intents. Based on the calculated final priority score P, the original semantic anchor data units are sorted in descending order to generate a weighted semantic anchor sequence with a strict execution order. A unique time-series scheduling identifier is assigned to each anchor in the sequence to ensure accurate tracking of the processing status and resource consumption of each anchor during subsequent concurrent processing.

[0067] By using the dynamic weight calculation and sorting process described above, the static raw semantic anchor data units generated in the previous step are transformed into weighted semantic anchor sequences with dynamic priority attributes. This enables the cloud computing resources to be allocated towards high-value, high-confidence semantic fragments, significantly reducing the processing latency of key interactive commands and improving the overall system response efficiency.

[0068] S2.4: Construct a control signaling channel independent of the original media stream based on the weighted semantic anchor sequence, and encapsulate the weighted semantic anchor sequence into a high-priority control data packet to generate an independently transmitted control flow semantic anchor sequence.

[0069] The weighted semantic anchor sequence with dynamic weight attributes generated in the preceding steps is obtained. The timestamp interval, confidence value, intent slot, and core entity field contained in each anchor data unit in the sequence are parsed and used as the raw input data for constructing an independent control signaling channel. Based on the data structure characteristics of the weighted semantic anchor sequence, a lightweight binary encapsulation protocol is defined. The protocol header includes a version number, payload type identifier, and sequence number. The payload uses a variable-length encoding format to store the serialized intent slot key-value pairs and time index, minimizing data packet size and reducing network transmission overhead.

[0070] By utilizing the priority parameters in the weighted semantic anchor sequence, the transmission priority level of each anchor data unit is calculated. Anchors with high confidence and located on the critical path of interaction are marked as having the highest priority, while anchors with low confidence or background noise are marked as having normal priority. This establishes the sorting rules for data packets in the transmission queue. Based on the calculated transmission priority level, an independent logical channel identifier is assigned to each weighted semantic anchor data unit. This identifier is distinct from the conventional data channel carrying large-volume media data such as raw image frames and audio waveforms, thereby achieving physical isolation between the control flow and the media stream at the logical level.

[0071] The encapsulated binary data blocks are mapped to the user datagram payload area of ​​the real-time transport protocol, and a differentiated service code point field is set in the protocol header, with its value configured as the accelerated forwarding service level, to ensure that network devices prioritize the processing of such control signaling in congested conditions.

[0072] Through the construction of independent channels and high-priority encapsulation processing, the weighted semantic anchor sequence of the previous step is transformed into an independent control flow attribute semantic anchor sequence with independent transmission paths and low latency characteristics. This enables the cloud-based intelligent processing module to quickly perceive the interaction intent and pre-schedule resources, effectively avoiding the semantic understanding delay problem caused by the blockage of large-volume media data.

[0073] S2.5: Receive a sequence of semantic anchors with independent control flow attributes, inject it into the input queue of the anchor scheduler in the cloud intelligent processing module to trigger the anchor scheduler to start an asynchronous resource allocation process driven by intent evolution, and finally output the registered semantic anchor sequence marked as a priority scheduling object.

[0074] The system receives semantic anchor sequence data packets with independent control flow attributes, encapsulated in previous steps. It parses the protocol identifier and payload length fields in the packet header and verifies data integrity to ensure correct execution of subsequent scheduling logic. Weighted semantic anchor units are extracted from the packet payload, and the dynamic weight attribute value, start and end points of the occurrence time interval, intent slot index, and confidence score parameters embedded in each unit are read. An anchor scheduler input queue based on a hybrid mechanism of time-slice round-robin and priority preemption is constructed. The parsed semantic anchor units are sorted in descending order according to their dynamic weight attribute values. For anchor units with the same weight value, a secondary sort is performed in ascending order based on the start time point of the occurrence time interval to establish the initial execution order.

[0075] The system initializes the computing resource pool status register of the cloud-based intelligent processing module, monitors the current GPU memory utilization, CPU core load, and memory bandwidth utilization in real time, and generates a resource availability vector reflecting the current system's idle computing capacity. The sorted semantic anchor sequence is then injected sequentially into the priority queue buffer of the anchor scheduler, triggering the intent evolution driving engine within the scheduler. This engine dynamically allocates asynchronous feature extraction threads and semantic distillation computing units based on the resource availability vector and the temporal urgency of the anchor sequence.

[0076] For high-priority and time-sensitive semantic anchors, the scheduler immediately locks the corresponding image frame micro-time window processing channel and preloads the relevant context cache; for low-priority or time-flexible anchors, they are suspended to a waiting queue and a timeout wake-up flag is set. The scheduler outputs a set of control instructions for the registered semantic anchor sequence, which includes thread ID, memory address mapping table, and execution priority flag. This set of instructions directly points to the specific computation node of the downstream asynchronous feature extraction module.

[0077] By using the asynchronous resource allocation processing method based on dynamic weight sorting and resource status awareness, the independent control flow attribute semantic anchor sequence generated in the previous step is transformed into parallel task scheduling instructions that can be executed in real time in the cloud, thereby achieving the expected technical effect of on-demand accurate allocation of computing resources and low-latency response in the process of multimodal semantic understanding.

[0078] For example, in a smart meeting scenario, the cloud server receives a data packet containing three semantic anchors: anchor A (weight 0.95, time interval [100ms, 500ms], intent "turn on projection"), anchor B (weight 0.85, time interval [600ms, 1200ms], intent "show chart"), and anchor C (weight 0.95, time interval [100ms, 400ms], intent "mute microphone"). After the scheduler parses the data packet, it first compares the weights. Anchor A and C have the same weight, and anchor C has a higher weight than anchor B. Next, it compares the starting points of the time intervals. Anchor C starts at 100ms, while anchor A starts at 100ms but lasts slightly longer, or according to secondary sorting rules (such as entity complexity), it is assumed that C takes precedence over A, and B last. The sorting result is C, A, B. The system detects that the current GPU memory utilization is 60%, the CPU load is 40%, and the resource availability vector indicates sufficient resources. The scheduler immediately assigns anchor C to GPU thread 1 for image region locking and feature extraction, while pre-allocating GPU thread 2 to standby for anchor A, and placing anchor B in a waiting queue. If network fluctuations cause a delay in the arrival of image data for anchor C, the scheduler, based on a timeout wake-up flag, automatically switches to execute the task for anchor A after a 50ms wait, avoiding idle computing resources. In the final output set of registered semantic anchor sequence control instructions, anchor C is marked as "in execution," with associated thread ID T1 and memory address 0x7F...; anchor A is marked as "ready," with associated thread ID T2; and anchor B is marked as "waiting." This mechanism ensures that critical intents (such as mute) receive millisecond-level responses under high-concurrency interaction, significantly reducing user-perceived latency, while guaranteeing the orderliness of multimodal data processing and resource utilization efficiency.

[0079] like Figure 2 As shown, step S3: Obtain the image frame sequence, utilize the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence, and generate an attention mask to constrain the image encoder to focus on key regions. Specifically, this includes: S3.1: Obtain the complete image frame sequence received by the cloud intelligent processing module and the semantic anchor point sequence with independent control flow attributes from the previous steps. Based on the timestamps and occurrence time intervals contained in the semantic anchor point sequence, perform time-series slice mapping processing on the complete image frame sequence to generate a subset of micro-time windows of image frames that strictly correspond to each semantic anchor point.

[0080] S3.2: Based on a subset of image frames with micro-time windows, a lightweight spatial saliency detection algorithm is used to perform foreground target localization processing to extract an initial set of region of interest coordinates containing user gestures, facial expressions, or interactive objects.

[0081] The image encoder is a deep neural network model based on a convolutional neural network or a visual Transformer structure, used to convert input image frames into high-dimensional feature vectors. As an example, and not a limitation, the image encoder can employ a lightweight MobileNetV3 architecture (such as the backbone network used in embodiment S4.4 of this application), achieving feature extraction with low computational overhead through its inverse residual module and efficient channel attention mechanism; alternatively, it can employ a ViT (VisionTransformer) architecture, capturing global image context information through a multi-head self-attention mechanism. Regardless of the specific architecture used, the computational focus of the image encoder is constrained by the attention mask of this application, ensuring that it performs fine-grained feature extraction operations only in key visual regions.

[0082] The system receives a subset of micro-time window image frames generated in step S3.1 that strictly corresponds to a specific semantic anchor point. This subset contains continuous video frame data captured within the time interval defined by the semantic anchor point. A color space conversion operation is performed on each original RGB image frame in the micro-time window image frame subset, mapping the three-channel red, green, and blue pixel values ​​to the YCbCr or HSV color space to separate the luminance and chrominance components and eliminate interference from drastic changes in illumination intensity on foreground target detection.

[0083] A grayscale gradient map is constructed based on the separated luminance components. The Sobel operator is used to calculate the gradient magnitude of pixels in both the horizontal and vertical directions, generating a two-dimensional gradient matrix reflecting the intensity of image edge texture. A Gaussian blur kernel function is used to smooth the two-dimensional gradient matrix, suppressing false edge responses caused by high-frequency noise and obtaining a denoised salient edge distribution map. Combining the saturation information of the chrominance components, a color contrast feature map is constructed. The mean Euclidean distance between the current pixel and its neighboring pixels in the hue and saturation dimensions is calculated to quantify the degree of color singularity in local regions.

[0084] The denoised salient edge distribution map and the color contrast feature map are weighted and fused, with the weight coefficients dynamically adjusted according to scene lighting conditions to generate an initial spatial salient heatmap. A global thresholding operation is performed on the initial spatial salient heatmap, using Otsu's maximum inter-class variance method to automatically determine the binarization segmentation threshold, transforming the heatmap into a foreground candidate region mask. Morphological opening operations are performed on the foreground candidate region mask, using structuring elements to remove isolated noise points and smooth region boundaries, extracting target blocks with connected component areas greater than a preset minimum threshold. The coordinates of the minimum bounding rectangle for each retained target block are calculated, including the x and y coordinates of the top-left corner and the width and height dimensions, forming an initial set of region of interest coordinates.

[0085] By employing a lightweight spatial saliency detection algorithm and morphological post-processing workflow, a subset of micro-temporal window image frames is transformed into an initial set of region of interest coordinates containing user gestures, facial expressions, or the location information of interactive objects. This enables rapid localization and coarse-grained screening of key visual elements in high-latency modalities, providing accurate spatial prior data for subsequent semantic-spatial correlation calculations.

[0086] S3.3: Based on the initial set of region of interest coordinates and the intent slots in the semantic anchor sequence, perform semantic-spatial correlation calculation to filter out a candidate set of key visual regions that highly match the current coarse-grained intent label.

[0087] The system receives the initial set of region of interest coordinates output from step S3.2. This set contains the bounding box coordinates and saliency scores of all foreground targets detected within the micro-time window. Simultaneously, it obtains the registered semantic anchor sequence from step S2.5, extracting the intent slot fields and core entity fields that overlap with the current micro-time window. Semantic vectorization mapping is performed on the intent slot fields, using a pre-trained domain-adaptive word embedding model to convert text labels such as action verbs and interactive objects into high-dimensional dense vectors, constructing an intent semantic feature space.

[0088] For each candidate region in the initial set of region of interest coordinates, its corresponding visual context features are extracted. A lightweight convolutional neural network is used to extract the texture, shape, and color histogram features within the region, forming a visual representation vector. A semantic-spatial correlation calculation model is established, mapping the intention semantic feature vector and the visual representation vector to the same alignment space. The cosine similarity between the two is calculated as the initial matching score. A spatiotemporal constraint factor is introduced. Based on the time sequence interval defined in the semantic anchor point and the timestamp deviation of the current image frame, a time decay weight is calculated to weight and correct the initial matching score, thereby enhancing the matching degree of regions with high temporal consistency.

[0089] A multimodal attention mechanism is employed to normalize the corrected matching scores, generating an attention weight distribution for each region of interest relative to the current semantic anchor. A dynamic confidence threshold is set, adaptively adjusted based on the confidence parameter of the semantic anchor, to select regions with attention weights higher than the threshold as a candidate set of key visual regions. Non-maximum suppression is then applied to the selected candidate set of key visual regions to remove spatially overlapping and semantically redundant regions, retaining the most representative visual focus.

[0090] By using the semantic-spatial correlation calculation and dynamic filtering methods described above, the generalized region of interest obtained in the previous step is transformed into a candidate set of key visual regions that are highly consistent with the user's current interaction intent. This enables the visual computing resources to be accurately focused on regions with high semantic value, and significantly reduces the feature extraction overhead of invalid background regions.

[0091] S3.4: Based on the candidate set of key visual regions, a spatial weight distribution matrix is ​​constructed using a dynamic Gaussian kernel function, and a binarization threshold truncation process is performed on the spatial weight distribution matrix to generate the original binarized attention mask template that limits the image encoder to calculate the focus.

[0092] The system receives a key visual region candidate set output from the preceding step S3.3. This candidate set contains the center coordinates and initial bounding box sizes of multiple regions of interest (ROIs) highly relevant to the current semantic anchor intent. For each key visual region candidate, its normalized spatial distance relative to the center of the current micro-time window image frame is calculated, constructing a local coordinate system with the ROI center as the origin. A dynamic Gaussian kernel model is employed, dynamically adjusting the standard deviation of the Gaussian distribution based on the confidence parameter in the semantic anchor. Higher confidence results in a smaller standard deviation, thus forming a more focused spatial weight distribution. Specifically, for any pixel (x, y) in the image frame, its corresponding spatial weight value W(x, y) is calculated using the following formula: in, This represents the spatial weight value of the pixel (x, y). and These are the horizontal and vertical coordinates of the center of the key visual region candidate, respectively. The Gaussian kernel standard deviation is dynamically adjusted, and its value is determined by the semantic anchor confidence level. The mapping is obtained, and the mapping relationship is as follows: , The base diffusion radius constant is used. By traversing all pixels of the image frame or using a sparse sampling strategy, a continuous spatial weight distribution matrix covering the entire image is generated. This matrix exhibits a peak at the center of the key region and smoothly decays towards the surrounding areas.

[0093] The generated continuous spatial weight distribution matrix is ​​subjected to binarization threshold truncation, and a global activation threshold is set. When the weight value of a certain pixel is greater than or equal to When the weight value is less than 1, the position is marked as 1, indicating that it is included in the high-precision calculation focus of the image encoder; when the weight value is less than 1, the position is marked as 1. When this occurs, the position is marked as 0, indicating that deep feature extraction in that region is suppressed. Through the above dynamic Gaussian weighting and binarization truncation processing method, the key visual region candidate set in the previous step is transformed into the original binarized attention mask template that limits the image encoder's computation focus. This achieves precise targeted allocation of high-latency modality computing resources and significantly reduces the computational power consumption of invalid background regions.

[0094] S3.5: Based on the resolution parameters of the original binarized attention mask template and the subset of micro-time windows image frames, perform multi-scale morphological smoothing optimization to output the final attention mask used to constrain the image encoder to perform fine-grained visual modeling only in key regions.

[0095] The system receives the original binary attention mask template generated in the preceding step S3.4 and the resolution parameters of the micro-temporal window image frame subset. The original binary attention mask template is a discrete matrix containing 0 and 1 values, where 1 represents a key visual region and 0 represents the background or non-key region. The resolution parameters include the image width W and height H. Morphological closing operations are performed on the original binary attention mask template. Structural elements are used to fill holes in the mask and smooth boundary spikes, eliminating isolated noise points and broken regions caused by threshold truncation, generating a preliminarily smoothed binary mask matrix. A multi-scale Gaussian pyramid is constructed based on the resolution parameters of the micro-temporal window image frame subset. The preliminarily smoothed binary mask matrix is ​​sampled to different resolution levels, forming a multi-level mask representation sequence from coarse-grained to fine-grained, to capture the semantic region distribution features at different spatial scales.

[0096] At each resolution level, a distance transformation operation is performed on the multi-level mask representation sequence. The Euclidean distance from each background pixel to the nearest foreground pixel is calculated, generating a continuous distance field mapping matrix. The gradient information of the distance field characterizes the edge transition characteristics of key regions. An adaptive soft thresholding function is used to perform nonlinear mapping processing on the distance field mapping matrix, transforming the discrete binary boundary into a continuous attention weight distribution with gradually decreasing weights. This results in pixels closer to the center of the key region receiving higher weights, while the weights of background pixels farther from the center gradually decay to zero, generating a multi-scale soft attention weight map. The multi-scale soft attention weight maps of each resolution level are upsampled to the original image resolution, and a weighted average fusion strategy is used to integrate the weight maps of each level, preserving high-frequency detail information and low-frequency global structural information. The final output is an attention mask used to constrain the image encoder to perform fine-grained visual modeling only in key regions.

[0097] By employing a multi-scale morphological smoothing optimization process, the discrete binary mask from the previous step is transformed into soft attention weight distribution data with spatial continuity and multi-scale adaptability. This achieves the expected technical effect of accurately softening the focus of the image encoder, avoiding feature extraction truncation errors caused by hard boundaries, and improving the robustness of cross-modal semantic alignment.

[0098] For example, the resolution of a subset of images within a micro-time window is set to 1920×1080 pixels. The key regions in the original binarized attention mask template have a value of 1, while the background regions have a value of 0. A 3×3 rectangular structuring element is selected to perform a morphological closing operation on the original binarized attention mask template, filling internal holes smaller than 3 pixels and connecting break boundaries to generate a preliminarily smoothed binary mask matrix. A multi-scale Gaussian pyramid with four levels is constructed, with level resolutions of 1920×1080, 960×540, 480×270, and 240×135. The preliminarily smoothed binary mask matrix is ​​downsampled sequentially to each level. At the 240×135 resolution level, a distance transformation is performed on the mask, calculating the Euclidean distance from each background pixel to the nearest foreground pixel, generating a distance field mapping matrix D, where the distance values ​​range from 0 to 50 pixels. An adaptive soft thresholding function is then used. A nonlinear mapping is applied to the distance field mapping matrix, setting the slope parameter k to 0.2 and the center offset d0 to 10. This transforms the distance values ​​into continuous weight values ​​between 0 and 1, generating a multi-scale soft attention weight map for this level. The masks for other levels are processed similarly and upsampled to 1920×1080 resolution. A weighted average is then performed using layer weight coefficients of 0.4, 0.3, 0.2, and 0.1 to obtain the final attention mask. This mask has a weight close to 1.0 at the center of key regions, exhibits a smooth transition at edges, and a weight close to 0.0 in the background. Applying this mask to the input of the convolutional layer of the image encoder allows the encoder to activate deep neurons only in high-weight regions and suppress activation in low-weight regions, significantly reducing unnecessary computation while maintaining the integrity of key semantic features. This greatly improves the alignment accuracy between subsequent local visual representation vectors and semantic anchor points.

[0099] like Figure 3 As shown, step S4: Based on the attention mask constraint on the computational focus of the image encoder, the local visual representation of the high-delay modality is divided into multiple micro-time windows to output a set of local visual representations corresponding to each micro-time window. Specifically, this includes: S4.1: Obtain the attention mask and the complete image frame sequence. Perform key region locking processing on the complete image frame sequence based on the temporal interval information in the mask to generate a key region image subset containing spatial coordinate constraints.

[0100] The system receives the attention mask matrix output from the preceding step S3 and the complete original image frame sequence cached by the cloud-based intelligent processing module. The attention mask matrix contains a binarized spatial weight distribution and corresponding temporal interval identifiers. The complete original image frame sequence is an uncompressed or lightweight decoded RGB pixel data stream. The system parses the temporal metadata in the attention mask, extracts the start timestamp t_start and end timestamp t_end corresponding to the current semantic anchor point to be processed, and constructs a temporal filtering window. It iterates through the frame header timestamp information of each frame in the complete original image frame sequence, marking image frames whose frame timestamps fall within the closed interval [t_start, t_end] as candidate keyframes, and removing redundant background frames or invalid transition frames outside the interval to generate a temporally aligned subset of candidate image frames.

[0101] The spatial resolution parameter W×H of each frame in the candidate image frame subset is read, and the attention mask is adjusted to the same spatial dimension as the candidate image frame to ensure a one-to-one correspondence at the pixel level. A pixel-wise Hadamard product operation is performed, and the adjusted binarized attention mask is multiplied element-wise with the pixel matrix of the candidate image frame. For regions with a mask value of 0, their pixel values ​​are forced to zero or a very low-weight background noise baseline is retained. For regions with a mask value of 1, the original pixel intensity remains unchanged, thus achieving signal suppression in non-critical regions in the spatial domain. A connected component analysis algorithm is then performed on the masked image frames to identify all continuous regions composed of non-zero pixels, and the coordinates (x_min, y_min, x_max, y_max) of the minimum bounding box of each connected component are calculated.

[0102] Connected regions with an area greater than a preset threshold Th_area and an aspect ratio that conforms to the geometric features of the human body or interactive object are selected to exclude isolated noise points or minor artifacts caused by noise, thus determining the final effective visual attention region. The bounding box coordinates of the effective visual attention region are encapsulated as spatial constraint metadata and bound to the corresponding image pixel data to generate a subset of key region images containing precise spatial coordinate constraints.

[0103] By employing the dual locking mechanism based on temporal filtering and spatial masking constraints, the global image data generated in the previous step is transformed into a subset of key region images containing only highly semantically relevant local areas. This enables the precise focus of computing resources on high-value visual information, significantly reducing the computational overhead in the subsequent feature extraction stage and improving the signal-to-noise ratio.

[0104] S4.2: Receive a subset of key region images and use a shallow feature caching mechanism to perform low-power feature suppression operations on non-key regions to generate a sparse visual feature stream focused on high-confidence regions.

[0105] The system receives a subset of key region images containing spatial coordinate constraints and the corresponding attention mask from the output of the preceding step S4.1, and inputs this subset of key region images into a lightweight shallow feature extraction network. The first three convolutional layers of the pre-trained MobileNetV3 backbone network are used to perform low-order visual feature extraction on the key region image subset, generating a primary feature map containing edge texture, color distribution, and basic geometric shape information. Based on the binarized weight distribution in the attention mask, a spatial gating mechanism is constructed to force the activation values ​​of the primary feature map channels corresponding to non-key regions with a value of 0 in the mask to zero, thereby achieving feature suppression of non-key regions.

[0106] Global average pooling along the channel dimension is performed on the primary feature map after spatial gating to calculate the energy response value of each feature channel, generating a channel attention vector reflecting local visual saliency. A sparse regularization term is introduced to constrain the channel attention vector with the L1 norm. Redundant channels with energy responses below a preset noise threshold are removed using a soft threshold function, further compressing the feature dimension and eliminating background noise interference. The sparsified channel attention vector is broadcast back to the original feature map space and subjected to element-wise multiplication with the primary feature map to generate a sparse visual feature stream with high-confidence region focusing characteristics.

[0107] By using the aforementioned shallow feature caching and spatial-channel dual suppression processing method, the key region image subset from the previous step is transformed into sparse visual feature stream data with high signal-to-noise ratio and low computational load. This achieves low-power filtering of non-critical background information and accurate preservation of core semantic region features, providing high-quality and compact feature input for subsequent micro-time window segmentation.

[0108] S4.3: Based on the timestamps of the sparse visual feature stream and semantic anchor sequence, a sliding window segmentation algorithm is used to perform micro-time window partitioning on the sparse visual feature stream to generate a set of micro-time window feature segments with independent temporal identifiers.

[0109] The system receives a sparse visual feature stream focused on high-confidence regions from the output of previous step S4.2. This data stream contains local image patch feature vectors filtered by spatial masking and their corresponding original acquisition timestamp indices. The timestamp information associated with each feature vector in the sparse visual feature stream is parsed, and its absolute temporal coordinate values ​​are extracted to construct a continuous feature sequence index table based on the time dimension. Based on the micro-time window reference duration parameter defined in the semantic anchor sequence, the step size and window width of the sliding window are set. The window width corresponds to the length of the temporal interval covered by a single semantic anchor, and the step size is set to half the window width to achieve overlapping sampling and enhance temporal continuity.

[0110] A linear interpolation algorithm is employed to fill in the temporal gaps in the sparse visual feature stream caused by suppression of non-critical regions, ensuring the uniform distribution and integrity of the feature sequence along the time axis, and generating a completed dense visual feature time series. Based on the set sliding window parameters, a frame-by-frame sliding segmentation operation is performed on the dense visual feature time series, truncating the continuous feature stream into multiple fixed-length subsequence segments, each corresponding to an independent micro-time window. A unique temporal identifier is assigned to each generated micro-time window feature segment, which is composed of a start timestamp and an end timestamp and associated with the corresponding semantic anchor ID, forming a set of micro-time window feature segments with independent temporal identifiers.

[0111] By using the above-mentioned sliding window segmentation and temporal identifier binding processing method, the sparse and discontinuous visual feature stream obtained in the previous step is transformed into a structured, temporally aligned set of micro-temporal window feature fragments, realizing the accurate mapping of high-latency visual modalities and low-latency semantic anchors on a fine-grained temporal scale, and providing standardized input units for subsequent parallel fine-grained visual modeling.

[0112] S4.4: For each micro-time window feature segment in the micro-time window feature segment set, call the fine-grained visual modeling engine to perform deep feature extraction operations to generate the local visual representation vector of the corresponding micro-time window.

[0113] For each micro-time window feature segment in the micro-time window feature segment set, a fine-grained visual modeling engine is invoked to perform deep feature extraction operations. The system receives micro-time window feature segments with independent temporal identifiers generated in previous steps, along with their corresponding attention masks. These micro-time window feature segments are then input into a pre-loaded lightweight convolutional neural network backbone layer. Element-wise multiplication of the input feature map is performed using the spatial weight distribution in the mask to suppress activation responses in non-critical background regions while preserving feature intensity in high-confidence foreground regions.

[0114] Based on the weighted feature map, a residual connection module is initiated to perform multi-scale feature fusion processing. Local texture details and global structural semantics are extracted through parallel branches, and the weight coefficients of each branch feature are dynamically adjusted using a channel attention mechanism to enhance the representation ability of visual elements related to interactive intent. An adaptive pooling layer is used to compress the spatial dimension of the fused multi-dimensional feature map, converting the irregularly sized two-dimensional feature map into a fixed-length one-dimensional feature vector sequence, ensuring that the output dimension is consistent with the input interface of the subsequent semantic alignment module. A temporal position encoding algorithm is introduced to generate sine and cosine position embedding vectors based on the relative temporal position of the micro-time window in the complete interaction sequence. This position embedding vector is then linearly superimposed with the compressed one-dimensional feature vector to endow static visual features with dynamic temporal context information. A fully connected projection layer maps the superimposed feature vector to a high-dimensional latent semantic space, and the ReLU nonlinear activation function is used to sparsify the mapping result to eliminate redundant noise and highlight core semantic components.

[0115] Through the chain-like derivation process described above, the micro-time window image data from the previous step is transformed into a local visual representation vector carrying spatiotemporal context information, enabling fine-grained semantic capture of high-latency modalities with low computational overhead, and providing high-precision visual primitives for subsequent cross-modal contrastive learning.

[0116] For example, in a smart meeting scenario, when a semantic anchor indicates that the user is displaying a "project progress chart," the micro-time window duration is set to 200ms, corresponding to 5 frames of 1080P video images. The fine-grained visual modeling engine uses an improved MobileNetV3 as the backbone network, with the input image resolution cropped to 224x224 pixels. In the attention mask, the weight of the chart region is set to 1.0, and the weight of the background region is set to 0.1. After element-wise multiplication, the background features are suppressed by 90%. In the residual module, the local texture branch extracts the edge features of the chart lines, the global structure branch extracts the overall layout features of the chart, and the channel attention mechanism assigns a weight of 0.7 to the local texture branch and a weight of 0.3 to the global structure branch. The adaptive pooling layer compresses the 7x7x512 feature map into a 1x1x512 vector. Temporal position encoding uses a sine wave with a frequency of 10000 to generate a 512-dimensional position embedding for the third micro-time window. The fully connected projection layer maps the 512-dimensional vector to a 768-dimensional latent space. The final output local visual representation vector has a length of 768. Its Euclidean distance to the semantic embedding of "chart" is significantly smaller than its distance to the semantic embedding of "person" or "whiteboard", which verifies the semantic directionality and accuracy of feature extraction.

[0117] S4.5: Summarize the local visual representation vectors of all micro-time windows, and perform structured recombination according to the occurrence time interval to output a set of local visual representations that strictly correspond to each micro-time window.

[0118] Step S5: Input the intent label sequence into the distillation buffer, and perform online comparative learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory. Specifically, this includes: S5.1: Obtain the intent slot field and core entity field from the intent tag sequence, and use the domain-adaptive word vector mapping algorithm to perform high-dimensional semantic space projection processing on the intent slot field and core entity field to generate a set of teacher semantic embedding vectors with temporal alignment attributes.

[0119] It should be noted that the distillation buffer is a memory cache area used to temporarily store the teacher's semantic embedding vector set and establish a timestamp-based fast retrieval index to achieve millisecond-level pairing of teacher signals and student features during the contrastive learning process. "Distillation" refers to the process of using the teacher's semantic embedding vector as a supervision signal to guide the convergence of student visual features to the teacher's semantic space through online contrastive learning; when the semantic consistency index reaches a preset threshold, it is called the "distilled state".

[0120] Extract the temporal index of the current processing cycle from the micro-time window local visual representation set output by step S4, and simultaneously read the intent slot field and core entity field corresponding to the timestamp interval in the intent tag sequence generated by step S1. Perform lexical standardization on the extracted intent slot field, remove stop words and unify verb tenses, and map the core entity field to the pre-built domain knowledge graph node ID to form a standardized semantic token sequence. Load the domain-adaptive word vector mapping model trained offline, which is fine-tuned based on a large-scale office meeting corpus and includes a static word embedding layer and a dynamic context encoding layer. Input the standardized semantic token sequence into the static word embedding layer, find the initial low-dimensional dense vector representation corresponding to each token, and generate the basic semantic feature matrix.

[0121] The basic semantic feature matrix is ​​input into the dynamic context encoding layer, and a bidirectional long short-term memory network is used to capture the dependencies between tokens, calculating the context-aware weight of each token in a specific context. The basic semantic feature matrix is ​​then weighted and fused based on the context-aware weights to generate an intermediate-layer semantic representation vector with temporal semantic coherence. A time-position encoding mechanism is introduced, generating sine and cosine position encoding vectors based on the absolute timestamps and relative temporal intervals in the intent tag sequence.

[0122] The intermediate layer semantic representation vector and the temporal location encoding vector are added element-wise to inject temporal dimension information and eliminate semantic ambiguity. A multi-layer Transformer encoder is then used to perform self-attention on the vectors with injected temporal information to capture long-distance semantic dependencies, outputting a preliminary high-dimensional semantic embedding result. Layer normalization is applied to the preliminary high-dimensional semantic embedding result to stabilize the feature distribution and prevent gradient vanishing or exploding. A fully connected projection layer is then used to map the normalized features to a predefined high-dimensional semantic space. This space maintains the same dimension as the dimension required for subsequent visual feature alignment, generating a set of teacher semantic embedding vectors with temporal alignment properties.

[0123] By using a domain-adaptive word vector mapping algorithm and context-aware weighted processing, discrete structured intent labels are transformed into continuous high-dimensional semantic vectors, enabling the comparability of semantic information and visual features in a unified latent space, and providing accurate teacher signals for subsequent cross-modal contrastive learning.

[0124] S5.2: Receive the teacher semantic embedding vector set and inject it into the write port of the distillation buffer. At the same time, obtain the micro-time window local visual representation vector in the local visual representation set. Based on the timestamp matching rule, pair and associate the micro-time window local visual representation vector with the teacher semantic embedding vector set to generate modality contrast learning sample pairs to be optimized.

[0125] The system receives the teacher semantic embedding vector set with temporal alignment attributes generated in step S5.1 and obtains the local visual representation set strictly corresponding to each micro-time window from the output of step S4. The teacher semantic embedding vector set is loaded into the write port of the distillation buffer, and a fast retrieval table based on timestamp indexes is established to ensure that the embedding vector corresponding to each semantic anchor can be located in milliseconds. The local visual representation vectors of each micro-time window in the local visual representation set are read in parallel, and the independent temporal identifier carried by each vector is extracted.

[0126] The timestamp matching rules are executed to calculate the overlap between the start timestamp of the local visual representation vector of the micro-time window and the occurrence time intervals associated with each embedding vector in the teacher semantic embedding vector set. When the overlap is greater than a preset time synchronization threshold, it is determined that the local visual representation vector of the micro-time window and the corresponding teacher semantic embedding vector have a semantic association. The local visual representation vector of the micro-time window with semantic association is used as the student input, and the corresponding teacher semantic embedding vector is used as the teacher target to construct modal contrast learning sample pairs. The validity of the constructed modal contrast learning sample pairs is verified, and invalid pairs with misaligned timestamps or confidence scores below the threshold are removed to ensure the data quality entering the subsequent contrast learning process.

[0127] By using the pairing and association processing method based on timestamp matching rules, the discrete multimodal features generated in the previous step are transformed into structured modality contrastive learning sample pairs to be optimized, achieving accurate alignment of cross-modal semantics in the temporal dimension, and providing a high signal-to-noise ratio training data foundation for subsequent online contrastive learning and gradient update.

[0128] S5.3: Construct a cross-modal contrastive loss function based on modal contrastive learning sample pairs, use a temperature scaling factor to perform probability distribution smoothing on the teacher semantic embedding vector set to generate a soft-label target distribution, and calculate the prediction probability of the local visual representation vector under the soft-label target distribution in the micro-time window to output the initial backpropagation signal containing the semantic consistency error gradient.

[0129] Based on the modal contrast learning sample pairs generated in step S5.2, an optimization objective function for cross-modal semantic alignment is constructed. The teacher semantic embedding vector set is defined as the baseline distribution in a high-dimensional space, and the student local visual representation vector is the distribution to be optimized. A temperature scaling factor τ is introduced to smooth the probability distribution of the teacher semantic embedding vectors, softening the label distribution and enhancing the model's learning ability for difficult examples. The dot product similarity between the teacher semantic embedding vector and all candidate local visual representation vectors is calculated as the input to logits. The Softmax function is applied to convert logits into a normalized probability distribution, generating the soft-label target distribution. Specifically, for the local visual representation vector v_i of the i-th micro-time window and its corresponding positive sample teacher semantic embedding vector t_i, as well as the other negative sample teacher semantic embedding vectors t_j (j≠i) within the batch, the relative probability of the positive sample is calculated. The probability value P_i of the i-th sample in the soft-label target distribution is calculated using the following formula: in, This is the local visual representation vector of the current micro-time window. For positive sample teacher semantic embedding vectors, For negative sample teacher semantic embedding vectors, This is the temperature scaling factor. The batch size is used. This step controls the smoothness of the distribution by adjusting the τ value. A larger τ value makes the probability distribution more uniform, helping the model focus on the global structure rather than just fitting hard labels. Based on the generated soft-label target distribution, the predicted probability of the local visual representation vector under the distribution within the micro-time window is calculated. A cross-entropy loss function is constructed as a cross-modal contrastive loss function to measure the difference between the predicted probability distribution and the true label distribution (usually set as a unit vector or a confidence-weighted vector). The gradient of the loss function with respect to the local visual representation vector and its feature extraction network parameters is calculated. Specifically, the cross-entropy loss L is calculated as follows: in, For the first The true label probability value of each sample (usually 1 / N or confidence-weighted in self-supervised contrastive learning). This represents the probability value predicted by the model. Backpropagation is performed on the loss function L to solve for the gradient. L / v. This gradient reflects the direction and magnitude of the consistency error between the current visual representation and the teacher's semantic embedding in the semantic space. The gradient is passed to the underlying parameters of the visual feature extraction network using a chain rule. The output contains the initial backpropagation signal containing the semantic consistency error gradient, which directly points to the direction of how to adjust the visual features to better match the teacher's semantic anchors.

[0130] By constructing the cross-modal contrastive loss function and processing the gradient calculation as described above, the modal contrastive learning sample pairs from the previous step are transformed into an initial backpropagation signal containing the semantic consistency error gradient. This achieves refined semantic guidance and alignment optimization of high-latency visual modal features, providing accurate error feedback for subsequent gradient truncation and parameter updates.

[0131] S5.4: Perform norm threshold detection on the initial backpropagation signal. When the signal amplitude exceeds the preset gradient explosion suppression threshold, perform amplitude truncation using the gradient clipping algorithm. When the signal amplitude does not exceed the preset gradient explosion suppression threshold, keep the original value unchanged to generate a truncated gradient update amount optimized for stability.

[0132] The system receives the initial backpropagation signal, output from previous step S5.3, which includes the semantic consistency error gradient. This signal characterizes the direction and magnitude of the deviation between the local visual representation vector within the micro-time window and the teacher's semantic embedding vector during cross-modal contrastive learning. An L2 norm calculation operation is performed on the initial backpropagation signal. By iterating through each gradient component in the signal vector, the arithmetic square root of the sum of squares of each component is calculated to quantify the overall amplitude of the current backpropagation signal. The L2 norm value of the initial backpropagation signal is calculated using the following formula: Where G is the L2 norm value of the initial backpropagation signal, n is the total number of dimensions of the gradient vector, and gi is the gradient component value of the i-th dimension. The calculated L2 norm value G is compared with a preset gradient explosion suppression threshold Tgrad. This threshold Tgrad is preset based on the numerical stability boundary and floating-point precision limit of the image encoder in the cloud intelligent processing module, and is used to define the safe range for gradient updates. When it is determined that the L2 norm value G is greater than the preset gradient explosion suppression threshold Tgrad, the gradient clipping logic is triggered, and the scaling factor α is calculated. This factor is defined as the ratio of the threshold Tgrad to the current norm value G, i.e., α = Tgrad / G, ensuring that the scaled gradient norm is strictly equal to the threshold Tgrad.

[0133] Each gradient component gi in the initial backpropagation signal is linearly scaled using a scaling factor α to generate a truncated gradient component gi', which is calculated using the following formula: in, The first after amplitude truncation One gradient component, The scaling factor is calculated based on the threshold constraint. For the first The gradient components are defined in each dimension. When the L2 norm value G is less than or equal to the preset gradient explosion suppression threshold Tgrad, the current gradient is considered to be in a stable convergence interval. The original value of the initial backpropagation signal is directly retained as the truncated gradient update amount without any scaling or modification operations. All gradient components processed by the conditional branch are summarized and reconstructed into a truncated gradient update vector with the same dimensional structure. This vector eliminates the risk of abnormal gradient fluctuations caused by huge differences in multimodal feature spaces or the randomness of parameters in the early stage of training.

[0134] By employing the norm threshold detection and adaptive amplitude truncation methods described above, the potentially unstable backpropagation signal generated in the previous step is transformed into a truncated gradient update amount optimized for stability. This prevents gradient explosion during the neural network parameter update process and ensures the expected technical effects of numerical convergence and training robustness of online contrastive learning under the flow distillation architecture.

[0135] S5.5: The feature extraction network parameters of the local visual representation vector of the micro-time window are iteratively updated by using the truncated gradient update amount. The updated feature extraction network is then applied to the current micro-time window image frame subset to reconstruct the visual feature representation. The reconstructed visual feature representations of all micro-time windows are accumulated in chronological order to generate the final visual feature trajectory.

[0136] The algorithm receives the truncated gradient update amount, optimized for stability, and the subset of image frames corresponding to the current micro-time window. It then maps the truncated gradient update amount to the weight parameter space of the corresponding convolutional or Transformer layer in the fine-grained visual modeling engine. Based on the backpropagation chain rule, it uses the truncated gradient update amount to perform local parameter fine-tuning on the weight matrices of the last fully connected layer and the penultimate attention mechanism layer in the feature extraction network, maintaining the parameter freeze state of the underlying general visual feature extractor to preserve its basic representational capabilities.

[0137] The fine-grained visual modeling engine, after parameter fine-tuning, is reapplied to the current micro-time window image frame subset, generating a reconstructed visual feature representation with higher semantic consistency through forward propagation. This reconstructed visual feature representation is obtained and its high-dimensional vector form is extracted, marking it as the convergent visual feature point of the current time step. The convergent visual feature points of the current time step are appended to the end of the global visual feature trajectory queue according to the chronological order of their occurrence. This process of parameter fine-tuning, forward reconstruction, and trajectory appending is repeated across all unprocessed micro-time windows until the complete temporal interval corresponding to the entire semantic anchor point is covered.

[0138] By using iterative weight updates and re-action mechanisms, discrete local visual representations are transformed into visual feature trajectories with temporal continuity and semantic convergence, achieving progressive alignment of high-latency visual modalities to low-latency semantic anchors. This significantly reduces computational oscillations in cross-modal fusion and improves the real-time response accuracy of semantic understanding.

[0139] For example, in a smart meeting scenario, for a 2-second video stream of user gesture commands, the system divides it into four micro-time windows of 500ms each. For the first micro-time window, the initial truncated gradient update output from step S5.4 is a 768-dimensional vector with an L2 norm of 0.05, which is lower than the preset gradient explosion suppression threshold of 0.1, and therefore directly used as the effective update. This update is applied to the Query and Key projection matrices of the last two TransformerBlock layers of the visual encoder, with a learning rate set to 1e-4. The updated encoder performs forward inference on the key region image (including the user's raised hand) of the first micro-time window, outputting a reconstructed visual feature vector V1, whose cosine similarity with the teacher's semantic embedding vector (intent label: 'Start Presentation') increases from the initial 0.65 to 0.82. Subsequently, V1 is stored in the visual feature trajectory queue. For the second micro-time window, due to the change in gesture (palm pushing forward), the L2 norm of the initial gradient update surged to 0.15, exceeding the threshold of 0.1. The system performed gradient pruning, scaling the update to an L2 norm of 0.1 before applying it to the network parameters. The cosine similarity between the reconstructed visual feature vector V2 and the teacher's semantic embedding improved from 0.58 to 0.79. The third and fourth micro-time windows were processed sequentially, ultimately generating a visual feature trajectory of length 4 [V1, V2, V3, V4]. This trajectory smoothly transitioned temporally, accurately reflecting the gesture evolution process under the 'start demonstration' intent. Compared to the original feature sequence without distillation optimization, its confidence score in the downstream intent classification task significantly improved from 0.72 to 0.91, and the first-word response latency remained within 200ms, meeting the requirements of real-time interaction.

[0140] Step S6: Calculate the semantic consistency index based on the visual feature trajectory and intent label sequence, determine whether the index reaches the preset alignment verification threshold, and generate an anchor point fusion result marked as distilled or an incremental update request marked as incomplete. Specifically, this includes: S6.1: Obtain the local visual representation set in the visual feature trajectory and the corresponding semantic embedding vector in the intent label sequence. Use the cross-modal projection mapping algorithm to map the local visual representation set to the feature space where the semantic embedding vector is located, so as to generate modal aligned feature pairs with spatial comparability.

[0141] S6.2: Based on the modal alignment feature pair, perform cosine similarity measurement operation to calculate the directional consistency value between visual features and semantic embedding within each micro-time window, so as to output an instantaneous similarity score sequence reflecting the degree of modal matching at a single moment.

[0142] Obtain spatially comparable modal alignment feature pairs generated in the preceding step S6.1. Each modal alignment feature pair contains a micro-temporal window local visual representation vector mapped to the same semantic feature space and its corresponding teacher semantic embedding vector. Perform L2 norm normalization on the micro-temporal window local visual representation vector and the teacher semantic embedding vector in the modal alignment feature pair to eliminate the interference of vector magnitude differences on the directional consistency measure, generating a standardized visual feature vector and a standardized semantic embedding vector of unit length. Based on the standardized visual feature vector and the standardized semantic embedding vector, calculate the cosine of the angle between them in the multi-dimensional feature space using dot product operation to quantify the directional matching degree between the visual information and semantic intent at a single moment. Calculate the instantaneous similarity score using the following formula: in, For instantaneous similarity score, for Local visual representation vector at time step, for The teacher's semantic embedding vector at time step. This indicates the transpose operation. Let L2 norm be denoted. Since normalization was performed in the previous step, the denominator is always 1, and the actual calculation simplifies to the dot product of two vectors. Numerical range constraints are applied to the calculated instantaneous similarity scores, limiting them to the interval [-1, 1] to ensure the physical meaning of the measurement results is clear, where 1 represents completely in the same direction, -1 represents completely in opposite directions, and 0 represents orthogonal and unrelated. The instantaneous similarity scores corresponding to all micro-time windows are sequentially assembled to construct an instantaneous similarity score sequence reflecting the dynamic changes in modal matching at each moment during the interaction process.

[0143] By using cosine similarity measurement and serialization assembly, the modal alignment feature pairs from the previous step are transformed into an instantaneous similarity score sequence that reflects the degree of modal matching at a single moment. This enables a fine-grained, temporal quantitative evaluation of the quality of multimodal semantic alignment, providing basic data support for the subsequent calculation of the comprehensive consistency index.

[0144] S6.3: Apply sliding window weighted average filtering to the instantaneous similarity score sequence, combine it with time decay factor to eliminate short-term noise interference and accumulate temporal context information, so as to generate a comprehensive semantic consistency index that characterizes the overall semantic evolution stability.

[0145] The system receives the instantaneous similarity score sequence output from the preceding step S6.2. This sequence records the directional consistency values ​​between visual features and semantic embeddings within each micro-time window. A sliding time window of length L is constructed, incorporating the instantaneous similarity scores of the current time t and its L-1 preceding times into the calculation, forming a local temporal data subset. Each instantaneous similarity score within the sliding window is assigned a weight coefficient that decays over time, with scores closer to the current time having higher weights and scores farther away having lower weights, to reflect the dominant role of recent semantic state in the overall judgment.

[0146] The weight values ​​at each time step are calculated using an exponential decay function to ensure that the sum of weights is normalized, thereby eliminating the influence of dimensional differences on the filtering results. Specifically, for the k-th historical time step within the sliding window, its corresponding weight coefficient is calculated using the following formula: in, This represents the weight coefficient for the k-th historical moment. This is a time decay factor used to control the rate at which historical information is forgotten. The length of the sliding window. and The time index within the window is used. The instantaneous similarity scores within the sliding window are weighted and summed using the calculated weight coefficients to generate a smoothed similarity estimate for the current time step. This weighted averaging process effectively suppresses drastic fluctuations in instantaneous similarity caused by image frame jitter, sudden changes in illumination, or short-term speech noise, preserving the long-term trend characteristics of semantic evolution.

[0147] As the interaction continues, the sliding window moves forward along the time axis, continuously updating the set of instantaneous similarity scores involved in the calculation, and recalculating the weighted average, thus forming a continuous and stable comprehensive semantic consistency index curve.

[0148] By combining sliding window weighted average filtering with a time decay factor, the discrete and noisy instantaneous similarity score sequence in the previous step is transformed into a comprehensive semantic consistency index that characterizes the overall semantic evolution stability. This achieves a robust assessment of multimodal alignment quality and provides a reliable basis for subsequent alignment verification threshold determination.

[0149] S6.4: Perform a logical comparison operation between the comprehensive semantic consistency index and the preset alignment verification threshold. If the index is greater than or equal to the threshold, the alignment condition is determined to be met. If the index is less than the threshold, the alignment condition is determined to be not met. Output a binary discrimination result containing the state label.

[0150] The system receives the comprehensive semantic consistency index value, representing the overall semantic evolution stability, output from the preceding step S6.3. This value is a floating-point scalar data after being corrected by a sliding window weighted average filter and a time decay factor. Simultaneously, it reads a preset alignment verification threshold parameter from the system configuration memory. This threshold parameter is set based on the lower limit of the statistical distribution of multimodal alignment accuracy in historical interaction scenarios and is used to define the acceptable confidence boundary of semantic understanding. A binary logic comparison unit is constructed, using the comprehensive semantic consistency index value as the comparison operand and the alignment verification threshold parameter as the reference operand. A Boolean logic operation with a greater than or equal to relation is performed to determine whether the current cross-modal semantic alignment state meets the accuracy requirements of downstream task execution.

[0151] When the overall semantic consistency index is greater than or equal to the alignment verification threshold parameter, the logic comparison unit outputs a high-level valid signal. This signal is mapped to a 'alignment condition met' status indicator, indicating that the projection distance between the current visual feature trajectory and the intent label sequence in the semantic space has converged to within the allowable error range, possessing sufficient semantic mutual information to support subsequent business logic processing. When the overall semantic consistency index is less than the alignment verification threshold parameter, the logic comparison unit outputs a low-level invalid signal. This signal is mapped to a 'alignment condition not met' status indicator, indicating that there is still a significant semantic gap or noise interference between the current visual representation and the semantic anchor point, requiring further accumulation of temporal context information or the introduction of more visual feature fragments for iterative optimization.

[0152] Based on the logical comparison results, a binary discrimination result data structure containing state tags is generated. This structure not only contains the state enumeration values ​​of 'distilled' or 'incomplete', but also encapsulates the specific timestamp index that triggered the judgment and the current similarity score snapshot, so that the decision basis can be traced and the corresponding data encapsulation or incremental request generation operation can be performed when performing branch routing control in subsequent step S6.5.

[0153] By using the above logical comparison and state mapping processing methods, the continuous comprehensive semantic consistency index is transformed into a discrete binary discrimination result, realizing real-time quantitative evaluation and process diversion control of multimodal semantic alignment quality, and ensuring that the system maximizes response speed while ensuring semantic understanding accuracy.

[0154] For example, in a smart meeting scenario, when a user issues the command "Please show the sales report for the previous quarter," the system needs to align the voice command with the chart image on the projection screen. The preset alignment verification threshold parameter is set to 0.85. This value is determined based on the statistical mean of voice-image semantic matching in a large number of meeting scenarios minus two standard deviations to ensure extremely high semantic accuracy. After processing in step S6.3, the comprehensive semantic consistency index value calculated for the micro-time window sequence corresponding to this command is 0.92. This value of 0.92 and the threshold of 0.85 are input into the logic comparison unit, and a judgment is made that 0.92 ≥ 0.85, with the result being true. The logic comparison unit outputs a high-level valid signal, generating a binary discrimination result with the status marker 'distilled', which includes the status code '1', the timestamp 'T=1520ms', and the current score '0.92'. This result directly triggers step S7.2, which encapsulates the current visual feature trajectory (containing high-dimensional feature vectors of key areas in the chart) and the intent label sequence (containing the action 'display', the entity 'sales report', and the time 'previous quarter') into a data packet and immediately sends it to the downstream rendering module, achieving millisecond-level instant response. In another scenario, if the user's gesture is blurry and the background is noisy, resulting in a comprehensive semantic consistency index of only 0.78, a judgment of 0.78 < 0.85 is executed, resulting in a false result. The logic comparison unit outputs a low-level invalid signal, generating a binary discrimination result marked 'incomplete', which includes the status code '0', the timestamp 'T=1520ms', and the current score '0.78'. This result triggers step S7.4. The system does not block the main thread but instead generates an incremental update request carrying information about the remaining time interval, continuing to receive the local visual representation set of the next micro-time window for iterative alignment until the comprehensive semantic consistency index at subsequent moments improves to above 0.85. This ensures the accuracy of the final understanding while avoiding overall interaction lag caused by waiting for perfect alignment of a single frame.

[0155] S6.5: Execute branch routing control based on the binarization discrimination result. When the result meets the alignment condition, encapsulate the current visual feature trajectory and intent label sequence as the anchor point fusion result in the distilled state. When the result does not meet the alignment condition, generate an incremental update request for the incomplete state carrying the remaining time interval information.

[0156] Step S7: If it is determined that a distilled anchor point fusion result has been generated, the distilled anchor point fusion result is sent to the downstream task to perform non-blocking semantic evolution. If it is determined that an incomplete incremental update request has been generated, new local visual representation sets are continuously received for iterative alignment. Specifically, this includes: S7.1: Obtain the semantic consistency index value generated by the previous steps and the preset alignment verification threshold parameter. Use the comparison logic circuit to compare the semantic consistency index value and the alignment verification threshold parameter to generate a binary judgment signal containing a compliance status indicator or a non-compliance status indicator.

[0157] It should be noted that downstream tasks include, but are not limited to, at least one of the following: natural language understanding, dialogue management, visualization rendering, speech synthesis, and action execution. Non-blocking semantic evolution refers to the immediate transmission of the fusion result to downstream tasks once a semantic anchor has completed multimodal alignment verification (i.e., reached the distilled state), without waiting for the processing results of other incomplete anchors or subsequent micro-time windows, thereby achieving a streaming response where some results are available first.

[0158] S7.2: Based on the compliance status identifier in the binary judgment signal, extract the visual feature trajectory and intent label sequence corresponding to the current moment, and use the data encapsulation protocol to combine the visual feature trajectory and intent label sequence to construct a distilled state anchor point fusion result data packet.

[0159] The system receives a binary judgment signal containing a compliance status identifier generated in the preceding step S7.1, parses the signal to confirm that the current semantic anchor has passed multimodal alignment verification, and triggers the data encapsulation process. It extracts a set of local visual representations corresponding to the current compliance anchor from the visual feature trajectory storage area. This set contains a sequence of multidimensional visual feature vectors optimized through micro-time window partitioning and online contrastive learning. It retrieves the intent slot field, core entity field, and confidence value that strictly match the timestamp interval of the anchor from the intent label sequence cache, ensuring the consistency of the semantic context. It invokes the domain adaptive serialization protocol to convert the extracted set of local visual representations into a binary stream format, and adds feature dimension identifiers and temporal synchronization markers to the header, forming a standardized visual payload.

[0160] The retrieved intent slots, core entities, and confidence scores are recombined into key-value pairs according to a predefined JSON-LD semantic framework to generate machine-readable text semantic payloads. A timestamp alignment verification mechanism is introduced to compare the start and end timestamps of the visual payload with those of the text semantic payload. If the deviation exceeds a millisecond-level tolerance threshold, interpolation compensation is performed to ensure accurate overlap of multimodal data in the temporal dimension. A data packet header structure is constructed, including a version number, a unique anchor identifier, a payload type indicator, and an integrity check code. The integrity check code is calculated using a cyclic redundancy check algorithm to ensure bit-level accuracy during data transmission.

[0161] The standardized visual payload, textual semantic payload, and data packet header structure are concatenated in memory according to big-endian byte order to form a continuous byte stream buffer. Lightweight compression encoding is performed on the byte stream buffer to remove redundant padding bits and duplicate metadata, reducing data packet size and lowering network transmission overhead. An encrypted signature is appended to the compressed byte stream; the data packet content is hashed using the session key to generate a digital signature, preventing data tampering or eavesdropping during transmission. The encrypted signature is appended to the end of the data packet, completing the final data packet assembly and generating a fully structured, time-aligned, and secure distilled anchor fusion result data packet.

[0162] Through the above data encapsulation and serialization processing, the multimodal features and semantic tags stored in the previous step are transformed into a standard communication data packet with a compact structure and time synchronization, enabling downstream task modules to perform real-time parsing and non-blocking calls to high-confidence semantic information, and significantly reducing end-to-end interaction latency.

[0163] S7.3: For the anchor point fusion result data packet in the distilled state, perform a non-blocking routing forwarding operation. The distilled anchor point fusion result data packet is directly sent to the input buffer of the downstream task module through an independent control channel to trigger the immediate execution of subsequent business logic and release the current computing resources.

[0164] The system receives the distilled anchor fusion result data packet constructed in the preceding step S7.2. This data packet contains visual feature trajectories, intent label sequences, and metadata header information. Protocol decapsulation processing is performed on the anchor fusion result data packet to extract the payload field and routing control identifier, separating the semantic fusion entity data to be distributed and the target downstream module address index. Based on the target downstream module address index, the system queries the dynamic service registry maintained internally by the cloud-based intelligent processing module to obtain the current load status and available input buffer capacity of the corresponding downstream task module (such as a natural language understanding engine, dialogue management module, or visualization rendering engine).

[0165] A logical comparison is performed between the available capacity of the input buffer and a preset backpressure threshold. If the available capacity is higher than the backpressure threshold, it is determined to be a low-congestion state, and a direct-path routing instruction is generated directly. If the available capacity is lower than the backpressure threshold, it is determined to be a high-congestion state, triggering a priority preemption mechanism and marking the current data packet as a high-priority real-time interactive signaling. For the low-congestion state, zero-copy memory mapping technology is used to directly map the physical memory pointer of the semantic fusion entity data to the input buffer address space of the downstream task module, avoiding redundant data copying operations between user mode and kernel mode. For the high-congestion state, an interrupt-driven non-blocking I / O model is adopted, sending a hardware interrupt signal to the downstream task module to forcibly suspend its current low-priority background computing task and immediately allocate a temporary high-speed cache to accept the current semantic fusion entity data. After the data is written, the status register of the downstream task module is updated, the 'new semantic input ready' flag is set, and the computing resource locks associated with the semantic anchor in the current anchor scheduler are released, including the space occupied by the local visual representation cache and distillation buffer in the video memory.

[0166] Through the aforementioned non-blocking routing and resource release mechanism, the distilled anchor point fusion result generated in the previous step is transformed into an input signal that can be consumed by downstream business logic in real time, realizing millisecond-level connection of multimodal interaction response links and dynamic recycling of computing resources, significantly reducing end-to-end interaction latency and improving system concurrency throughput.

[0167] For example, in a smart meeting scenario, when a user issues the command "Show the sales report for the previous quarter," the anchor point fusion result data packet generated by S7.2 is 4KB in size, containing a sequence of visual feature vectors pointing to the "sales report" entity and the intent label "ShowChart." The system queries the downstream "chart rendering engine" and finds that the input buffer has 80% remaining capacity, which is higher than the preset 50% backpressure threshold, thus determining a low congestion state. The system uses zero-copy technology to directly map the memory address 0x7F3A... of the 4KB data block to the input queue of the rendering engine, taking less than 1 microsecond. At the same time, the anchor point scheduler immediately releases the occupied 256MB of video memory cache and 32KB of semantic buffer. If the downstream "speech synthesis module" is under high load at this time (buffer remaining 5%), the system will send a high-priority interrupt signal to suspend the background noise analysis task and write the data packet to a temporary L1 cache to ensure that the voice feedback is not interrupted. Ultimately, the chart rendering engine starts the drawing process within 5ms after receiving the data. Compared with the traditional solution of waiting for all modalities to be aligned in sequence, the first frame response time is shortened by about 40ms, and the overall GPU memory usage of the system is reduced by 15%, supporting more concurrent users to initiate multimodal queries at the same time.

[0168] S7.4: Based on the non-compliance status identifier in the binary judgment signal, read the end position information of the current micro time window and the remaining amount of the time interval, and use the incremental request generation algorithm to construct the incremental update request signaling of the non-completed status containing the target time interval expansion instruction and feature completion requirements.

[0169] The system receives the binary decision signal output from the previous step, parses the non-compliance status identifier carried within it, confirms that the current semantic anchor point has not yet met the alignment verification threshold, and triggers the incremental update request generation process. It reads the end position index of the current micro-time window on the timeline and, combined with the total occurrence time interval length defined in the semantic anchor point sequence, calculates the remaining unprocessed time interval span. Based on the remaining time interval span and the preset minimum processing unit duration, it uses a linear interpolation algorithm to calculate the target time interval boundary to be covered in the next iteration cycle.

[0170] The shallow feature residual distribution map in the current image encoder cache is extracted to assess the degree of feature information loss in non-critical regions, generating a feature completion requirement vector representing the integrity of the visual context. The target temporal interval boundary and the feature completion requirement vector are structurally encapsulated to construct an incomplete state incremental update request signaling containing an extended instruction header, a temporal load segment, and a feature constraint segment. The incremental update request signaling is serialized and compressed using a signaling encoding protocol to reduce control flow transmission bandwidth usage and ensure that the signaling arrives at the asynchronous feature extraction channel before the media stream data.

[0171] By using the above processing method, the non-compliance judgment result of the previous step is transformed into an incremental update request signaling with clear timing direction and feature compensation requirements. This enables dynamic on-demand scheduling of high-latency modal processing resources, avoids the waste of computing power caused by global recalculation, and significantly improves the iterative efficiency of multimodal semantic alignment and the real-time response of the system.

[0172] S7.5: Based on the incremental update request signaling of the incomplete state, send a new local visual representation set pull instruction to the asynchronous feature extraction channel, continuously receive and cache the local visual representation set output by the next micro time window, so as to start the iterative alignment loop for the same semantic anchor point until the alignment verification threshold is met.

[0173] This application also provides an intelligent interactive screen system supporting multimodal input and output. This system corresponds to the aforementioned method and can implement all the technical processes described in the method section. The system includes: a semantic segmentation module, an anchor point scheduling module, an attention mask module, a micro-time window module, a contrastive learning module, a consistency judgment module, and a task distribution module. Specifically: The semantic segmentation module is used to acquire speech stream data and text input data and perform semantic segmentation processing to generate intent label sequences. The anchor scheduling module is used to construct a semantic anchor sequence based on the intent label sequence, and input the semantic anchor sequence as the priority scheduling object into the anchor scheduler; The attention mask module is used to acquire the image frame sequence, use the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence, and generate an attention mask to constrain the image encoder to focus on key regions. The micro-time window module is used to divide the local visual representation of the high-delay modality into multiple micro-time windows according to the computational focus of the image encoder constrained by the attention mask, so as to output the set of local visual representations corresponding to each micro-time window. The contrastive learning module is used to input the intent label sequence into the distillation buffer and perform online contrastive learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate visual feature trajectories. The consistency judgment module is used to calculate the semantic consistency index based on the visual feature trajectory and intent label sequence, determine whether the index has reached the preset alignment verification threshold, and generate anchor point fusion results marked as distilled or incremental update requests marked as incomplete. The task distribution module is used to send the distilled anchor point fusion result to the downstream task to perform non-blocking semantic evolution if it determines that a distilled state has been generated. If it determines that an incremental update request for an incomplete state has been generated, it continuously receives new local visual representation sets for iterative alignment.

[0174] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

[0175] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the element or object preceding “comprising” or “including” encompasses the element or object listed following “comprising” or “including” and its equivalents, and do not exclude other elements or objects. The “multiple” mentioned in the embodiments of this application refers to two or more. A and / or B indicate three possibilities: A; B; and A and B.

[0176] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent interaction method supporting multimodal input and output, characterized in that, include: Acquire speech stream data and text input data, perform semantic segmentation processing, and generate intent label sequences; Based on the intent tag sequence, a semantic anchor sequence is constructed, and the semantic anchor sequence is input into the anchor scheduler as a priority scheduling object; Obtain an image frame sequence, and use the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence to generate an attention mask to constrain the image encoder to focus on key regions; Based on the attention mask constraining the computation focus of the image encoder, the local visual representation of the high-delay modality is divided into multiple micro-time windows to output a set of local visual representations corresponding to each micro-time window. The intent label sequence is input into the distillation buffer, and the output of each micro-time window in the local visual representation set is compared and learned online with the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory. Based on the visual feature trajectory and the intent label sequence, a semantic consistency index is calculated. It is then determined whether the index reaches a preset alignment verification threshold, and an anchor point fusion result marked as distilled or an incremental update request marked as incomplete is generated. If it is determined that the anchor point fusion result of the distilled state has been generated, the anchor point fusion result of the distilled state is sent to the downstream task to perform non-blocking semantic evolution. If it is determined that the incremental update request of the incomplete state has been generated, new local visual representation sets are continuously received for iterative alignment.

2. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The step of dividing the local visual representation of the high-delay modality into multiple micro-time windows based on the calculated focus of the image encoder constrained by the attention mask, and outputting a set of local visual representations corresponding to each micro-time window, includes: The attention mask and the complete image frame sequence are obtained. Based on the temporal interval information in the attention mask, key region locking processing is performed on the complete image frame sequence to generate a subset of key region images containing spatial coordinate constraints. Receive a subset of images of key regions and use a shallow feature caching mechanism to perform low-power feature suppression operations on non-key regions to generate a sparse visual feature stream focused on high-confidence regions. Based on the timestamps of the sparse visual feature stream and semantic anchor sequence, a sliding window segmentation algorithm is used to perform micro-time window partitioning on the sparse visual feature stream to generate a set of micro-time window feature segments with independent temporal identifiers. For each micro-time window feature segment in the set of micro-time window feature segments, the fine-grained visual modeling engine is invoked to perform deep feature extraction operations to generate the local visual representation vector of the corresponding micro-time window; The local visual representation vectors of all micro-time windows are aggregated and restructured according to their occurrence time intervals to output the set of local visual representations.

3. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The step of inputting the intent label sequence into the distillation buffer and performing online comparative learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory includes: The intent slot field and core entity field in the intent tag sequence are obtained, and the intent slot field and core entity field are subjected to high-dimensional semantic space projection processing using a domain-adaptive word vector mapping algorithm to generate a set of teacher semantic embedding vectors with temporal alignment attributes. The teacher semantic embedding vector set is received and injected into the write port of the distillation buffer. At the same time, the micro-time window local visual representation vector in the local visual representation set is obtained. Based on the timestamp matching rule, the micro-time window local visual representation vector is paired and associated with the teacher semantic embedding vector set to generate modality contrast learning sample pairs to be optimized. Based on the modal contrast learning sample pair, a cross-modal contrast loss function is constructed. The probability distribution of the teacher semantic embedding vector set is smoothed using a temperature scaling factor to generate a soft-label target distribution. The predicted probability of the micro-time window local visual representation vector under the soft-label target distribution is calculated to output an initial backpropagation signal containing the semantic consistency error gradient. The initial backpropagation signal is subjected to a norm threshold detection operation. When the signal amplitude exceeds the preset gradient explosion suppression threshold, the amplitude is truncated using a gradient clipping algorithm. When the signal amplitude does not exceed the preset gradient explosion suppression threshold, the original value remains unchanged, so as to generate a truncated gradient update amount optimized for stability. The truncated gradient update amount is used to perform an iterative weight update operation on the feature extraction network parameters of the local visual representation vector of the micro-time window, and the updated feature extraction network is reapplied to the current micro-time window image frame subset to reconstruct the visual feature representation. The reconstructed visual feature representations of all micro-time windows are accumulated in chronological order to generate the visual feature trajectory.

4. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The generation of the intent tag sequence includes: performing endpoint localization and streaming speech recognition on the speech stream data to generate candidate transcribed text; performing character normalization and logical segmentation on the text input data to generate a set of discrete text fragments; merging the candidate transcribed text and the set of discrete text fragments to perform entity extraction and temporal alignment; and generating the intent tag sequence containing timestamp intervals, confidence levels, intent slots, and core entities.

5. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The construction of the semantic anchor sequence includes: mapping the timestamps in the intent tag sequence to continuous time intervals, fusing the mapping result with the intent slots in the intent tag sequence to generate original semantic anchor data units, and using the confidence in the intent tag sequence to prioritize and weight the original semantic anchor data units to generate a weighted semantic anchor sequence as the semantic anchor sequence.

6. The intelligent interaction method supporting multimodal input and output according to claim 5, characterized in that, The semantic anchor sequence serves as a control flow independent of the original media stream transmission, and is used to trigger the asynchronous resource allocation process of the anchor scheduler.

7. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The generation of the attention mask includes: performing temporal slice mapping on the image frame sequence based on the temporal interval in the semantic anchor sequence, generating a subset of micro-temporal window image frames, extracting the coordinates of the foreground target and calculating the semantic-spatial correlation with the intention slot, screening a candidate set of key visual regions, and generating the attention mask after binarization and morphological smoothing.

8. The intelligent interaction method supporting multimodal input and output according to claim 7, characterized in that, The candidate set of key visual regions is obtained by calculating and filtering the semantic-spatial correlation between the intent slots in the semantic anchor sequence and the initial set of region of interest coordinates.

9. The intelligent interaction method supporting multimodal input and output according to claim 1, characterized in that, The semantic consistency index uses cosine similarity as a metric and is calculated using a sliding window weighted average with exponentially decaying weights.

10. An intelligent interactive screen system supporting multimodal input and output, characterized in that, include: The semantic segmentation module is used to acquire speech stream data and text input data and perform semantic segmentation processing to generate intent label sequences. Anchor scheduling module is used to construct a semantic anchor sequence based on the intent tag sequence, and input the semantic anchor sequence as a priority scheduling object into the anchor scheduler; The attention mask module is used to acquire an image frame sequence, use the temporal information in the anchor point sequence to perform asynchronous feature extraction on the image frame sequence, and generate an attention mask to constrain the image encoder to focus on key regions. The micro-time window module is used to divide the local visual representation of the high-delay modality into multiple micro-time windows according to the computational focus of the image encoder constrained by the attention mask, so as to output a set of local visual representations corresponding to each micro-time window. The contrastive learning module is used to input the intent label sequence into the distillation buffer and perform online contrastive learning between the output of each micro-time window in the local visual representation set and the semantic embedding of the corresponding semantic anchor point to generate a visual feature trajectory. The consistency judgment module is used to calculate the semantic consistency index based on the visual feature trajectory and the intent label sequence, determine whether the index reaches the preset alignment verification threshold, and generate an anchor point fusion result marked as distilled or an incremental update request marked as incomplete. The task distribution module is used to send the anchor point fusion result of the distilled state to the downstream task to perform non-blocking semantic evolution if it is determined that the anchor point fusion result of the distilled state has been generated, and to continuously receive new local visual representation sets for iterative alignment if it is determined that the incremental update request of the incomplete state has been generated.

Images