A system and method for processing and feature fusion of moniliform long sequence multi-modal data

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a system and method for processing and fusing single-unit long-sequence multimodal data. The invention achieves dynamic adaptive sampling through spatiotemporal information density quantization, truncating redundant frames at the feature extraction source. It embeds multimodal hybrid experts into the LLM layer and, in conjunction with RMS variance recalibration, achieves spatial physical isomorphism between audiovisual features and text representations, eliminating feature loss and parameter redundancy in cascaded architectures. It employs dual information theory constraints of mutual information maximization and minimum description length to achieve lossless cross-modal alignment, constructing a hierarchical temporal index and a temporal decay sparse retrieval mechanism to accurately locate key long-sequence features within a limited context window. This invention significantly reduces the memory overhead of ultra-long video inference, improves the accuracy of long-sequence causal inference, and significantly suppresses multimodal hallucination generation. It is suitable for scenarios such as intelligent surveillance long-video auditing, long-video question answering, multimodal complex document parsing, and embodied intelligent perception. Tests show that the accuracy of long-sequence tasks is improved by up to 22.6% compared to traditional cascaded models, and the accuracy of temporal localization tasks is improved by 27.3%, with a significant reduction in the incidence of hallucinations.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the fields of artificial intelligence and multimodal large model technology, and particularly relates to a system and method for processing and feature fusion of single-unit long sequence multimodal data. Background Technology

[0002] In terms of industry development overview and existing technological evolution, artificial intelligence technologies represented by large language models have made breakthrough progress in recent years, demonstrating unprecedented generalization capabilities in natural language understanding, logical reasoning, and text generation. However, information in the real world is multimodal, and pure text models cannot directly perceive the rich representations of the physical world. Therefore, the development of multimodal large models aims to endow artificial intelligence systems with the ability to perceive vision and hearing. Currently, the technological evolution of multimodal large models is undergoing a leap from static single-frame image understanding to dynamic long-sequence video and audio understanding. In practical application scenarios, such as long-sequence meeting record analysis, movie plot question answering, or all-weather monitoring video auditing for autonomous driving, the system needs to simultaneously process ultra-long sequence inputs containing massive visual frames, continuous audio streams, and complex user commands. This requires the model to not only have single-modal representation capabilities, but also strong cross-modal information integration capabilities, long-term memory capabilities, and the ability to accurately retrieve information within a vast context.

[0003] In current mainstream multimodal technology architectures, a cascaded architecture is typically used to achieve multimodal understanding. This involves extracting features using an externally pre-trained visual encoder and then mapping them to the token space of a language model via a cross-modal bridging layer. For long sequence data, conventional approaches are limited by the large model context window length and the quadratic computational complexity of self-attention mechanisms, often employing strategies such as uniform frame sampling, sliding window block processing, or pooling compression. These methods attempt to sacrifice temporal density or local details for computational feasibility. At the level of cross-modal semantic alignment, existing technologies largely rely on contrastive learning or simple autoregressive next-word prediction tasks, implicitly bridging the distance between different modalities in the latent space to achieve alignment.

[0004] However, a deeper analysis of existing technologies reveals that the primary problem with cascaded architectures is the severe fragmentation of the representation space. The training objective of the external visual encoder is typically global contrastive learning, lacking the capture of local details and dense spatial relationships. When these heterogeneous features are introduced into the language model space through a simple projection layer, severe feature loss occurs. Simultaneously, the external encoder and the larger model have independent parameter spaces, requiring separate forward propagation for long sequences, resulting in extremely high memory consumption and exponentially increasing inference latency. Furthermore, when attempting to fine-tune the language model to adapt to multimodal tasks, the vast differences in feature distribution between visual and text make forced joint optimization highly susceptible to gradient conflicts, destroying the linguistic logic and world knowledge accumulated during pre-training and leading to catastrophic forgetting.

[0005] When processing ultra-long multimodal sequences, the contradiction between computational overhead and information dilution becomes increasingly prominent. For long video data, even with low-resolution extraction, an hour of video can generate hundreds of thousands of tokens, exceeding the processing limits of most Transformer architectures. Existing rigid uniform sampling mechanisms completely ignore the uneven information density in long sequences, leading to the extensive repetitive computation of low-density information such as static backgrounds, while high-density information such as instantaneous key dialogues or action conflicts may be lost due to omissions. This mechanism severs long-range dependencies, causing models to often get lost in massive amounts of redundant features when facing complex relational reasoning spanning long time sequences, making it difficult to accurately locate sparse key features related to the current instruction from ultra-long contexts.

[0006] Furthermore, alignment methods lacking rigorous information theory constraints lead to severe semantic shifts and multimodal illusions. Because existing techniques lack rigorous mathematical constraints when performing cross-modal mappings, models often only learn shallow statistical correlations without achieving true semantic isomorphism. The forced fusion of spatial-temporal information from videos, frequency-domain information from audio, and symbolic information from text without normalization easily results in disordered representation spaces within the model. When the model cannot find precise visual evidence in long contexts, the language model's text generation inertia may fabricate non-existent facts based on prior knowledge. In long-term tasks, this illusion rate typically spikes non-linearly, severely impacting the logical rigor and reliability of the output.

[0007] In recent years, academia has begun to explore the introduction of hybrid expert architectures (MoEs) into the multimodal domain (e.g., visual sparse MoEs for embodied intelligence or hierarchical MoEs for action recognition). However, these existing solutions are essentially still "plug-in" or "patchwork" expert networks. When processing multimodal features, they often simply assign experts to specific network layers, ignoring the "variance drift" problem when cross-modal data enters large language models. Furthermore, when dealing with extremely long sequences, existing hierarchical MoEs largely rely on static spatiotemporal layering, lacking dynamic adaptive truncation mechanisms based on spatiotemporal information density. More critically, existing technologies rely solely on empirical contrastive loss or cross-entropy loss for cross-modal feature fusion, lacking strict mathematical constraints based on information theory (such as maximizing mutual information and minimizing description length). This lack of underlying constraints makes existing models highly susceptible to severe "multimodal illusions" due to the accumulation of visual redundancy noise when handling long-term complex logical reasoning tasks exceeding 30 minutes. Summary of the Invention

[0008] To address the underlying shortcomings of the aforementioned cascaded architecture, the purpose of this invention is to provide a system and method for processing and fusing single-unit long-sequence multimodal data.

[0009] This invention is implemented as follows: a system for processing and fusing single-unit long-sequence multimodal data, comprising:

[0010] The long sequence temporal adaptive retrieval module is used to receive ultra-long video streams, audio streams and text commands in parallel, calculate the spatiotemporal information density of the multimodal data stream in real time, perform dynamic adaptive sampling based on the spatiotemporal information density, physically truncate redundant segments with low information density, densely sample temporal intervals with high information density, and output compressed key frame feature sequences.

[0011] The endogenous multimodal perception module is embedded in the underlying Transformer network of the monolithic large language model. It incorporates a multimodal hybrid expert MoE network, a gated routing network with modality bias, and an RMS variance recalibration operator. It receives compressed keyframe feature sequences, distributes audiovisual features to matching endogenous visual experts and text experts through gated dynamic routing, and aligns the distribution and magnitude of visual features to the text word embedding benchmark through RMS variance recalibration, outputting a multimodal endogenous representation in a unified parameter space.

[0012] The cross-modal information theory alignment module constructs a hierarchical temporal index tree coupled with macro-level segments and micro-level keyframes. After encoding text instructions into query vectors, it performs sparse similarity retrieval with temporal decay bias to filter core visual features that match the query. Irrelevant visual noise is filtered out by dual information theory constraints of maximizing mutual information MIM loss and minimizing description length MDL loss. The aligned and purified multimodal features are then fed into a single large language model decoder, which outputs natural language or structured semantic results via autoregression.

[0013] Furthermore, the formula for calculating the spatiotemporal information density by the long-sequence time-adaptive retrieval module is as follows:

[0014]

[0015] Representative extracted from the first Spatial tensor of each feature channel This represents the Shannon entropy calculated from the pixel grayscale or color histogram distribution. and These are the hyperparameters that are fixed in the system during the pre-training phase;

[0016] The dynamic sampling rate cutoff function is:

[0017]

[0018] Typically, the extraction rate is set based on the maximum memory capacity of a single GPU, which determines the number of key feature blocks extracted per second. It may extract as few background features as per second, where γ is the density scaling factor.

[0019] Furthermore, the hybrid expert dynamic routing output formula of the endogenous multimodal perception module is as follows:

[0020]

[0021] Gating weight calculation formula:

[0022]

[0023] W bias For modal bias terms;

[0024] Routing load balancing loss:

[0025]

[0026] Visual feature RMS recalibration alignment formula:

[0027] ⊙γ

[0028] x v ϵ represents the original visual features, γ is the scaling factor, and ϵ is the numerically stable minimum value.

[0029] Accompanying two-stage endogenous visual pre-training loss:

[0030] .

[0031] Furthermore, the time-aware sparse retrieval similarity formula for the cross-modal information theory alignment module is:

[0032]

[0033] Mt emporal (i) is the temporal decay bias matrix, which applies a similarity penalty to long-distance temporal features;

[0034] Mutual information maximization loss:

[0035]

[0036] Minimum description length information bottleneck loss:

[0037]

[0038] The model jointly fine-tunes the total loss function:

[0039]

[0040] L LLMGen For the autoregressive generation loss of large language models, w 1、 w2 is the multi-objective loss balancing weight.

[0041] Another objective of this invention is to provide a method for processing and fusing features of single-unit long sequence multimodal data, characterized by the following steps, executed by the system:

[0042] S101: Parallel reception of ultra-long video streams, audio streams and user text commands; long sequence time-adaptive retrieval module calculates spatiotemporal information density frame by frame in real time.

[0043] S102: Perform dynamic adaptive sampling based on spatiotemporal information density, physically truncate low-information-density background frames, densely sample high-information-density temporal intervals, and output compressed keyframe sequences.

[0044] S103: Input the keyframe into the endogenous multimodal perception module, and complete the spatial physical isomorphism between visual representation and text representation through the RMS variance recalibration operator;

[0045] S104: The dynamic routing network with modal bias distributes audiovisual features to corresponding endogenous visual experts and text experts, and the residual connection fuses and outputs a unified multimodal endogenous representation.

[0046] S105: Construct a hierarchical temporal index tree of macro segments + micro keyframes, encode text instructions into query vectors, and perform sparse similarity retrieval with temporal decay bias to filter core visual features; filter irrelevant visual noise through dual information theory constraints of maximizing mutual information and minimizing description length.

[0047] S106: The aligned and purified clean multimodal context is fed into the monolithic large model decoder, and the autoregressive decoder outputs time-aligned, logically unillusioned natural language or structured analysis text.

[0048] Furthermore, the dynamic adaptive sampling in step S102 includes: when the spatiotemporal information density ρ(t) is lower than a set threshold, physically truncating and discarding the corresponding frame of the current video segment; when ρ(t) is higher than the set threshold, increasing the sampling rate to extract key frames at high density, and avoiding token explosion by limiting the upper and lower bounds of the sampling quantity.

[0049] Furthermore, the hierarchical temporal index tree construction method in step S105 is as follows: macroscopic time segments are divided according to the peak of spatiotemporal information density, and each segment generates a macroscopic index vector through global average pooling; all microscopic key frame features obtained by adaptive sampling are stored inside the segment, and the macroscopic segment index is matched first, and then the internal microscopic key frames are retrieved to reduce the amount of retrieval computation.

[0050] Another object of the present invention is to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, the computer program being executed by the processor causing the processor to perform the steps of the single-unit long sequence multimodal data processing and feature fusion method.

[0051] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the single-unit long sequence multimodal data processing and feature fusion method.

[0052] Another objective of this invention is to provide an information data processing terminal, which is used to implement the single-unit long sequence multimodal data processing and feature fusion system.

[0053] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0054] This invention also provides a complete method for processing and fusing single-unit long-sequence multimodal data. This method fully presents the end-to-end processing chain of the system from receiving raw heterogeneous data to outputting high-precision logical text. In the actual inference execution process, the system first receives ultra-long temporal video streams, audio streams, and user natural language commands in parallel. For the input multimodal data stream, the system does not immediately perform global feature encoding, but instead first activates the long-sequence temporal adaptive retrieval module. In the shallow network, the system calculates the spatiotemporal information density of the multidimensional data stream in real time, and dynamically allocates the feature sampling rate based on the non-uniform distribution characteristics of action amplitude and scene complexity. For background redundant segments determined to have low information density, the system physically truncates them before feature extraction, while for key feature intervals with information abrupt changes, the highest density sampling strategy is applied.

[0055] After completing information density-based adaptive compression, the preserved sparse keyframes and audio segments are simultaneously fed into the endogenous multimodal perception module. At this point, the system uses a parameterized gating network to precisely route different visual or auditory tensors to the best-matching endogenous hybrid expert layer. The extraction and dimensionality reduction of the visual tensor are performed entirely within the low-level parameter space of the large language model, and an improved root mean square layer normalization technique forces the variance and magnitude of the visual features to be calibrated to a physical baseline completely consistent with the text word embeddings. This process completely eliminates the representational barrier between the external perception module and the inference core, enabling discrete physical signals to be successfully transformed into endogenous continuous representations that the language model can process.

[0056] Subsequently, the system feeds the extracted multimodal endogenous representations and the user's text query instructions into the cross-modal information theory alignment module. At this stage, the system no longer calculates the global attention matrix but performs context-aware local sparse retrieval, activating only feature blocks with extremely high cosine similarity to the user's instructions. Crucially, these feature blocks are strictly constrained by the principles of maximizing cross-modal mutual information and minimizing description length before entering the core language decoder. While ensuring absolute isomorphism across semantic boundaries, the system filters out all visual noise irrelevant to the current instruction. Finally, in a highly pure and temporally continuous multimodal context, the language model initiates an autoregressive generation mechanism, outputting a logically rigorous natural language answer or structured analysis report that is highly aligned with the user's intent.

[0057] Compared with the existing mainstream technical solutions in this field, the single-unit long sequence multimodal data processing and feature fusion system and method proposed in this invention have extremely significant technical breakthroughs and beneficial effects.

[0058] First, at the underlying architecture level, this invention completely eliminates the inherent problems of fragmented representations and catastrophic forgetting inherent in traditional cascaded architectures through its original monolithic endogenous perception design. By directly embedding multimodal experts into the underlying language model and performing feature normalization within a unified parameter space, this system reduces the loss of cross-modal feature transformation to a theoretical minimum. This architecture not only enables the model to maintain absolute stability of its original text reasoning capabilities during large-scale multimodal joint training, but also significantly reduces the overall parameter redundancy and inference memory overhead of the system, making it possible to deploy high-precision multimodal models on resource-constrained hardware devices.

[0059] Secondly, in terms of temporal processing, this invention introduces for the first time an adaptive sampling and sparse retrieval mechanism based on information theory constraints within a multimodal large model, perfectly solving the computational disaster caused by ultra-long sequences. Traditional uniform sampling methods often suffer from inconsistencies in long video processing, while this system can dynamically allocate resources based on priority, much like human visual mechanisms. When faced with surveillance streaming media or complex movie sequences exceeding thirty minutes or even several hours in length, this system can accurately locate sparse key features hidden in massive amounts of redundant information with extremely high recall under strict context window constraints, completely eliminating the loss of intermediate information in ultra-long contextual reasoning and significantly improving the model's accuracy in temporal causal association and long-range memory tasks.

[0060] Finally, in terms of semantic alignment and hallucination suppression, this invention constructs a highly rigorous mathematical optimization foundation. Through joint optimization of maximizing mutual information and minimizing description length, this system achieves deep semantic isomorphism between heterogeneous physical signals and discrete text symbols. This mechanism ensures that each output logic of the language model has a clear visual or auditory feature anchor point when generating the final inference result. When facing highly complex long video understanding tasks, this system effectively curbs the bottomless associations caused by the autoregressive generation of the language model, significantly reducing the incidence of multimodal hallucinations. This invention not only ensures the factual accuracy of the output results but also provides a solid and reliable technical foundation for artificial intelligence in application scenarios with extremely high fault tolerance requirements, such as autonomous driving, medical assisted diagnosis, and high-precision industrial auditing.

[0061] This invention addresses the pain points of existing cascaded multimodal large models in long sequence processing, such as feature fragmentation, computational bottlenecks, and severe illusions. It proposes a single-unit solution with solid underlying logic, and its core innovations are reflected in the following three dimensions:

[0062] 1. Breakthrough in Architectural Dimension: Intrinsic Perception and Physical Isomorphism under a Unified Parameter Space

[0063] This invention completely abandons the traditional external visual encoder and linear bridging layer, directly embedding the multimodal hybrid expert (MoE) network into the basic structure of a pre-trained large language model. By implementing layer normalization and feature recalibration based on the variance benchmark of the pre-trained vocabulary at the model's bottom layer, this system achieves physical isomorphism from pure visual signals to the pure text semantic space. This innovation not only fundamentally eliminates variance drift and catastrophic forgetting caused by cross-domain feature transfer, but also significantly reduces the massive parameter redundancy brought by external modules, opening up a completely new path for deploying high-precision multimodal models under hardware conditions with limited computing resources.

[0064] 2. Breakthrough in temporal processing: Adaptive truncation and sparse retrieval based on spatiotemporal information density

[0065] To address the context token explosion problem caused by long sequences of data, such as ultra-long videos, this invention breaks away from the traditional model's reliance on static uniform frame extraction or rigid spatial hierarchical segmentation. The system innovatively introduces a dynamic evaluation mechanism for spatiotemporal information density, adaptively allocating sampling weights by quantifying inter-frame motion mutations and single-frame spatial complexity in real time. This mechanism can accurately truncate low-information-density background redundant frames at the physical source and perform dense feature extraction within key action intervals. Combined with hierarchical indexing and context-aware retrieval, this invention effectively mitigates the computational curse in long video processing and completely avoids the intermediate information loss trap in long-range inference.

[0066] 3. Breakthroughs in Cognition and Alignment Dimensions: Multimodal Illusion Suppression under the Dual Constraints of Information Theory

[0067] In the deep fusion stage of cross-modal features, this invention constructs an extremely rigorous theoretical constraint foundation. The system not only ensures absolute isomorphism between the visual manifold and textual semantics through Mutual Information Maximization (MIM), but also innovatively introduces the Minimum Description Length (MDL) principle to construct an information bottleneck. This dual constraint forces the perceptual network to strip away all visual redundancy noise that cannot provide text prediction value, ensuring that every visual feature fed into the decoder is highly pure. This innovation blocks the controlled association bias caused by redundant feature interference in large models from the logical source, enabling the system to exhibit an extremely low multimodal illusion incidence rate and rigorous factual accuracy when handling complex causal reasoning tasks. Attached Figure Description

[0068] Figure 1 This is a diagram of the overall architecture of a monolithic multimodal system.

[0069] Figure 2 For endogenous multimodal perception and expert routing graphs;

[0070] Figure 3 Flowchart for adaptive dynamic compression of long sequences;

[0071] Figure 4 A flowchart illustrating the hierarchical indexing and sparse retrieval process;

[0072] Figure 5 This defines the end-to-end inference workflow for the system. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0074] like Figure 1 As shown, this embodiment of the invention provides a system for processing and fusing single-unit long sequence multimodal data, the system comprising:

[0075] like Figure 3 , 4 As shown, the long sequence temporal adaptive retrieval module is used to receive ultra-long video streams, audio streams and text commands in parallel, calculate the spatiotemporal information density of the multimodal data stream in real time, perform dynamic adaptive sampling based on the spatiotemporal information density, physically truncate redundant segments with low information density, densely sample temporal intervals with high information density, and output the compressed key frame feature sequence.

[0076] like Figure 2 As shown, the endogenous multimodal perception module is embedded in the underlying Transformer network of the monolithic large language model. It incorporates a multimodal hybrid expert MoE network, a gated routing network with modality bias, and an RMS variance recalibration operator. It receives compressed keyframe feature sequences, distributes audiovisual features to matching endogenous visual experts and text experts through gated dynamic routing, and aligns the distribution and magnitude of visual features to the text word embedding benchmark through RMS variance recalibration, outputting a multimodal endogenous representation in a unified parameter space.

[0077] The cross-modal information theory alignment module constructs a hierarchical temporal index tree coupled with macro-level segments and micro-level keyframes. After encoding text instructions into query vectors, it performs sparse similarity retrieval with temporal decay bias to filter core visual features that match the query. Irrelevant visual noise is filtered out by dual information theory constraints of maximizing mutual information MIM loss and minimizing description length MDL loss. The aligned and purified multimodal features are then fed into a single large language model decoder, which outputs natural language or structured semantic results via autoregression.

[0078] Furthermore, the formula for calculating the spatiotemporal information density by the long-sequence time-adaptive retrieval module is as follows:

[0079]

[0080] Representative extracted from the first Spatial tensor of each feature channel This represents the Shannon entropy calculated from the pixel grayscale or color histogram distribution. and These are the hyperparameters that are fixed in the system during the pre-training phase;

[0081] The dynamic sampling rate cutoff function is:

[0082]

[0083] Typically, the extraction rate is set based on the maximum memory capacity of a single GPU, which determines the number of key feature blocks extracted per second. It may extract as few background features as per second, where γ is the density scaling factor.

[0084] Furthermore, the hybrid expert dynamic routing output formula of the endogenous multimodal perception module is as follows:

[0085]

[0086] Gating weight calculation formula:

[0087]

[0088] W bias For modal bias terms;

[0089] Routing load balancing loss:

[0090]

[0091] Visual feature RMS recalibration alignment formula:

[0092] ⊙γ

[0093] x v ϵ represents the original visual features, γ is the scaling factor, and ϵ is the numerically stable minimum value.

[0094] Accompanying two-stage endogenous visual pre-training loss:

[0095] .

[0096] Furthermore, the time-aware sparse retrieval similarity formula for the cross-modal information theory alignment module is:

[0097]

[0098] Mt emporal(i) is the temporal decay bias matrix, which applies a similarity penalty to long-distance temporal features;

[0099] Mutual information maximization loss:

[0100]

[0101] Minimum description length information bottleneck loss:

[0102]

[0103] The model jointly fine-tunes the total loss function:

[0104]

[0105] L LLMGen For the autoregressive generation loss of large language models, w 1、 w2 is the multi-objective loss balancing weight.

[0106] like Figure 5 As shown, this embodiment of the invention provides a method for processing and fusing single-unit long sequence multimodal data, characterized in that, based on the system execution, it includes the following steps:

[0107] S101: Parallel reception of ultra-long video streams, audio streams and user text commands; long sequence time-adaptive retrieval module calculates spatiotemporal information density frame by frame in real time.

[0108] S102: Perform dynamic adaptive sampling based on spatiotemporal information density, physically truncate low-information-density background frames, densely sample high-information-density temporal intervals, and output compressed keyframe sequences.

[0109] S103: Input the keyframe into the endogenous multimodal perception module, and complete the spatial physical isomorphism between visual representation and text representation through the RMS variance recalibration operator;

[0110] S104: The dynamic routing network with modal bias distributes audiovisual features to corresponding endogenous visual experts and text experts, and the residual connection fuses and outputs a unified multimodal endogenous representation.

[0111] S105: Construct a hierarchical temporal index tree of macro segments + micro keyframes, encode text instructions into query vectors, and perform sparse similarity retrieval with temporal decay bias to filter core visual features; filter irrelevant visual noise through dual information theory constraints of maximizing mutual information and minimizing description length.

[0112] S106: The aligned and purified clean multimodal context is fed into the monolithic large model decoder, and the autoregressive decoder outputs time-aligned, logically unillusioned natural language or structured analysis text.

[0113] Furthermore, the dynamic adaptive sampling in step S102 includes: when the spatiotemporal information density ρ(t) is lower than a set threshold, physically truncating and discarding the corresponding frame of the current video segment; when ρ(t) is higher than the set threshold, increasing the sampling rate to extract key frames at high density, and avoiding token explosion by limiting the upper and lower bounds of the sampling quantity.

[0114] Furthermore, the hierarchical temporal index tree construction method in step S105 is as follows: macroscopic time segments are divided according to the peak of spatiotemporal information density, and each segment generates a macroscopic index vector through global average pooling; all microscopic key frame features obtained by adaptive sampling are stored inside the segment, and the macroscopic segment index is matched first, and then the internal microscopic key frames are retrieved to reduce the amount of retrieval computation.

[0115] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the single-unit long sequence multimodal data processing and feature fusion method.

[0116] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the single-unit long sequence multimodal data processing and feature fusion method.

[0117] This invention provides an information data processing terminal, which is used to implement the single-unit long sequence multimodal data processing and feature fusion system.

[0118] After completing the underlying hardware deployment of the monolithic large model and the physical isomorphism of the intrinsic perception expert, the core engineering challenge facing the system is how to process ultra-long multimodal sequences lasting several hours under limited GPU memory and computational constraints. To address the problems of inefficient dissipation of computational resources and omission of key information caused by traditional uniform frame extraction, this invention introduces a long-sequence temporal adaptive retrieval module. This module acts as a bridge connecting the perception base and the core logic inference engine. Its core logic lies in abandoning static temporal dimension segmentation and instead adopting a dynamic density evaluation and sparse retrieval mechanism guided by information theory. After the system receives a continuous video stream, it first pre-evaluates the physical information content of the original sequence through a shallow three-dimensional convolution operator or temporal difference network with extremely small parameters, thereby achieving the first physical truncation before entering the deep network of the large model.

[0119] In the specific implementation of the adaptive sampling mechanism, the system requires real-time quantization of the spatiotemporal information density within any continuous time window. (Regarding time...) The system employs multimodal feature slicing, extracting pixel-level motion vectors between frames to assess temporal abrupt changes, and simultaneously calculating spatial information entropy within a single frame to evaluate scene complexity. The density evaluation function built into the system controller is rigorously defined as a weighted sum of the Euclidean distance norm and spatial entropy of the feature differences between adjacent frame channels.

[0120]

[0121] In this expression, Representative extracted from the first Spatial tensor of each feature channel This represents the Shannon entropy calculated from the pixel grayscale or color histogram distribution. and These are the hyperparameters that the system solidifies during the pre-training phase. Through this underlying quantization calculation, the system can accurately capture high-dimensional semantically dense regions such as rapid movement and camera transitions, and clearly distinguish them from long-term static monitoring backgrounds or single-color block images.

[0122] Based on the obtained spatiotemporal information density scalar sequence, the system then triggers a dynamic adaptive sampling actuator. This actuator internally stores a nonlinear sampling mapping curve, which is based on the density value of the current time window. The number of feature units to be extracted for real-time simulation. To prevent multimodal token explosion in extremely chaotic scenarios while ensuring minimum contextual coherence in extremely static scenarios, this sampling rate function... Strict upper and lower bound truncation constraints were imposed:

[0123]

[0124] In a real-world deployment environment, Typically, the extraction rate is set based on the maximum memory capacity of a single GPU, which determines the number of key feature blocks extracted per second. This could result in extracting as few background features as per second. After physical filtering by this function, the original input stream, which was two hours long and contained millions of video frames, was adaptively compressed into a highly condensed non-uniform feature sequence with a length within a computational threshold. This cleared the computational barrier for subsequent deep network inference.

[0125] After completing the physical layer feature compression, the system constructs a hierarchical content representation architecture that couples macroscopic and microscopic features. Directly mixing tens of thousands of visual features with text instructions for global self-attention computation would still trigger a quadratic increase in the computational complexity of large language models. Therefore, the system dynamically constructs a temporal feature index tree in memory. In this index tree, the video is divided into multiple macroscopic time segments based on the physical boundaries of scene transitions, and a globally averaged semantic vector is calculated for each segment as a macroscopic index; within each segment, all microscopic keyframe features that have undergone adaptive sampling are retained. This data structure construction makes multimodal features no longer a flat linear sequence, but a hierarchical addressing capability with extremely high parallel retrieval efficiency.

[0126] When a user inputs a complex natural language query, the system's core scheduler directly bypasses the computationally expensive global self-attention layer, triggering a context-aware sparse retrieval mechanism based on hierarchical content representation. In this stage, the system first maps the user's text command into a query vector through the language model's underlying word embedding network. Subsequently, the system uses a microscopic keyframe feature library. The system performs efficient dot product similarity retrieval. To prevent similarity calculation from disrupting the original causal temporal logic, a temporal decay bias matrix is ​​forcibly implanted into the standard dot product attention, and its retrieval scoring formula is reconstructed as follows:

[0127]

[0128] matrix The system applies non-linear penalty weights based on the time span between feature frames and the current inference time, ensuring that the system prioritizes extracting causal evidence chains that conform to common sense. Finally, the system sorts the calculated scores in descending order, selecting only the very few visual feature blocks corresponding to the highest scores for deep logical operations in the large language model. Through this high-precision sparse retrieval mechanism, the system completely avoids the pitfall of losing intermediate information in long video inference, achieving accurate visual evidence localization and focusing.

[0129] After achieving physical isomorphism of the endogenous perception layer of the monolithic large model, the core engineering challenge facing the system is how to process ultra-long multimodal sequences lasting several hours under limited GPU memory and computational constraints. To address the issues of inefficient computational resource dissipation and missing key information caused by traditional uniform frame extraction, this invention constructs a long-sequence temporal adaptive retrieval module. The underlying computation graph of this module is typically built based on mainstream deep learning frameworks, serving as a bridge connecting the perception base and the core logic inference engine. Its core logic lies in abandoning static temporal dimension segmentation and instead adopting a dynamic density evaluation and sparse feature sampling mechanism guided by information theory. After receiving a continuous video feature tensor stream, the system does not directly feed it into the deep network. Instead, it first performs forward propagation calculation on the physical information of the original sequence through a lightweight temporal difference evaluation operator, thereby achieving the first stage of dynamic physical truncation before entering the deep structure of the large model.

[0130] In the specific engineering implementation of the adaptive sampling mechanism, the system requires real-time quantization of the spatiotemporal information density within any continuous time window. (Regarding time...) The system employs multimodal feature slicing, extracting not only pixel-level feature distances between frames to assess temporal abrupt changes, but also calculating the activation value distribution of the spatial tensor of a single frame image to assess scene complexity. An internal density evaluation network is embedded in the system controller; the computational logic of this network is rigorously defined as a weighted sum of the Euclidean distance norm of the feature differences between adjacent frame channels and the spatial information entropy.

[0131]

[0132] In the tensor operations described above, Represents the first layer extracted from the lower-level perception layer. High-dimensional tensors of each feature channel This represents the spatial information entropy obtained by calculating the activation probability distribution of the feature map. and These are the scalar hyperparameters that the system solidifies during the backpropagation pre-training phase. Through this underlying quantization calculation, the system can accurately capture high-dimensional semantically dense regions such as rapid motion and camera transitions, and clearly distinguish them numerically from low-information-density sequences such as long-term static surveillance backgrounds.

[0133] Based on the one-dimensional tensor sequence of spatiotemporal information density obtained through the forward propagation, the system then triggers a dynamic adaptive sampling executor. This executor internally maps a nonlinear dynamic sampling curve, based on the density value of the current time window. The number of sequence tokens to be retained in real-time simulation. To prevent feature dimension explosion in extremely chaotic scenarios while ensuring minimum contextual semantic coherence in extremely static scenarios, this dynamic sampling rate function... Strict upper and lower bound truncation operations were applied to the computation graph:

[0134]

[0135] In a real-world hardware deployment environment, The value of is strictly limited by the GPU's memory limit, and is usually set to extract a certain number of key feature blocks per second. The processing power is then reduced to extracting only a very small number of background features per second. After physical filtering by this truncation function, the original input stream, which lasts for hours and contains a massive number of redundant frames, is adaptively compressed into a highly condensed non-uniform feature sequence with a length within the threshold of the model context window. This removes the computational barrier for subsequent deep multi-head attention computation.

[0136] After completing the tensor compression at the physical layer, the system constructs a hierarchical content representation architecture coupling macroscopic segments and microscopic keyframes based on the requirements of long sequence modeling. Directly mixing tens of thousands of compressed visual feature tokens with text instructions for global self-attention computation still leads to memory overflow due to the quadratic complexity of the underlying Query-Key-Value (QKV) matrix multiplication. Therefore, the system dynamically instantiates a temporal feature index tree in the video memory. In this data structure, continuous video sequences are divided into multiple macroscopic time segments based on the peak physical boundary of the density function. Each segment is reduced in dimensionality through global average pooling to generate a highly abstract semantic vector as a macroscopic index; while the leaf nodes of the segment fully retain all adaptively sampled microscopic keyframe feature tensors. This composite data structure transforms multimodal features from flat linear sequences into a hierarchical addressing space with extremely high parallel retrieval efficiency.

[0137] When a complex natural language intent query command is input, the system's core scheduler directly bypasses the computationally expensive global cross-modal attention layer, triggering a context-aware sparse retrieval mechanism based on hierarchical content representation. In this stage, the system first maps the user's intent command into a query vector through the language model's underlying embedding network and linear layer. Subsequently, the key vector matrix is ​​constructed from the micro-keyframe feature library. The system performs efficient dot-product similarity retrieval. To prevent purely semantic similarity matching from disrupting the original causal temporal logic of the video, the system forcibly implants a temporal decay bias matrix into the standard dot-product attention scoring mechanism using tensor broadcasting. The forward calculation formula for its retrieval scoring is reconstructed as follows:

[0138]

[0139] In this formula, The matrix applies an exponential, non-linear penalty weight based on the absolute time span of the feature frame from the current inference moment, ensuring that the system prioritizes extracting time-based evidence chains that conform to common-sense causality. The system then truncates the similarity score tensor in descending order, activating only the few visual value vectors corresponding to the highest scores, and feeding them into the deep decoder of the large language model for cross-modal intent recognition and logical reasoning. Through this high-precision sparse retrieval and underlying routing mechanism, the system completely avoids the pitfall of intermediate information loss in long video sequence parsing at the underlying code logic level.

[0140] After completing the adaptive sparse retrieval of long sequence features, the extracted micro-keyframe tensors need to be forced to be isomorphic with plain text instructions in a deeper semantic space. In a specific embodiment of this invention, this process is performed by a cross-modal information theory alignment module. Traditional multimodal alignment typically relies on simple mean squared error or cross-entropy loss, which often only captures shallow statistical correlations between features and is prone to causing severe semantic shifts and multimodal illusions when processing long contexts. To fundamentally address this physical deficiency, the implementation of this system introduces a dual constraint mechanism based on information theory—maximizing mutual information and minimizing description length—in the underlying computation graph.

[0141] In the specific engineering implementation of maximizing cross-modal mutual information, the system does not directly compute the joint probability distribution of the high-dimensional tensor space. Instead, it performs surrogate optimization during mini-batch training by constructing a lower bound for noise contrast estimation. During the model's forward propagation phase, the system defines paired visual feature representations and text semantic representations within the same physical spacetime as positive sample pairs, while defining other unrelated visual and text combinations within the batch as negative sample pairs. Between the output of the multimodal hybrid expert network and the core decoding layer of the large language model, an automatic differential operator for calculating cross-modal mutual information loss is instantiated. The loss function of this operator is rigorously defined as:

[0142]

[0143] In this formula, scalar This represents the score calculated after the inner product similarity between the visual feature tensor and the text feature tensor. The temperature hyperparameter, used to control the smoothness of the probability distribution, is typically decayed to a minimum value using an annealing algorithm in practical hardware deployments. This represents the total number of negative samples in the current computation batch. By continuously minimizing this loss value during backpropagation, the system forces the neuronal representations excited by the endogenous visual expert network to approximate and predict the corresponding text distribution to the maximum extent in terms of statistical topology, thereby achieving true cross-modal semantic isomorphism at the physical manifold level.

[0144] However, in the complex logical reasoning of ultra-long videos, simply achieving semantic isomorphism still presents underlying risks. Excessively rich visual feature tensors often contain a large amount of background noise unrelated to the current text instructions, such as lighting changes and irrelevant entities. These redundant features can severely interfere with the causal reasoning of the language model. Therefore, this embodiment further incorporates a minimum description length constraint mechanism into the feature normalization computation graph. The engineering essence of this mechanism is to construct a multimodal information bottleneck, forcing the system to perform extreme physical compression of the visual tensor while preserving task-related information. The system defines the latent variable tensor after feature compression in the video memory. And an optimization objective including a length penalty term was constructed:

[0145]

[0146] In actual code and operator implementation, mutual information The calculation is approximated as the Kulbeck-Leibler divergence between the posterior distribution and the prior normal distribution of the latent variable tensor, while the penalty coefficient... As a dynamic Lagrange multiplier, it adaptively increases as the training cycle progresses. This hard constraint forces the underlying perceptual network to forget those visually redundant bits that cannot be converted into effective text symbols, ensuring that every visual token that is ultimately fed into the context window of the large language model is highly condensed and pure.

[0147] In the specific implementation of the joint fine-tuning phase of the monolithic architecture, the system's core scheduler performs tensor-level summation and fusion of the aforementioned information-theoretic alignment loss and the native autoregressive generation loss of the large language model. The overall loss function computation node of the system is defined in the computation graph as follows:

[0148]

[0149] The generation loss term is calculated based on the cross-entropy between the plain text instruction and the standard answer, while the weight parameters are used to balance the gradient magnitude during the multi-objective optimization process. Through this underlying architecture design of multi-objective joint optimization, the system achieves resonance between perceptual and cognitive parameters during the gradient update phase. This deeply coupled engineering implementation not only blocks the controlled associative bias of the language model in long-term reasoning tasks at the source, but also enables the system to exhibit an extremely low hallucination rate and extremely high logical rigor when handling highly difficult tasks such as answering questions about extremely long and complex movies and precise monitoring and tracing.

[0150] Based on the aforementioned system's underlying architecture configuration, adaptive retrieval mechanism, and cross-modal information theory alignment constraints, the specific implementation of this invention further provides a complete method for processing and fusing single-unit long-sequence multimodal data. This method defines the complete forward inference pipeline of the system in a practical application environment and elaborates in detail the end-to-end transformation process of multidimensional heterogeneous data from physical input to final logical text output.

[0151] In actual inference tasks, the system's first processing step is the dynamic truncation and initial feature screening of ultra-long temporal physical inputs. When the system receives a multimodal query task consisting of a long video stream and complex natural language instructions, the underlying controller does not load the complete video frame sequence into the core memory. Instead, it first triggers the long-sequence temporal adaptive retrieval module. In this stage, the system uses lightweight operators to traverse the video stream in real time and calculates the spatiotemporal information density tensor within a continuous time window. Based on the calculated density distribution curve, the system applies a nonlinear dynamic sampling rate function to hard-truncate low-information-density background redundant frames and densely sample high-information-density key action intervals. After this step, the original physical video stream, which can last for several hours, is adaptively compressed into a short, highly condensed microscopic keyframe sequence, fundamentally eliminating the risk of context token explosion faced by subsequent deep networks.

[0152] After initial physical compression, the system enters the physical isomorphism stage between endogenous multimodal perception and the plain text space. The preserved microscopic keyframe tensor sequence is fed into the underlying network of the monolithic large language model. In specific multimodal hybrid expert layers, the system uses a dynamic routing algorithm with modality bias terms to accurately distribute different visual feature blocks to the endogenous visual expert network best suited for processing the frequency of that space. In the forward propagation computation graph of the expert network, the system invokes an improved root mean square layer normalization operator to perform forced scaling and distribution recalibration of the visual feature tensors based on the vocabulary variance of the pre-trained language model. This physical operation completely breaks down the representational barriers of cross-domain data transmission, allowing discrete visual pixels to be smoothly mapped into continuous endogenous visual representation tensors.

[0153] Subsequently, the system constructs a temporal feature index tree coupling macro and micro dimensions in the video memory and initiates context-aware sparse retrieval based on information theory constraints. The system's text encoding layer first transforms the user's natural language instructions into a high-dimensional query tensor. Next, the system performs cosine similarity retrieval with a time decay bias in the temporal feature index tree to accurately locate local visual feature blocks highly relevant to the current query intent. More importantly, these retrieved visual features must undergo secondary filtering through information bottlenecks before entering the core decoder. By minimizing the strict constraint of the description length objective function, the system forcibly removes all redundant noise in the visual tensor that cannot provide text prediction value. At this point, the multimodal features fed into the language model's context window not only achieve extreme compression in the physical dimension but also achieve absolute cross-modal mutual information maximization with the instruction text in terms of the mathematical manifold.

[0154] After aligning and cleaning all features, the final step of the system is to initiate deep autoregressive logic decoding of the monolithic large model. The filtered and isomorphic clean multimodal context sequences are concatenated into a unified input tensor and fed into the deep Transformer block. For any time during the decoding phase... The system calculates the conditional probability distribution of the next target word through a fully connected output layer and a normalized exponential function. Its forward generation formula is strictly defined as follows:

[0155]

[0156] In this formula, Representing the large language model Layer at time The output is a high-dimensional hidden state tensor. This represents the vocabulary projection matrix of the language model. The system employs a beam search or kernel sampling decoding algorithm to sequentially generate target lexical units through autoregression until the output terminator is reached. Since the input multimodal context has achieved lossless alignment in both the underlying physical and mathematical dimensions, this autoregressive generation process effectively immunizes against controlled association bias caused by the lack of visual evidence, resulting in a natural language response with extremely high factual accuracy and logical rigor.

[0157] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0158] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A system for processing and fusing single-unit long-sequence multimodal data, characterized in that, The system includes: The long sequence temporal adaptive retrieval module is used to receive ultra-long video streams, audio streams and text commands in parallel, calculate the spatiotemporal information density of the multimodal data stream in real time, perform dynamic adaptive sampling based on the spatiotemporal information density, physically truncate redundant segments with low information density, densely sample temporal intervals with high information density, and output compressed key frame feature sequences. The endogenous multimodal perception module is embedded in the underlying Transformer network of the monolithic large language model. It incorporates a multimodal hybrid expert MoE network, a gated routing network with modality bias, and an RMS variance recalibration operator. It receives compressed keyframe feature sequences, distributes audiovisual features to matching endogenous visual experts and text experts through gated dynamic routing, and aligns the distribution and magnitude of visual features to the text word embedding benchmark through RMS variance recalibration, outputting a multimodal endogenous representation in a unified parameter space. The cross-modal information theory alignment module constructs a hierarchical temporal index tree coupled with macro-level segments and micro-level keyframes. After encoding text instructions into query vectors, it performs sparse similarity retrieval with temporal decay bias to filter core visual features that match the query. Irrelevant visual noise is filtered out by dual information theory constraints of maximizing mutual information MIM loss and minimizing description length MDL loss. The aligned and purified multimodal features are then fed into a single large language model decoder, which outputs natural language or structured semantic results via autoregression.

2. The single-unit long sequence multimodal data processing and feature fusion system according to claim 1, characterized in that, The formula for calculating the spatiotemporal information density by the long-sequence time-adaptive retrieval module is as follows: ； Representative extracted from the first Spatial tensor of each feature channel This represents the Shannon entropy calculated from the pixel grayscale or color histogram distribution. and These are the hyperparameters that are fixed in the system during the pre-training phase; The dynamic sampling rate cutoff function is: ； Typically, the extraction rate is set based on the maximum memory capacity of a single GPU, which determines the number of key feature blocks extracted per second. It may extract as few background features as per second, where γ is the density scaling factor.

3. The single-unit long sequence multimodal data processing and feature fusion system according to claim 1, characterized in that, The hybrid expert dynamic routing output formula of the endogenous multimodal perception module is as follows: ； Gating weight calculation formula: ； W bias For modal bias terms; Routing load balancing loss: ； Visual feature RMS recalibration alignment formula: ⊙γ; x v ϵ represents the original visual features, γ is the scaling factor, and ϵ is the numerically stable minimum value. Accompanying two-stage endogenous visual pre-training loss: 。 4. The single-unit long sequence multimodal data processing and feature fusion system according to claim 1, characterized in that, The time-aware sparse retrieval similarity formula for the cross-modal information theory alignment module is as follows: ； Mt emporal (i) is the temporal decay bias matrix, which applies a similarity penalty to long-distance temporal features; Mutual information maximization loss: ； Minimum description length information bottleneck loss: ； The model jointly fine-tunes the total loss function: ； L LLMGen For the autoregressive generation loss of large language models, w 1、 w2 is the multi-objective loss balancing weight.

5. A method for processing and fusing features of single-unit long sequence multimodal data, characterized in that, The system, as described in any one of claims 1-4, comprises the following steps: S101: Parallel reception of ultra-long video streams, audio streams and user text commands; long sequence time-adaptive retrieval module calculates spatiotemporal information density frame by frame in real time. S102: Perform dynamic adaptive sampling based on spatiotemporal information density, physically truncate low-information-density background frames, densely sample high-information-density temporal intervals, and output compressed keyframe sequences. S103: Input the keyframe into the endogenous multimodal perception module, and complete the spatial physical isomorphism between visual representation and text representation through the RMS variance recalibration operator; S104: The dynamic routing network with modal bias distributes audiovisual features to corresponding endogenous visual experts and text experts, and the residual connection fuses and outputs a unified multimodal endogenous representation. S105: Construct a hierarchical temporal index tree of macro segments + micro key frames, encode text instructions into query vectors, and perform sparse similarity retrieval with temporal decay bias to filter core visual features; Irrelevant visual noise is filtered out by dual information theory constraints of maximizing mutual information and minimizing description length. S106: The aligned and purified clean multimodal context is fed into the monolithic large model decoder, and the autoregressive decoder outputs time-aligned, logically unillusioned natural language or structured analysis text.

6. The method for processing and feature fusion of single-unit long sequence multimodal data according to claim 5, characterized in that, The dynamic adaptive sampling in step S102 includes: when the spatiotemporal information density ρ(t) is lower than a set threshold, physically truncating and discarding the corresponding frame of the current video segment; when ρ(t) is higher than the set threshold, increasing the sampling rate to extract key frames at high density, and limiting the upper and lower bounds of the sampling quantity to avoid token explosion.

7. The method for processing and fusing single-unit long sequence multimodal data according to claim 5, characterized in that, The hierarchical temporal index tree construction method in step S105 is as follows: macroscopic time segments are divided according to the peak of spatiotemporal information density, and each segment generates a macroscopic index vector through global average pooling; all microscopic key frame features obtained by adaptive sampling are stored inside the segment. During retrieval, the macroscopic segment index is matched first, and then the internal microscopic key frames are retrieved to reduce the retrieval computation.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the single-unit long sequence multimodal data processing and feature fusion method as described in claim 5.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the single-unit long sequence multimodal data processing and feature fusion method as described in claim 5.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the single-unit long sequence multimodal data processing and feature fusion system as described in any one of claims 1-4.

Images