Lightweight multi-modal content identification system based on double-track migration framework

Abstract

The invention discloses a lightweight multi-modal content recognition system based on a double-track migration framework, and relates to the technical field of content recognition, and the system comprises a data collection module which is used for synchronously collecting multi-source content of a text and an image and carrying out standardization processing and tensor construction to form a fusion tensor X; and the model construction module is used for inputting the fusion tensor into a dual-track migration structure constructed based on a Transform backbone network, and the dual-track migration structure realizes task semantic alignment and structure migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs an intermediate representation of modal alignment. According to the method, the training and deployment cost of multi-modal content recognition is remarkably reduced, the semantic expression ability in the modal is enhanced, the cross-modal alignment precision and fusion depth are effectively improved, and the perception and recognition ability of the model to the complex semantic relationship is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of content recognition technology, and in particular to a lightweight multimodal content recognition system based on a dual-track migration framework. Background Art

[0002] In recent years, with the rapid development of social media, news, and multimedia platforms, the combination of images and text has become a mainstream form of information dissemination, prompting increasing attention to multimodal content recognition technology. Multimodal content recognition aims to semantically understand, classify, and annotate composite content consisting of images and text. Its core lies in achieving effective information fusion and semantic consistency modeling across modalities. Traditional methods rely on joint embedding spaces or attention mechanisms for image and text alignment and recognition, often using large-scale trained deep network models (such as CLIP and ViLT) to improve cross-modal recognition accuracy. However, when transferring such models to new tasks or specific scenarios, they often require full parameter fine-tuning, resulting in high training costs and high deployment resource consumption, making them unsuitable for lightweight scenarios and edge computing platforms. Furthermore, existing models lack robust mechanisms to handle temporal and spatial feature variations in image modalities and vocabulary omissions (such as out-of-view (OOV) words) in text modalities, resulting in unstable cross-modal semantic alignment and affecting recognition accuracy.

[0003] Existing multimodal content recognition methods use only a single strategy (such as Adapter or Prompt), making it difficult to simultaneously address both structural transfer and semantic alignment requirements, resulting in limited fusion effectiveness. In multimodal scenarios, they fail to enhance modeling of modal differences between images and text, lack decoupled extraction and difference compensation mechanisms, and typically employ additive attention or serial embedding, making it difficult to capture complex non-additive interactions and thus inhibiting the expressive power of the fused representation. Therefore, there is an urgent need to build a multimodal content recognition system that supports flexible transfer under parameter freezing, modality-specific enhancement, and non-additive interaction modeling, balancing lightweightness, transferability, and recognition performance. Summary of the Invention

[0004] In view of the problems existing in the existing lightweight multimodal content recognition system based on the dual-track migration framework, the present invention is proposed.

[0005] Therefore, the present invention provides a lightweight multimodal content recognition system based on a dual-track migration framework to solve the urgent need to build a multimodal content recognition system that can support flexible migration under parameter freezing, modality-specific enhancement and non-additive interaction modeling, so as to take into account lightweight, transferable and recognition performance.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a lightweight multimodal content recognition system based on a dual-track migration framework, which includes: The data acquisition module is used to synchronously collect multi-source content of text and images, perform standardization processing and tensor construction, and form a fused tensor X; The model construction module is used to input the fused tensor into a dual-track migration structure built on the Transformer backbone network. The dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs a modality-aligned intermediate representation; The data augmentation module is used to extract spatial and temporal features from the intermediate representation output by the dual-track migration structure in the image modality and fuse them into an enhanced image representation. In the text modality, it extracts contextual embeddings for existing OOV terms to generate semantic completion prompt vectors and replaces the original OOV word vectors to obtain an enhanced text representation. The data fusion module is used to fuse the enhanced image and text representations and construct a joint prompt vector, guide the non-additive interaction between text and image modalities, complete multimodal fusion through the residual attention mechanism, and generate fused alignment features; The recognition module is used to input the fused alignment features into the multi-task classifier, select Sigmoid or Softmax output according to the task attributes, and optimize the model with the cross entropy loss function to complete multi-label or single-label recognition.

[0007] As a preferred solution of the lightweight multimodal content recognition system method based on the dual-track migration framework described in the present invention, the dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and the intermediate representation of the output modality alignment includes: Extract the task type from the task context, encode the task instruction text through the pre-trained language model to form a task semantic embedding vector, and copy the task semantic embedding vector l times to form the initial prompt matrix o; Convert the initial prompt matrix o into a learnable parameter matrix P, which serves as the core guidance variable of the Prompt Learning channel. Use the Xavier uniform initialization method to initialize all elements of the matrix P. Insert the matrix P in the Prompt Learning channel into the front of the modality fusion tensor X along the time dimension to form an extended sequence B. Input B into the frozen Transformer backbone, define the three sets of linear transformation matrices in the attention mechanism and construct three types of tensors. Based on the constructed three types of tensors, perform Prompt-guided attention weight calculation; The fused Prompt context representation is obtained by weighted aggregation of the value tensor through the attention weight A. , replace the corresponding matrix P in the original extended sequence B with the fused Prompt context representation, keep the modal content part unchanged, and recombine it into the updated input sequence B'; The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operations as a feature encoder. After the feedforward submodule of each encoder layer in the Transformer backbone, a lightweight adapter module is inserted, and the bottleneck dimension of the adapter is set. For each layer in the Transformer encoder, an adapter module is attached to the standard residual path; Perform residual fusion on the Adapter module output and the original Transformer backbone output, maintaining the backbone features unchanged while adding compensatory perturbations for modal differences; The final output tensor after all adapter enhancements and residual guidance is recorded as B''; The generated final output tensor B'' is input into the Transformer backbone structure with the Adapter module inserted, freezing all parameters in the Transformer backbone and explicitly retaining only the trainable parameters involved in the Prompt guidance path and the Adapter compensation path as optimization targets; After the parameter freezing strategy is set, the Adam optimizer is used for iterative updates. The loss function is set according to the type of downstream task, and the Prompt and Adapter related parameters are gradient updated to optimize the model's expressiveness and generalization capabilities on the target task. During training, all frozen parts remain unchanged, and only the parameters within the Prompt and Adapter paths are updated. The fused sequence is input into the frozen backbone network, and a cross-modal fusion representation, namely a modality-aligned intermediate representation, is generated under the synergistic effect of Prompt intervention and Adapter regulation.

[0008] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, wherein: the decoupling and extraction of spatial and temporal features in the image modality and fusing them into an image enhancement representation refers to receiving the image modality vector in the modality-aligned intermediate representation as input; Perform spatial structure division on each frame of image, divide the image into several fixed-size blocks, perform linear embedding and position encoding on each block, and convert it into a sequence of spatial feature vectors; A sliding window strategy is used to perform local sampling on the image frame sequence to construct image subsequences of short time periods. The time period modeling network is used to extract the change information between consecutive frames of each subsequence and generate a sequence of temporal feature vectors. The extracted spatial feature vector sequence and temporal feature vector sequence are spliced ​​together to form a complete image spatiotemporal feature set. A fusion mapping matrix is ​​constructed to perform unified mapping operations on the spatial and temporal features to generate an enhanced representation of the image modality.

[0009] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, wherein: the context embedding is extracted for the existing OOV terms in the text modality to generate a semantic completion Prompt vector and replace the original OOV word vector to obtain the enhanced text representation, which means checking the visibility of each term in the text sequence in the vocabulary in the corpus, marking the unrecognizable terms as OOV, and intercepting the context sequence of a fixed window length before and after each identified OOV position. The construction form is Context fragment of , where and They represent the semantic context on the left and right sides of the OOV word respectively, and the [MASK] placeholder is used to guide the language model to generate context-dependent semantic embeddings; The constructed context fragment is input into the pre-trained language model to output the hidden state vector at the [MASK] position, which is recorded as And as the semantic completion process representation of the OOV term, the representation vector is used to replace the initial embedding of the corresponding OOV position in the original text vector to form an updated text modality embedding sequence F, that is, the enhanced text representation.

[0010] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, wherein: the fusing of the enhanced image and text representations and constructing a joint prompt vector refers to splicing the enhanced image representation and the enhanced text representation in the sequence dimension to form a unified fused sequence representation; Based on the text positions of the identified OOV terms, a set of position masks are constructed to focus on the key context in the text modality in the fused sequence. An attention pooling mechanism is applied to perform weighted aggregation on the feature representations of each embedded position in the fused sequence, where the attention weights are guided by the OOV position mask to highlight the contextual information related to OOV, and a joint prompt guidance vector is output.

[0011] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, wherein: the non-additive interaction between the guiding text and image modalities is completed by the residual attention mechanism to achieve multimodal fusion, and the generation of fused alignment features includes: Taking the fused sequence as input, it performs a multi-head self-attention operation within the modality, extracts the self-preserving semantic structure of the internal features of each modality, and outputs the intra-modal attention representation; Based on the joint Prompt vector as the central query vector, a cross-modal interaction attention mechanism is constructed to extend the interaction result to the same length as the original sequence to form a cross-feature representation; Perform residual fusion of the intra-modality self-attention representation and the cross-feature representation to generate a residual interaction representation; Standard normalization and feedforward network processing are performed on the residual interaction representation to obtain a fused expression as the fused alignment feature.

[0012] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, wherein: the fused alignment features are input into the multi-task classifier, Sigmoid or Softmax output is selected according to the task attributes, and the model is optimized with the cross-entropy loss function to complete multi-label or single-label recognition. It means that the fused alignment features are input into the multi-task classifier, and the multi-task classifier divides different recognition tasks into single-label or multi-label categories according to the task attributes, and configures an independent output branch for each task. If it is a multi-label task, the Sigmoid activation function is used to output the probability class by class, and if it is a single-label task, the Softmax function is used to generate the normalized category distribution probability; Perform forward reasoning on each task to obtain a set of prediction results, which are compared with the true labels. The error of each task is calculated separately based on the cross-entropy loss function. Binary cross entropy is used for multi-label tasks, and category cross entropy is used for single-label tasks. All task losses are combined into a total loss according to preset weights, and used for backpropagation to optimize model parameters.

[0013] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, the synchronous acquisition of multi-source content of text and images refers to using the frame timestamp t as the minimum sampling granularity, continuously extracting frame data from the content source as image modal data at a fixed frame rate through the RGB camera module, and extracting text modal data from the subtitles, labels and explanatory text paired with the image.

[0014] As an optimal solution for the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, the standardization processing and tensor construction refer to performing mean-variance normalization on the image modality data, converting the original pixel values ​​into a standardized image tensor, selecting the WordPiece word segmenter to perform subword splitting on the original text sequence and construct a word segmentation sequence, obtaining the vector representation of each subword through a pre-trained word embedding model and performing normalization processing, and constructing a unified text input tensor.

[0015] As a preferred solution of the lightweight multimodal content recognition system based on the dual-track migration framework described in the present invention, the formation of the fusion tensor X refers to mapping the feature tensors of the image modality and the text modality to a unified dimensional space, combining the unified tensors of the image modality and the text modality, and generating the fusion tensor X.

[0016] The beneficial effects of the present invention are as follows: by constructing a dual-track migration structure, the present invention effectively realizes task adaptation and structure migration under the condition of frozen backbone model parameters, significantly reduces the training and deployment costs of multimodal content recognition, and is suitable for lightweight and edge computing scenarios. It introduces image spatiotemporal feature decoupling and text OOV word semantic completion mechanism, enhances the semantic expression ability within the modality, and improves the robustness of the model to unstructured, missing or heterogeneous information. Further, through the non-additive interactive fusion guided by the joint prompt, it effectively improves the cross-modal alignment accuracy and fusion depth, and enhances the model's perception and recognition ability of complex semantic relationships. The overall solution takes into account recognition accuracy, model lightweight and application flexibility while ensuring efficient migration capabilities, and is significantly superior to traditional single migration or additive fusion methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 Schematic diagram of the data enhancement process in Example 1; Figure 2 Schematic diagram of the structure of the lightweight multimodal content recognition system based on the dual-track migration framework in Example 1. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0022] Example 1, with reference to Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a lightweight multimodal content recognition system based on a dual-track migration framework. The lightweight multimodal content recognition system based on the dual-track migration framework includes the following steps: S1, data acquisition module, is used to synchronously collect multi-source content of text and images and perform standardization processing and tensor construction to form a fused tensor X; Specifically, to achieve efficient fusion and alignment of the multimodal content recognition system, this step uses a frame-level synchronous acquisition mechanism to acquire image and text modal data. This solution selects the fusion of image and text modalities as the optimal path. The reason is that the image modality has the advantage of spatial perception, and the text modality provides semantic constraints. The two modalities are highly complementary, which can achieve better recognition results with a lightweight model structure. Using the frame timestamp t as the minimum sampling granularity, the RGB camera module continuously extracts frame data from the content source (such as video frames or captured image sequences) at a fixed frame rate as image modality data, and extracts text modality data from the subtitles, labels, and explanatory text paired with the image; Perform mean-variance normalization on image modal data, converting raw pixel values ​​into standardized image tensors. Use the WordPiece tokenizer to perform subword splitting on the original text sequence and construct a word segmentation sequence. Use a pre-trained word embedding model (such as BERT embedding) to obtain the vector representation of each subword and perform normalization to construct a unified text input tensor. The feature tensors of the image modality and text modality are mapped to a unified dimensional space. That is, by multiplying the image and text tensors by their respective linear projection matrices, the image feature representation and text feature representation in a unified dimension are obtained. The unified tensors of the image modality and text modality are combined to generate a fused tensor X.

[0023] Through frame-level synchronous acquisition, the modal temporal consistency is enhanced, and the model's adaptability to real scenarios is improved; through modal normalization and subword embedding strategies, the model's ability to capture local image differences and text semantic details is improved; through unified mapping to generate fusion tensors, a more efficient information expression carrier is provided for the lightweight model structure, effectively reducing the computational burden of the fusion stage; this strategy has good scalability and portability, and is suitable for various application scenarios such as social media content analysis, short video understanding, and cross-modal retrieval. It is a highly adaptable solution that takes into account both performance and resource constraints.

[0024] S2, a model building module, is used to input the fused tensor into a dual-track migration structure built on the Transformer backbone network. The dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through prompt learning embedding and adapter tuning insertion, and outputs a modality-aligned intermediate representation; Specifically, the dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion. The intermediate representation of the output modality alignment includes: Extract the task type from the task context (e.g., the input instruction "identify human actions in an image and match them to text descriptions"), encode the task instruction text through a pre-trained language model (e.g., BERT or T5 encoder) to form a task semantic embedding vector, and replicate the task semantic embedding vector l times to form the initial prompt matrix o. For example, if the prompt sequence length to guide the model is set to 8, the task semantic embedding vector is replicated 8 times continuously to form a stacked structure in which each row has exactly the same content, namely, the same task semantic embedding vector. This operation ensures that the prompt content is directly related to the current recognition task and has a learnable semantic guidance starting point rather than arbitrary initialization; The initial prompt matrix o is converted into a learnable parameter matrix P, which is registered as the nn.Parameter type in the neural network and serves as the core guidance variable of the Prompt Learning channel. To avoid convergence abnormalities caused by excessively large or small initial weight distribution, all elements of the matrix P are initialized using the Xavier uniform initialization method. To ensure that the Prompt Learning pipeline can independently adapt to new tasks, freeze the Transformer backbone parameters (including attention weights and feedforward layer parameters). Only the matrix P in the Prompt Learning pipeline is allowed to participate in backpropagation updates. In the optimizer (such as AdamW), specify P as the only optimizable variable. Insert the matrix P in the Prompt Learning channel into the front of the modal fusion tensor X according to the time dimension to form an extended sequence B, ensuring that the Prompt vector is at the front of the sequence in the Transformer. It naturally has a larger information coverage in the self-attention mechanism. Through self-attention calculation, Prompt can "pay attention" to all subsequent modal contents and adjust its representation through backpropagation. Input B into the frozen Transformer backbone to define the three sets of linear transformation matrices in the attention mechanism. 、 and ,in 、 and They are the linear transformation matrices (projection matrices) of query / key / value respectively, and construct three types of tensors, including constructing the query tensor based on the Prompt vector in the matrix P:

[0025] Where, It is the intercepted sub-tensor of the Prompt part, which refers to the front of B. rows, keep all columns, Q is the query tensor; Construct key and value tensors based on the prompt vector and modality fusion tensor X:

[0026] Where K is the key tensor and V is the value tensor; Based on the three types of tensors constructed, Prompt-guided attention weight calculation is performed:

[0027] Where, Represents the hidden dimension of each attention head in the Transformer, also known as the projection dimension; The fused Prompt context representation is obtained by weighted aggregation of the value tensor through the attention weight A. :

[0028] The fused Prompt contextual representation replaces the corresponding matrix P in the original extended sequence B, keeping the modal content unchanged. The resulting sequence is reassembled into the updated input sequence B'. This new sequence serves as the input to the next layer of the Transformer backbone structure, promoting the further propagation and interaction of Prompt-guided information in deeper encoding structures, thereby achieving continuous guidance and enhancement of cross-modal information. This completes the first round of contextual information injection, guided by Prompt and based on modal content, and establishes a unified semantic input foundation for subsequent adaptation modules (such as the Adapter structure). The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operations as a feature encoder. After the Feed Forward Network (FFN) of each Encoder Layer in the Transformer backbone, a lightweight Adapter module is inserted, and the bottleneck dimension of the Adapter is set to , , for each layer in the Transformer Encoder, the following Adapter module is attached to the standard residual path:

[0029] Where Adapter(x) is the output vector after lightweight perturbation compensation processing of the input vector x, ReLU() is a nonlinear activation function, is the down-projection matrix, which restores the compressed features to their original dimensions. is the up-projection matrix, which compresses high-dimensional features into low-dimensional representations, and x is the feature vector at a certain position in the current Transformer layer; The Adapter module output is fused with the original Transformer backbone output to perform residual fusion, while maintaining the backbone features unchanged, adding compensatory perturbations for modal differences:

[0030] Where, It is the final output vector after processing by the Adapter module, which is the result of adding Adapter(x) as a perturbation term to the original feature through the residual connection; The final output tensor after all adapter enhancements and residual guidance is denoted as B''. The structure of this tensor retains the prompt guidance information, modal time structure and feature dimension consistency. The generated final output tensor B'' is input into the Transformer backbone structure with the Adapter module inserted. At this time, the Transformer backbone contains the Prompt guidance path and the Adapter compensation path. The Prompt vector guides the attention mechanism to focus on important positions of the modality, and the Adapter module is used to compensate for feature perturbations between layers. To avoid damaging the original pre-trained model's capabilities in transfer tasks with limited samples or resources, all parameters in the Transformer backbone are frozen, including the query, key, value, and output matrices in the multi-head attention layer, all weight matrices in the feedforward layer, LayerNorm normalization parameters, and all positional encodings and embedding weights. This operation ensures that the backbone remains unchanged, and adaptation to the target task is achieved only by fine-tuning some lightweight modules. Based on parameter freezing, only the trainable parameters involved in the Prompt guidance path and the Adapter compensation path are explicitly retained as optimization targets. Specifically, these include the learnable Prompt vector matrix and the linear transformation matrices for the upper and lower projections in each layer of the Adapter module. This lightweight training configuration significantly compresses the parameter space, ensuring good system adaptability and training efficiency. After the parameter freezing strategy is set, the Adam optimizer is used for iterative updates. The loss function is set according to the type of downstream task (such as cross entropy for classification tasks and mean squared error for regression tasks). The Prompt and Adapter related parameters are gradient updated to optimize the model's expressiveness and generalization capabilities for the target task. During training, all frozen parts remain unchanged, and only the parameters within the Prompt and Adapter paths are updated. This guides the network to gradually adapt to the new modal combination structure while retaining its original representation capabilities. The fused sequence is input into the frozen backbone network, and a cross-modal fusion representation is generated under the synergistic effect of Prompt intervention and Adapter adjustment. This representation not only retains the original structural advantages of the Transformer backbone, but also integrates multimodal residual compensation information. It serves as the input of the subsequent modality alignment module, thus completing the complete migration path from the fusion input structure to the lightweight optimization configuration. The output result is a modality-aligned intermediate representation with task adaptability.

[0031] The proposed dual-track migration framework effectively improves the accuracy, efficiency and adaptability of multimodal content recognition tasks by synergistically integrating two lightweight migration mechanisms, Prompt Learning and Adapter-Tuning. The Learning module introduces a learnable task guidance matrix to embed task semantics into the input sequence in a structured manner, enhancing the model's ability to focus attention on semantically related modal regions and avoiding the generalization issues that plague traditional methods in small sample sizes. Furthermore, the Adapter-Tuning module introduces a residual perturbation adjustment channel into each Transformer layer. This fine-tunes modal features through low-dimensional up-down projections and nonlinear activations, significantly reducing parameter overhead while retaining the original expressive power and enhancing the model's stability in modality alignment and structure preservation. Furthermore, the scheme introduces a non-additive interaction mechanism and a residual attention optimization strategy to significantly improve the quality of information fusion between image and text modalities, making the fused representation more discriminative and consistent. The modality-aligned intermediate representation generated by this mechanism not only balances task semantics and modality characteristics but also supports multi-task outputs for both multi-label and single-label recognition, demonstrating high versatility. Overall, this invention achieves parameter compression, improved training efficiency, and enhanced multimodal feature representation without sacrificing recognition performance. This provides an efficient, low-cost, and easily deployable multimodal content recognition solution for real-world scenarios with limited computing resources or frequent task switching.

[0032] S3, the data augmentation module, is used to decouple and extract spatial and temporal features from the intermediate representation output by the dual-track migration structure in the image modality and fuse them into an enhanced image representation. In the text modality, it extracts contextual embeddings for existing OOV (Out Of Vocabulary) terms to generate semantic completion prompt vectors and replaces the original OOV word vectors to obtain an enhanced text representation. Specifically, decoupling and extracting spatial and temporal features from the image modality and fusing them into an enhanced image representation means receiving the image modality vector in the modality-aligned intermediate representation as input. This representation has embedded semantic information through the Prompt and Adapter paths and has alignment features for the text modality, thus serving as a unified input basis for subsequent image structure analysis. A spatial structure partitioning operation is performed on each frame of the image, dividing the image into several fixed-size blocks. Each block is linearly embedded and position-encoded, converting it into a sequence of spatial feature vectors. By introducing position encoding, the model can extract local visual information while maintaining the recognizability of spatial order, thereby enhancing the modeling ability of static image structures. A sliding window strategy is used to locally sample the image frame sequence to construct short-term image subsequences. A time segment modeling network (such as TSN) is used to extract the change information between consecutive frames in each subsequence, generating a temporal feature vector sequence to supplement the image's expressive ability in terms of action, transition, and contextual changes. The extracted spatial feature vector sequence and temporal feature vector sequence are spliced ​​to form a complete image spatiotemporal feature set, a fusion mapping matrix is ​​constructed, and a unified mapping operation is performed on the spatial and temporal features to generate an enhanced representation of the image modality, including splicing the spatial feature vectors and temporal feature vectors extracted from the image modality in the feature dimension direction to form a joint representation of dimension (n+k)×2e, where n is the number of spatial blocks obtained after the image modality is segmented, k is the number of time frames extracted by sliding sampling, and e is the embedding dimension of each feature vector. Through a linear transformation matrix, the spliced ​​features are mapped to a unified representation of dimension e, thereby realizing the fusion of spatial-temporal features and outputting an enhanced image representation to ensure that it contains both the composition information of the static structure and the dynamic features of the time dimension.

[0033] It achieves decoupled reconstruction and semantic completion of modal alignment results, effectively improving the expression accuracy and structural perception ability of image modality in complex multimodal tasks.

[0034] First, through separate modeling and unified mapping of spatial and temporal dimensions, the image modality's joint perception ability of static and dynamic features is significantly enhanced, and its expression robustness in complex contexts is improved. Second, the introduction of position encoding and temporal modeling mechanisms enables the model to have the ability to reason about cross-frame structural relationships and identify changing trends, and perform better in multimodal tasks such as character action understanding and scene change recognition. Third, this enhanced representation, as the input of the subsequent multimodal interaction module, not only retains the image's own structural hierarchical information, but also achieves high-quality fusion with the text modality through semantic alignment, effectively improving the accuracy of the final content recognition and context consistency.

[0035] Furthermore, for the existing OOV terms in the text modality, the context embedding is extracted to generate a semantic completion Prompt vector and replace the original OOV word vector to obtain the enhanced text representation. This means that the visibility of each term in the text sequence is checked in the vocabulary in the corpus, unrecognizable terms are marked as OOV, and the context sequence of a fixed window length before and after each identified OOV position is intercepted. The construction form is Context fragment of , where and They represent the semantic context on the left and right sides of the OOV word respectively, and the [MASK] placeholder is used to guide the language model to generate context-dependent semantic embeddings; Input the constructed context fragment into the pre-trained language model (such as BERT) and output the hidden state vector at the [MASK] position, denoted as And as the semantic completion process representation of the OOV term, the representation vector is used to replace the initial embedding of the corresponding OOV position in the original text vector to form an updated text modality embedding sequence F, that is, the enhanced text representation.

[0036] By introducing a context-driven OOV word semantic completion mechanism, the representation completeness and semantic accuracy of text modalities in multimodal content recognition tasks are significantly improved. To address the OOV problem caused by the inability of traditional vocabularies to cover domain-specific words or spelling variants, this solution constructs context fragments containing [MASK] placeholders to guide pre-trained language models (such as BERT) to generate dynamic semantic vectors in the current context. These vectors are then used to replace the original OOV word position embeddings, thereby restoring and enhancing semantic information. This approach not only ensures the structural continuity of text embedding sequences but also makes semantic representations more context-sensitive, improving the understanding of polysemous and ambiguous terms. During the multimodal feature alignment and fusion process, the enhanced text representations form more accurate semantic pairings with image modal features, reducing semantic drift and alignment errors between modalities. Furthermore, because this strategy is free from the constraints of a static vocabulary, it possesses excellent transfer adaptability and demonstrates greater robustness and adaptability in cold-start scenarios and vertical domain tasks, comprehensively promoting the stability and recognition accuracy of lightweight multimodal recognition systems.

[0037] S4, data fusion module, is used to fuse the enhanced image and text representations and construct a joint prompt vector, guide the non-additive interaction between text and image modalities, complete multimodal fusion through the residual attention mechanism, and generate fused alignment features; Specifically, fusing the enhanced image and text representations and constructing a joint prompt vector involves concatenating the enhanced image representation and the enhanced text representation in the sequence dimension to form a unified fused sequence representation. This fused sequence not only preserves the visual information and temporal dynamics in the image, but also incorporates the semantic completion content in the text, providing a structural foundation for cross-modal modeling. Based on the text positions of the identified OOV terms, a set of position masks is constructed to focus on the key context in the text modality in the fused sequence. An attention pooling mechanism is applied to perform weighted aggregation on the feature representations of each embedded position in the fused sequence. The attention weights are guided by the OOV position mask, highlighting the contextual information related to OOV. The joint prompt guidance vector is output, whose semantics focus reflects the comprehensive features related to the OOV terms in the fused modality. This joint prompt not only complements the missing semantics in the original text modality, but also carries background guidance information from the image modality, thereby improving the efficiency of information fusion in subsequent multimodal interactions. The fused sequence and the joint prompt vector are used as the input of the subsequent multimodal non-additive interaction module to achieve deep fusion representation between images and text.

[0038] By constructing a joint Prompt vector that fused and enhanced images and text, the problems of semantic separation between modalities and missing OOV word information in multimodal content recognition are effectively solved, and the depth and accuracy of image-text fusion are significantly improved. The construction of the fusion sequence retains the visual details of the image and the contextual semantics of the text in a unified representation space, laying a structural foundation for subsequent cross-modal interaction. The position mask mechanism focuses on the contextual area of ​​OOV words and combines the attention pooling strategy to weightedly aggregate key information. The generated joint Prompt not only fills the semantic gaps in the text modality, but also embeds background guidance information from the image modality, thereby achieving cross-modal completion and alignment of information. The joint Prompt and the fusion sequence are input into the non-additive interaction module together, which effectively guides the model to explore complex modal dependencies in the nonlinear expression space. Compared with the traditional weighted mechanism, it has stronger semantic penetration ability and fusion flexibility.

[0039] Furthermore, we guide the non-additive interaction between text and image modalities, complete multimodal fusion through the residual attention mechanism, and generate fused alignment features including: Taking the fused sequence as input, it performs a multi-head self-attention operation within the modality, extracts the self-preserving semantic structure of the internal features of each modality, and outputs the intra-modal attention representation; Based on the joint prompt vector as the central query vector, a cross-modal interactive attention mechanism is constructed. This mechanism uses the joint prompt vector as the query and the fusion sequence as the key and value. The calculated cross-attention output is used to capture the non-additive semantic dependency between the image and the text. To maintain the consistency of the features at each position, the interaction result is extended to the same length as the original sequence to form a cross-feature representation. Perform residual fusion of the intra-modality self-attention representation and the cross-feature representation to generate a residual interaction representation; Standard normalization and feedforward network processing are performed on the residual interaction representation to obtain a fused expression, which serves as the fusion alignment feature. Layer normalization improves numerical stability, and the feedforward network introduces nonlinear activation to improve the expression dimension and discriminative ability. This feature has the characteristics of modal consistency, prompt-guided semantic fusion, and residual information retention, completing the deep alignment and fusion of image and text modalities in a unified space.

[0040] By introducing a residual attention mechanism to achieve non-additive interactive fusion of image and text modalities, this approach significantly improves the consistency and expressiveness of multimodal representations. It addresses the information loss and insufficient alignment issues of traditional additive fusion when modal structures differ significantly. By constructing a joint prompt vector as the interaction center, this approach effectively guides cross-modal semantic fusion, enabling detailed modeling of non-additive dependencies between image and text, and enhancing the model's ability to perceive contextual associations and semantic cues. Furthermore, intra-modal multi-head self-attention maintains the semantic structure of each modality, while residual fusion balances modal consistency with interactive feature complementation, mitigating semantic drift and feature erosion during the fusion process. Furthermore, by combining layer normalization with feedforward neural network processing, this approach not only improves the numerical stability of features but also enhances nonlinear expressiveness and classification discriminative performance. The resulting fused, aligned features exhibit excellent modality consistency, contextual adaptability, and discriminative power, providing strong support for multi-task content recognition. While maintaining a lightweight model structure and adapting to real-time applications, the proposed model possesses significant practical value and potential for widespread adoption.

[0041] S5, recognition module, is used to input the fused alignment features into the multi-task classifier, select Sigmoid or Softmax output according to the task attributes, and optimize the model with the cross entropy loss function to complete multi-label or single-label recognition; Specifically, the fused alignment features are input into the multi-task classifier, Sigmoid or Softmax output is selected according to the task attributes, and the model is optimized with the cross entropy loss function to complete multi-label or single-label recognition. This means that the fused alignment features are input into the multi-task classifier (such as Multi-Task Learning). The multi-task classifier divides different recognition tasks into single-label or multi-label categories according to the task attributes, and configures independent output branches for each task. If it is a multi-label task, the Sigmoid activation function is used to output the probability class by class. If it is a single-label task, the Softmax function is used to generate the normalized category distribution probability; Forward reasoning is performed on each task to obtain a set of prediction results, which are compared with the true labels. The error of each task is calculated separately based on the cross-entropy loss function. Binary cross entropy is used for multi-label tasks, and category cross entropy is used for single-label tasks to ensure that the loss function matches the task attributes. All task losses are combined into a total loss according to preset weights, and used for backpropagation optimization of model parameters. The optimized model parameters refer to the joint backpropagation training of all unfrozen trainable parameters in the dual-track migration structure, residual attention mechanism and multi-task classifier to improve the performance of the entire multimodal content recognition system.

[0042] By analyzing the task attributes, the activation function and output structure are dynamically selected to ensure that the optimal recognition strategy is adopted for each type of task, thereby improving the model's generalization ability and adaptability to practical applications. Different cross-entropy loss functions are used to calculate errors for multi-label and single-label tasks, and they are merged into a total loss function according to the preset task importance weights. In multi-task training, the primary and secondary relationships are effectively controlled and the system adjustability is enhanced. Through backpropagation, all trainable parameters in the dual-track migration structure, residual attention mechanism and multi-task classifier are uniformly optimized and trained to ensure the co-evolution of modal representation learning and classification decision-making, thereby improving the overall recognition effect of the model. The classifier module design supports subsequent expansion to more subtasks, and the input of the fused representation enhances semantic consistency, which helps to maintain robust recognition capabilities in complex contexts.

[0043] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A lightweight multimodal content recognition system based on a dual-track migration framework, characterized by: include, The data acquisition module is used to synchronously collect multi-source content of text and images, perform standardization processing and tensor construction, and form a fused tensor X; The model construction module is used to input the fused tensor into a dual-track migration structure built on the Transformer backbone network. The dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs a modality-aligned intermediate representation; The data augmentation module is used to extract spatial and temporal features from the intermediate representation output by the dual-track migration structure in the image modality and fuse them into an enhanced image representation. In the text modality, it extracts contextual embeddings for existing OOV terms to generate semantic completion prompt vectors and replaces the original OOV word vectors to obtain an enhanced text representation. The data fusion module is used to fuse the enhanced image and text representations and construct a joint prompt vector, guide the non-additive interaction between text and image modalities, complete multimodal fusion through the residual attention mechanism, and generate fused alignment features; The recognition module is used to input the fused alignment features into the multi-task classifier, select Sigmoid or Softmax output according to the task attributes, and optimize the model with the cross entropy loss function to complete multi-label or single-label recognition.

2. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 1, characterized in that: The dual-track migration structure achieves task semantic alignment and structural migration under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion. The intermediate representation of the output modality alignment includes: Extract the task type from the task context, encode the task instruction text through the pre-trained language model to form a task semantic embedding vector, and copy the task semantic embedding vector l times to form the initial prompt matrix o; Convert the initial prompt matrix o into a learnable parameter matrix P, which serves as the core guidance variable of the Prompt Learning channel. Use the Xavier uniform initialization method to initialize all elements of the matrix P. Insert the matrix P in the Prompt Learning channel into the front of the modality fusion tensor X along the time dimension to form an extended sequence B. Input B into the frozen Transformer backbone, define the three sets of linear transformation matrices in the attention mechanism and construct three types of tensors. Based on the constructed three types of tensors, perform Prompt-guided attention weight calculation; The fused Prompt context representation is obtained by weighted aggregation of the value tensor through the attention weight A. , replace the corresponding matrix P in the original extended sequence B with the fused Prompt context representation, keep the modal content part unchanged, and recombine it into the updated input sequence B'; The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operations as a feature encoder. After the feedforward submodule of each encoder layer in the Transformer backbone, a lightweight adapter module is inserted, and the bottleneck dimension of the adapter is set. For each layer in the Transformer encoder, an adapter module is attached to the standard residual path; Perform residual fusion on the Adapter module output and the original Transformer backbone output, maintaining the backbone features unchanged while adding compensatory perturbations for modal differences; The final output tensor after all adapter enhancements and residual guidance is recorded as B''; The generated final output tensor B'' is input into the Transformer backbone structure with the Adapter module inserted, freezing all parameters in the Transformer backbone and explicitly retaining only the trainable parameters involved in the Prompt guidance path and the Adapter compensation path as optimization targets; After the parameter freezing strategy is set, the Adam optimizer is used for iterative updates. The loss function is set according to the type of downstream task, and the Prompt and Adapter related parameters are gradient updated to optimize the model's expressiveness and generalization capabilities on the target task. During training, all frozen parts remain unchanged, and only the parameters within the Prompt and Adapter paths are updated. The fused sequence is input into the frozen backbone network, and a cross-modal fusion representation, namely a modality-aligned intermediate representation, is generated under the synergistic effect of Prompt intervention and Adapter regulation.

3. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 2, characterized in that: Decoupling and extracting spatial and temporal features in the image modality and fusing them into an image enhancement representation refers to receiving the image modality vector in the modality aligned intermediate representation as input; Perform spatial structure division on each frame of image, divide the image into several fixed-size blocks, perform linear embedding and position encoding on each block, and convert it into a sequence of spatial feature vectors; A sliding window strategy is used to perform local sampling on the image frame sequence to construct image subsequences of short time periods. The time period modeling network is used to extract the change information between consecutive frames of each subsequence and generate a sequence of temporal feature vectors. The extracted spatial feature vector sequence and temporal feature vector sequence are spliced ​​together to form a complete image spatiotemporal feature set. A fusion mapping matrix is ​​constructed to perform unified mapping operations on the spatial and temporal features to generate an enhanced representation of the image modality.

4. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 3, characterized in that: The method extracts context embeddings for existing OOV terms in the text modality, generates semantic completion Prompt vectors, and replaces the original OOV word vectors to obtain enhanced text representation. This method checks the visibility of each term in the text sequence in the vocabulary in the corpus, marks unrecognizable terms as OOV, and intercepts the context sequence of a fixed window length before and after each recognized OOV position. The construction form is Context fragment of , where and They represent the semantic context on the left and right sides of the OOV word respectively, and the [MASK] placeholder is used to guide the language model to generate context-dependent semantic embeddings; The constructed context fragment is input into the pre-trained language model to output the hidden state vector at the [MASK] position, which is recorded as And as the semantic completion process representation of the OOV term, the hidden state vector is used to replace the initial embedding of the corresponding OOV position in the original text vector to form an updated text modality embedding sequence F, that is, the enhanced text representation.

5. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 4, characterized in that: The fusing of the enhanced image and text representations and constructing a joint Prompt vector refers to splicing the enhanced image representation and the enhanced text representation in the sequence dimension to form a unified fused sequence representation; Based on the text positions of the identified OOV terms, a set of position masks are constructed to focus on the key context in the text modality in the fused sequence. An attention pooling mechanism is applied to perform weighted aggregation on the feature representations of each embedded position in the fused sequence, where the attention weights are guided by the OOV position mask to highlight the contextual information related to OOV, and a joint prompt guidance vector is output.

6. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 5, characterized in that: The non-additive interaction between the guided text and image modalities is achieved through the residual attention mechanism to complete multimodal fusion and generate fused alignment features including: Taking the fused sequence as input, it performs a multi-head self-attention operation within the modality, extracts the self-preserving semantic structure of the internal features of each modality, and outputs the intra-modal attention representation; Based on the joint Prompt vector as the central query vector, a cross-modal interaction attention mechanism is constructed to extend the interaction result to the same length as the original sequence to form a cross-feature representation; Perform residual fusion of the intra-modality self-attention representation and the cross-feature representation to generate a residual interaction representation; Standard normalization and feedforward network processing are performed on the residual interaction representation to obtain a fused expression as the fused alignment feature.

7. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 6, characterized in that: The step of inputting the fused alignment features into a multi-task classifier, selecting Sigmoid or Softmax output according to the task attributes, and optimizing the model with the cross entropy loss function to complete multi-label or single-label recognition refers to inputting the fused alignment features into the multi-task classifier, and the multi-task classifier divides different recognition tasks into single-label or multi-label categories according to the task attributes, and configures an independent output branch for each task. If it is a multi-label task, the Sigmoid activation function is used to output the probability class by class, and if it is a single-label task, the Softmax function is used to generate the normalized category distribution probability; Perform forward reasoning on each task to obtain a set of prediction results, which are compared with the true labels. The error of each task is calculated separately based on the cross-entropy loss function. Binary cross entropy is used for multi-label tasks, and category cross entropy is used for single-label tasks. All task losses are combined into a total loss according to preset weights, and used for backpropagation to optimize model parameters.

8. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 7, characterized in that: The synchronous acquisition of multi-source content of text and images refers to using the frame timestamp t as the minimum sampling granularity, continuously extracting frame data from the content source as image modal data at a fixed frame rate through the RGB camera module, and extracting text modal data from the subtitles, labels and explanatory text paired with the image.

9. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 8, characterized in that: The standardization processing and tensor construction refer to performing mean-variance normalization on the image modality data, converting the original pixel values ​​into a standardized image tensor, selecting the WordPiece word segmenter to perform subword splitting on the original text sequence and construct a word segmentation sequence, obtaining the vector representation of each subword through a pre-trained word embedding model and performing normalization processing, and constructing a unified text input tensor.

10. The lightweight multimodal content recognition system based on the dual-track migration framework according to claim 9, characterized in that: The forming of the fused tensor X refers to mapping the feature tensors of the image modality and the text modality to a unified dimensional space, combining the unified tensors of the image modality and the text modality, and generating the fused tensor X.

Images