A lightweight multimodal content recognition system based on a dual-track transfer framework

Abstract

This invention discloses a lightweight multimodal content recognition system based on a dual-track transfer framework, belonging to the field of content recognition technology. It includes a data acquisition module for simultaneously acquiring multi-source content from text and images, performing standardization processing and tensor construction to form a fused tensor X; and a model construction module for inputting the fused tensor into a dual-track transfer structure built on a Transformer backbone network. This dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, outputting an intermediate representation of modality alignment. This invention significantly reduces the training and deployment costs of multimodal content recognition, enhances the semantic expression capabilities within modalities, effectively improves cross-modal alignment accuracy and fusion depth, and enhances the model's ability to perceive and recognize complex semantic relationships.

Description

Technical Field

[0001] This invention relates to the field of content recognition technology, and in particular to a lightweight multimodal content recognition system based on a dual-track transfer framework. Background Technology

[0002] In recent years, with the rapid development of social media, news, and multimedia platforms, the combination of images and text has become the mainstream form of information dissemination, prompting increasing attention to multimodal content recognition technology. Multimodal content recognition aims to perform semantic understanding, classification, and annotation of composite content containing images and text. Its core lies in achieving effective information fusion and semantic consistency modeling between different modalities. Traditional methods often rely on joint embedding spaces or attention mechanisms to align and recognize images and text, frequently using large-scale trained deep network models (such as CLIP and ViLT) to improve cross-modal recognition accuracy. However, such models often require full parameter fine-tuning when transferred to new tasks or specific scenarios, resulting in high training costs and high deployment resource consumption, making them unsuitable for lightweight scenarios and edge computing platforms. Furthermore, existing models lack robust processing mechanisms for spatiotemporal feature variations in image modalities and word loss (such as out-of-vocabulary words) in text modalities, leading to unstable cross-modal semantic alignment and affecting recognition accuracy.

[0003] Existing multimodal content recognition methods, relying solely on single strategies (such as Adapter or Prompt), struggle to simultaneously address the requirements of structural transfer and semantic alignment, resulting in limited fusion performance. In multimodal scenarios, they fail to enhance the modeling of modal differences between images and text, lack decoupling extraction and difference compensation mechanisms, and typically employ additive attention or concatenated embedding methods, which are inadequate for characterizing complex non-additive interactions, thus suppressing the expressive power of the fused representation. Therefore, there is an urgent need to construct a multimodal content recognition system capable of flexible transfer under parameter freezing, modality-specific enhancement, and non-additive interaction modeling, to balance lightweight design, transferability, and recognition performance. Summary of the Invention

[0004] In view of the problems existing in the lightweight multimodal content recognition system based on the dual-track transfer framework, the present invention is proposed.

[0005] Therefore, this invention provides a lightweight multimodal content recognition system based on a dual-track transfer framework to address the urgent need to construct a multimodal content recognition system that can support flexible transfer under parameter freezing, modality-specific enhancement, and non-additive interaction modeling, so as to balance lightweightness, transferability, and recognition performance.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a lightweight multimodal content recognition system based on a dual-track transfer framework, comprising,

[0008] The data acquisition module is used to simultaneously collect multi-source content of text and images, perform standardization processing and tensor construction to form a fused tensor X;

[0009] The model building module is used to input the fusion tensor into a dual-track transfer structure built based on the Transformer backbone network. The dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs an intermediate representation with modality alignment.

[0010] The data augmentation module is used to extract spatial and temporal features from the intermediate representation output by the dual-track transfer structure, respectively, in the image modality and fuse them into an image augmentation representation. In the text modality, for existing OOV terms, the module extracts context embeddings to generate semantic completion Prompt vectors and replaces the original OOV term vectors to obtain the augmented text representation.

[0011] The data fusion module is used to fuse the enhanced image and text representations and construct a joint Prompt vector, guiding the non-additive interaction between the text and image modalities. It completes multimodal fusion through a residual attention mechanism and generates fused aligned features.

[0012] The recognition module is used to input fused and aligned features into a multi-task classifier, select Sigmoid or Softmax output according to task attributes, and optimize the model with cross-entropy loss function to complete multi-label or single-label recognition.

[0013] As a preferred embodiment of the lightweight multimodal content recognition system method based on the dual-track transfer framework described in this invention, the dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and the intermediate representation of output modality alignment includes:

[0014] Extract the task type from the task context, encode the task instruction text using a pre-trained language model to form a task semantic embedding vector, and copy the task semantic embedding vector l times to form the initial prompt matrix o;

[0015] The initial prompt matrix o is transformed into a learnable parameter matrix P, which serves as the core guiding variable for the Prompt Learning channel. The Xavier uniform initialization method is used to initialize all elements in matrix P.

[0016] Insert matrix P from the Prompt Learning channel into the beginning of the modality fusion tensor X along the time dimension to form an extended sequence B. Input B into the frozen Transformer backbone, define three sets of linear transformation matrices in the attention mechanism and construct three types of tensors. Based on the constructed three types of tensors, perform Prompt guided attention weight calculation.

[0017] The fused Prompt context representation is obtained by weighted aggregation of the value tensors using attention weight A. The fused Prompt context representation replaces the corresponding matrix P in the original extended sequence B, while keeping the modal content unchanged, and is then recombined into the updated input sequence B'.

[0018] The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operation as a feature encoder.

[0019] After the feedforward submodule of each Encoder Layer in the Transformer backbone, insert a lightweight Adapter module, set the bottleneck dimension of the Adapter, and attach the Adapter module to the standard residual path for each layer in the Transformer Encoder.

[0020] The output of the Adapter module is residually fused with the original Transformer backbone output to maintain the backbone features while adding compensating perturbations for modal differences.

[0021] The final output tensor after all Adapter enhancements and residual guidance is denoted as B'';

[0022] The generated final output tensor B'' is input into the Transformer backbone structure that has been inserted into the Adapter module. All parameters in the Transformer backbone are frozen, and only the trainable parameters involved in the Prompt guidance path and the Adapter compensation path are retained as optimization targets.

[0023] After setting the parameter freezing strategy, the Adam optimizer is used for iterative updates, and the loss function is set according to the type of downstream task. Gradient updates are performed on the parameters related to Prompt and Adapter to optimize the model's expressive and generalization capabilities on the target task. During training, all frozen parts remain unchanged, and only the parameters in the Prompt and Adapter paths are updated. The fusion sequence is input into the frozen backbone network, and cross-modal fusion representations, i.e., modality-aligned intermediate representations, are generated under the synergistic effect of Prompt intervention and Adapter adjustment.

[0024] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track migration framework described in this invention, wherein: the step of decoupling and extracting spatial and temporal features from the image modality and fusing them into an image enhancement representation refers to receiving the image modality vector in the modality-aligned intermediate representation as input;

[0025] For each frame of the image, a spatial structure partitioning operation is performed, dividing the image into several fixed-size region blocks. For each region block, linear embedding and position encoding are performed to transform it into a spatial feature vector sequence.

[0026] A sliding window strategy is used to locally sample the image frame sequence to construct image subsequences with short time intervals. A time interval modeling network is used to extract the change information between consecutive frames of each subsequence and generate a time feature vector sequence.

[0027] The extracted spatial feature vector sequence is concatenated with the temporal feature vector sequence to form a complete set of image spatiotemporal features. A fusion mapping matrix is ​​constructed to perform a unified mapping operation on the spatial and temporal features, generating an enhanced representation of the image modality.

[0028] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track transfer framework described in this invention, the following steps are taken: For existing OOV (Out-of-V) terms in the text modality, context embeddings are extracted to generate semantic completion Prompt vectors, which are then used to replace the original OOV term vectors to obtain enhanced text representations. This involves checking the visibility of each term in the text sequence using the vocabulary in the corpus, marking unrecognizable terms as OOV, and extracting a fixed-length context sequence before and after each identified OOV position. The construction form is as follows: Context fragments, where, and These represent the semantic context to the left and right of the OOV word, respectively, while the [MASK] placeholder is used to guide the language model to generate context-related semantic embeddings;

[0029] The constructed context fragment is input into the pre-trained language model, which outputs the hidden state vector located at the [MASK] position, denoted as... It serves as the semantic completion process representation for OOV terms. This representation vector is used to replace the initial embedding at the corresponding OOV position in the original text vector, forming the updated text modality embedding sequence F, which is the enhanced text representation.

[0030] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track transfer framework described in this invention, wherein: fusing and enhancing the image and text representations and constructing a joint Prompt vector refers to concatenating the enhanced image representation and the enhanced text representation in the sequence dimension to form a unified fused sequence representation;

[0031] Based on the text position of the identified OOV terms, a set of position masks is constructed to focus on key context in the text modality in the fused sequence. An attention pooling mechanism is applied to perform weighted aggregation of the feature representations of each embedded position in the fused sequence. The attention weights are guided by the OOV position mask to highlight contextual information related to OOV and output a joint Prompt guiding vector.

[0032] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track transfer framework described in this invention, the non-additive interaction between the guiding text and the image modality is achieved through a residual attention mechanism to complete multimodal fusion and generate fused alignment features, including:

[0033] Taking the fused sequence as input, perform multi-head self-attention operation within the modality, extract the self-preserving semantic structure of the features within each modality, and output the intra-modal attention representation;

[0034] Based on the joint Prompt vector as the central query vector, a cross-modal interactive attention mechanism is constructed to extend the interaction results to the same length as the original sequence, forming a cross-feature representation.

[0035] Residual fusion is performed between intramodal self-attention representation and cross feature representation to generate residual interactive representation;

[0036] The residual interactive representation is normalized according to the standard and processed by a feedforward network to obtain a fused representation, which serves as the fused alignment feature.

[0037] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track transfer framework described in this invention, the step of inputting fused alignment features into a multi-task classifier, selecting Sigmoid or Softmax output according to task attributes, and optimizing the model with cross-entropy loss function to complete multi-label or single-label recognition refers to inputting fused alignment features into a multi-task classifier, which divides different recognition tasks into single-label or multi-label categories according to task attributes, and configures an independent output branch for each task. If it is a multi-label task, the Sigmoid activation function is used to output the probability of each category, and if it is a single-label task, the Softmax function is used to generate a normalized category distribution probability.

[0038] Forward inference is performed on each task to obtain a set of prediction results, which are then compared with the true labels. The error of each task is calculated based on the cross-entropy loss function. Binary cross-entropy is used for multi-label tasks, and class cross-entropy is used for single-label tasks. All task losses are combined into a total loss according to preset weights and used for backpropagation to optimize model parameters.

[0039] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track migration framework described in this invention, the simultaneous acquisition of multi-source content of text and images refers to continuously extracting frame data from the content source as image modal data through an RGB camera module at a fixed frame rate, with frame timestamp t as the minimum sampling granularity, and extracting text modal data from subtitles, tags, and explanatory text paired with the image.

[0040] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track transfer framework described in this invention, the standardization process and tensor construction refer to normalizing the mean and variance of the image modal data, converting the original pixel values ​​into a standardized image tensor, using the WordPiece word segmenter to perform sub-word splitting on the original text sequence and constructing a word segmentation sequence, obtaining the vector representation of each sub-word through a pre-trained word embedding model and performing normalization processing, and constructing a unified text input tensor.

[0041] As a preferred embodiment of the lightweight multimodal content recognition system based on the dual-track migration framework described in this invention, the formation of the fusion tensor X refers to mapping the feature tensors of the image modality and the text modality to a unified dimensional space, and combining the unified tensors of the image modality and the text modality to generate the fusion tensor X.

[0042] The beneficial effects of this invention are as follows: By constructing a dual-track transfer structure, this invention effectively achieves task adaptation and structural transfer under the condition of frozen backbone model parameters, significantly reducing the training and deployment costs of multimodal content recognition. It is suitable for lightweight and edge computing scenarios. By introducing image spatiotemporal feature decoupling and text OOV word semantic completion mechanism, it enhances the semantic expression ability within the modality and improves the robustness of the model to unstructured, missing, or heterogeneous information. Furthermore, by combining non-additive interactive fusion guided by Prompt, it effectively improves cross-modal alignment accuracy and fusion depth, and enhances the model's ability to perceive and recognize complex semantic relationships. The overall solution ensures efficient transfer capability while taking into account recognition accuracy, model lightweightness, and application flexibility, which is significantly better than traditional single transfer or additive fusion methods. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the data augmentation process in Example 1;

[0045] Figure 2 This is a schematic diagram of the lightweight multimodal content recognition system based on the dual-track migration framework in Example 1. Detailed Implementation

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0049] Example 1, referring to Figure 1 and Figure 2This is the first embodiment of the present invention. This embodiment provides a lightweight multimodal content recognition system based on a dual-track transfer framework. The lightweight multimodal content recognition system based on the dual-track transfer framework includes the following steps:

[0050] S1, Data Acquisition Module, is used to simultaneously acquire multi-source content of text and images, perform standardization processing and tensor construction to form a fused tensor X;

[0051] Specifically, in order to achieve efficient fusion and alignment of the multimodal content recognition system, this step adopts a frame-level synchronous acquisition mechanism to acquire image modality and text modality data. This scheme selects the fusion of image and text modalities as the optimal path because the image modality has spatial perception advantages and the text modality provides semantic constraints. The two are highly complementary and can achieve better recognition results under a lightweight model structure.

[0052] Using the frame timestamp t as the smallest sampling granularity, the RGB camera module continuously extracts frame data from the content source (such as video frames or captured image sequences) at a fixed frame rate as image modal data, and extracts text modal data from the captions, labels and explanatory text paired with the images;

[0053] The mean and variance of the image modal data are normalized to convert the original pixel values ​​into a standardized image tensor. The WordPiece word segmenter is used to perform sub-word splitting on the original text sequence and construct a word segmentation sequence. The vector representation of each sub-word is obtained through a pre-trained word embedding model (such as BERT embedding) and normalized to construct a unified text input tensor.

[0054] The feature tensors of the image modality and the text modality are mapped to a unified dimensional space. That is, by multiplying the image tensor and the text tensor by their respective linear projection matrices, the image feature representation and the text feature representation under the unified dimension are obtained. The unified tensors of the image modality and the text modality are combined to generate the fused tensor X.

[0055] By using frame-level synchronous acquisition, modal temporal consistency is enhanced, improving the model's adaptability to real-world scenarios. Modal normalization and word embedding strategies improve the model's ability to capture local differences in images and semantic details in text. By generating fusion tensors through unified mapping, a more efficient information representation carrier is provided for the lightweight model structure, effectively reducing the computational burden in the fusion stage. This strategy has good scalability and portability, and is suitable for various application scenarios such as social media content analysis, short video understanding, and cross-modal retrieval. It is a highly adaptable solution that balances performance and resource constraints.

[0056] S2, Model Building Module, is used to input the fusion tensor into a dual-track transfer structure built on the Transformer backbone network. The dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs an intermediate representation with modality alignment.

[0057] Specifically, the dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion. The intermediate representation of output modality alignment includes:

[0058] Extract the task type from the task context (such as the input instruction "identify the actions of people in the image and match the corresponding text description"), encode the task instruction text through a pre-trained language model (such as BERT or T5 encoder) to form a task semantic embedding vector, and copy the task semantic embedding vector l times to form an initial prompt matrix o. For example, if the prompt sequence length to guide the model is set to 8, the task semantic embedding vector is copied 8 times to form a stacked structure. Each row in this structure is exactly the same, which is the same task semantic embedding vector.

[0059] This operation ensures that the Prompt content is directly related to the current recognition task and has a learnable semantic starting point, rather than being arbitrarily initialized;

[0060] The initial prompt matrix o is converted into a learnable parameter matrix P, which is registered as nn.Parameter type in the neural network and serves as the core guiding variable of the Prompt Learning channel. To avoid abnormal convergence caused by the initial weight distribution being too large or too small, the Xavier uniform initialization method is used to initialize all elements of matrix P.

[0061] To ensure that the Prompt Learning channel has the ability to independently adapt to new tasks, the parameters of the Transformer backbone (including attention weights and feedforward layer parameters, etc.) are frozen, and only matrix P in the Prompt Learning channel is allowed to participate in backpropagation updates. In the optimizer (such as AdamW), P is specified as the only optimizable variable.

[0062] Insert the matrix P from the Prompt Learning channel at the beginning of the modality fusion tensor X along the time dimension to form an extended sequence B. This ensures that the Prompt vector is at the beginning of the sequence in the Transformer, naturally possessing a larger information coverage in the self-attention mechanism. Through self-attention computation, the Prompt can "pay attention" to all subsequent modal content and adjust its representation through backpropagation. Input B into the frozen Transformer backbone to define three sets of linear transformation matrices in the attention mechanism. , and ,in , and These are the linear transformation matrices (projection matrices) for the query / key / value pairs, and three types of tensors are constructed, including the query tensor constructed based on the Prompt vector in matrix P:

[0063]

[0064] In the formula, It is the truncated subtensor of the Prompt part, referring to taking the first part of B. Rows, keeping all columns, Q is the query tensor;

[0065] Construct key and value tensors based on Prompt vectors and modality fusion tensor X:

[0066]

[0067] In the formula, K is the key tensor and V is the value tensor;

[0068] Based on the constructed three types of tensors, Prompt-guided attention weight calculation is performed:

[0069]

[0070] In the formula, This represents the hidden dimension of each attention head in the Transformer, also known as the projection dimension;

[0071] The fused Prompt context representation is obtained by weighted aggregation of the value tensors using attention weight A. :

[0072]

[0073] The fused Prompt context representation replaces the corresponding matrix P in the original extended sequence B, keeping the modal content unchanged, and is recombined into the updated input sequence B'. This new sequence serves as the next layer input of the Transformer backbone structure, driving the Prompt guidance information to further propagate and interact in a deeper coding structure, realizing continuous guidance and enhancement of cross-modal information. Thus, the first round of context information injection process based on Prompt and modal content is completed, and a unified semantic input foundation is established for subsequent adaptation modules (such as the Adapter structure).

[0074] The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operation as a feature encoder.

[0075] After each Encoder Layer's feed-forward network (FFN) in the Transformer backbone, insert a lightweight Adapter module, setting the Adapter's bottleneck dimension to [value missing]. , For each layer in the Transformer Encoder, the following Adapter module is attached to the standard residual path:

[0076]

[0077] In the formula, Adapter(x) is the output vector after lightweight perturbation compensation processing of the input vector x, and ReLU() is a non-linear activation function. It is a downward projection matrix that restores the compressed features to their original dimensions. It is the projection matrix, which compresses high-dimensional features into low-dimensional representations, where x is the feature vector at a certain position in the current Transformer layer;

[0078] The output of the Adapter module is residually fused with the original Transformer backbone output. While maintaining the backbone features, compensation perturbations for modal differences are added:

[0079]

[0080] In the formula, It is the final output vector after processing by the Adapter module, which is the result of adding Adapter(x) as a perturbation term to the original feature through residual connection;

[0081] The final output tensor after all Adapter enhancements and residual guidance is denoted as B''. This tensor retains the Prompt guidance information, modal temporal structure, and feature dimension consistency in its structure.

[0082] The generated final output tensor B'' is input into the Transformer backbone structure that has been inserted into the Adapter module. At this time, the Transformer backbone contains the Prompt guidance path and the Adapter compensation path. The Prompt vector guides the attention mechanism to focus on the modality important position, and the Adapter module is used for feature perturbation compensation between layers.

[0083] To avoid compromising the capabilities of the original pre-trained model in transfer tasks with limited samples or resources, all parameters in the Transformer backbone are frozen, including the query, key, value, and output matrices in the multi-head attention layer, all weight matrices in the feedforward layer, LayerNorm normalization parameters, and all positional encoding and embedding weights. This operation ensures that the backbone remains unchanged, and adaptation to the target task is achieved only by fine-tuning some lightweight modules.

[0084] Based on parameter freezing, only trainable parameters involved in the Prompt guidance path and Adapter compensation path are explicitly retained as optimization targets. Specifically, these include the learnable Prompt vector matrix and the linear transformation matrix of the upper and lower projections in each Adapter module. This lightweight training configuration significantly compresses the parameter space, enabling the system to have good adaptability and training efficiency.

[0085] After the parameter freezing strategy is set, the Adam optimizer is used for iterative updates, and the loss function is set according to the type of downstream task (such as cross-entropy for classification tasks and mean squared error for regression tasks). Gradient updates are performed on the parameters related to Prompt and Adapter to optimize the model's expressive and generalization capabilities on the target task. During training, all frozen parts remain unchanged, and only the parameters in the Prompt and Adapter paths are updated, thereby guiding the network to gradually adapt to the new modality combination structure while retaining the original representation capabilities.

[0086] The fused sequence is input into the frozen backbone network, and a cross-modal fusion representation is generated under the synergistic effect of Prompt intervention and Adapter adjustment. This representation not only retains the original structural advantages of the Transformer backbone, but also integrates multimodal residual compensation information, which serves as the input for the subsequent modal alignment module. Thus, the complete migration path from the fused input structure to the lightweight optimized configuration is completed, and the output is the modal alignment intermediate representation with task adaptability.

[0087] The proposed dual-track transfer framework effectively improves accuracy, efficiency, and adaptability in multimodal content recognition tasks by synergistically fusing two lightweight transfer mechanisms: Prompt Learning and Adapter-Tuning. While maintaining the Transformer backbone parameters frozen, Prompt Learning... The Learning module introduces a learnable task guidance matrix, embedding task semantics into the input sequence in a structured form. This enhances the model's ability to focus attention on semantically relevant modal regions, avoiding the problem of insufficient generalization in small-sample scenarios that traditional methods often suffer from. Simultaneously, the Adapter-Tuning module introduces residual perturbation adjustment channels in each Transformer layer, achieving fine-tuning compensation of modal features through low-dimensional up-down projection and nonlinear activation. This significantly reduces parameter overhead while preserving original expressive power, strengthening the model's stability in modal alignment and structure preservation. Furthermore, the solution introduces a non-additive interaction mechanism and a residual attention optimization strategy, significantly improving the information fusion quality between image and text modalities, making the fused representation more discriminative and consistent. The modal-aligned intermediate representation generated through these mechanisms not only considers task semantics and modal characteristics but also supports multi-task output for both multi-label and single-label recognition, exhibiting high versatility. Overall, this invention achieves parameter compression, improved training efficiency, and enhanced multimodal feature expression without sacrificing recognition performance. It provides an efficient, low-cost, and easily deployable multimodal content recognition solution for real-world applications with limited computing resources or frequent task switching.

[0088] S3, the data augmentation module, is used to extract spatial and temporal features from the intermediate representation output based on the dual-track transfer structure, and then fuse them into an image augmentation representation. In the text modality, for existing OOV (Out Of Vocabulary) terms, it extracts context embeddings to generate semantic completion Prompt vectors and replaces the original OOV word vectors to obtain the augmented text representation.

[0089] Specifically, decoupling and extracting spatial and temporal features from image modalities and fusing them into an image enhancement representation refers to receiving the image modal vector in the modal alignment intermediate representation as input. This representation has been embedded with semantic information through the Prompt and Adapter paths and has alignment features for text modalities, thus serving as a unified input basis for subsequent image structure analysis.

[0090] For each frame of image, a spatial structure partitioning operation is performed, dividing the image into several fixed-size regions. Each region is linearly embedded and positionally encoded, transforming it into a spatial feature vector sequence. By introducing positional encoding, the model can extract local visual information while maintaining the recognizability of spatial order, thereby enhancing the modeling ability of static image structure.

[0091] A sliding window strategy is used to locally sample the image frame sequence to construct image subsequences with short time intervals. The change information between consecutive frames of each subsequence is extracted by a time interval modeling network (such as TSN) to generate a time feature vector sequence, which supplements the image's expressive ability in terms of action, transformation and context change.

[0092] The extracted spatial feature vector sequence is concatenated with the temporal feature vector sequence to form a complete set of spatiotemporal image features. A fusion mapping matrix is ​​constructed to perform a unified mapping operation on the spatial and temporal features, generating an enhanced representation of the image modality. This includes concatenating the spatial and temporal feature vectors extracted from the image modality along the feature dimension to form a joint representation with dimension (n+k)×2e, where n is the number of spatial blocks obtained after the image modality is segmented, k is the number of temporal frames extracted by sliding sampling, and e is the embedding dimension of each feature vector. Through a linear transformation matrix, the concatenated features are mapped to a unified representation with dimension e, thereby achieving the fusion of spatial and temporal features and outputting an enhanced image representation that ensures that it contains both the compositional information of the static structure and the dynamic features of the temporal dimension.

[0093] It achieves decoupled reconstruction and semantic completion of modality alignment results, effectively improving the representation accuracy and structural awareness of image modalities in complex multimodal tasks.

[0094] First, by separating and unifying the spatial and temporal dimensions through modeling, the image modality's ability to jointly perceive static and dynamic features is significantly enhanced, improving its robustness in complex contexts. Second, by introducing positional encoding and temporal modeling mechanisms, the model is equipped with the ability to reason about cross-frame structural relationships and identify changing trends, resulting in better performance in multimodal tasks such as character action understanding and scene change recognition. Third, this enhanced representation, as the input to subsequent multimodal interaction modules, not only preserves the image's own structural hierarchy information but also achieves high-quality fusion with the text modality through semantic alignment, effectively improving the accuracy and contextual consistency of the final content recognition.

[0095] Furthermore, for existing OOV (Out of Voice) terms in the text modality, context embeddings are extracted to generate semantic completion Prompt vectors, which are then used to replace the original OOV term vectors, resulting in an enhanced text representation. This involves checking the visibility of each term in the text sequence using the vocabulary in the corpus, marking unrecognizable terms as OOV, and extracting a fixed-length context sequence before and after each identified OOV position. The construction form is as follows: Context fragments, where, and These represent the semantic context to the left and right of the OOV word, respectively, while the [MASK] placeholder is used to guide the language model to generate context-related semantic embeddings;

[0096] The constructed context fragment is input into a pre-trained language model (such as BERT), which outputs the hidden state vector located at the [MASK] position, denoted as... It serves as the semantic completion process representation for OOV terms. This representation vector is used to replace the initial embedding at the corresponding OOV position in the original text vector, forming the updated text modality embedding sequence F, which is the enhanced text representation.

[0097] By introducing a context-driven OOV word semantic completion mechanism, the representational integrity and semantic accuracy of text modalities in multimodal content recognition tasks are significantly improved. Addressing the OOV problem caused by the inability of traditional vocabulary to cover domain-specific words or spelling variations, this solution constructs contextual fragments containing [MASK] placeholders, guiding pre-trained language models (such as BERT) to generate dynamic semantic vectors in the current context. These vectors are then used to replace the original OOV word position embeddings, thereby restoring and enhancing semantic information. This approach not only ensures the structural continuity of the text embedding sequence but also makes semantic expression more context-sensitive, improving the understanding of polysemous words and uncertain terms. During multimodal feature alignment and fusion, the enhanced text representation can form a more accurate semantic pair with image modal features, reducing semantic offset and alignment errors between modalities. Furthermore, because this strategy is free from the limitations of static vocabulary, it possesses good transfer adaptability, demonstrating stronger robustness and adaptability in cold-start scenarios and vertical domain tasks, comprehensively promoting the stability and recognition accuracy of lightweight multimodal recognition systems.

[0098] S4, the data fusion module, is used to fuse the enhanced image and text representations and construct a joint Prompt vector to guide the non-additive interaction between the text and image modalities. It completes multimodal fusion through the residual attention mechanism and generates fused aligned features.

[0099] Specifically, fusing the enhanced image and text representations and constructing a joint Prompt vector means concatenating the enhanced image and text representations in the sequence dimension to form a unified fused sequence representation. This fused sequence not only preserves the visual information and temporal dynamics in the image, but also incorporates the semantic completion content in the text, providing a structural foundation for cross-modal modeling.

[0100] Based on the text positions of the identified OOV terms, a set of positional masks is constructed to focus on key contexts in the text modality in the fusion sequence. An attention pooling mechanism is applied to perform weighted aggregation of the feature representations of each embedded position in the fusion sequence. The attention weights are guided by the OOV positional masks to highlight contextual information related to OOV. A joint prompt vector is output, whose semantics reflect the comprehensive features related to OOV terms in the fusion modality. This joint prompt not only completes the missing semantics in the original text modality but also carries background guidance information from the image modality, thereby improving the information fusion efficiency in subsequent multimodal interactions.

[0101] The fused sequence and the joint Prompt vector are used together as input to the subsequent multimodal non-additive interaction module to achieve deep fusion representation between text and images.

[0102] By constructing a joint Prompt vector that fuses and enhances images and text, the problem of semantic fragmentation between modalities and missing information of out-of-vocabulary (OOV) words in multimodal content recognition is effectively solved, significantly improving the depth and accuracy of image-text fusion. The construction of the fusion sequence preserves the visual details of the image and the contextual semantics of the text in a unified representation space, laying a structural foundation for subsequent cross-modal interaction. The position masking mechanism focuses on the contextual region of OOV words, and the key information is weighted and converged by the attention pooling strategy. The generated joint Prompt not only fills the semantic gaps in the text modality, but also embeds the background guidance information of the image modality, thereby realizing cross-modal information completion and alignment. The joint Prompt and the fusion sequence are input into the non-additive interaction module, which effectively guides the model to explore complex modal dependencies in the nonlinear representation space. Compared with the traditional weighted mechanism, it has stronger semantic penetration ability and fusion flexibility.

[0103] Furthermore, it guides non-additive interactions between text and image modalities, achieving multimodal fusion through a residual attention mechanism, and generating fused alignment features including:

[0104] Taking the fused sequence as input, perform multi-head self-attention operation within the modality, extract the self-preserving semantic structure of the features within each modality, and output the intra-modal attention representation;

[0105] Based on the joint Prompt vector as the central query vector, a cross-modal interactive attention mechanism is constructed. This mechanism uses the joint Prompt vector as the query and the fused sequence as the key and value. The calculated cross-attention output is used to capture the non-additive semantic dependency between the image and the text. In order to maintain the consistency of features at each location, the interaction result is extended to the same length as the original sequence to form a cross-feature representation.

[0106] Residual fusion is performed between intramodal self-attention representation and cross feature representation to generate residual interactive representation;

[0107] The residual interactive representation is normalized according to the standard and processed by the feedforward network to obtain the fused representation as the fusion alignment feature. The layer normalization improves the numerical stability, while the feedforward network introduces nonlinear activation to improve the representation dimension and discriminative ability. This feature has the characteristics of modality consistency, prompt-guided semantic fusion and residual information preservation, and completes the deep alignment and fusion of image and text modalities in a unified space.

[0108] By introducing a residual attention mechanism to achieve non-additive interactive fusion of image and text modalities, the consistency and expressiveness of multimodal representations are significantly improved. This addresses the issues of information loss and insufficient alignment in traditional additive fusion when there are large differences in modal structures. By constructing a joint Prompt vector as the interaction center, cross-modal semantic fusion can be effectively guided, enabling fine-grained modeling of non-additive dependencies between images and text. This enhances the model's ability to perceive contextual associations and semantic cues. Simultaneously, multi-head self-attention within each modality maintains its respective semantic structure, while residual fusion balances intramodal consistency and interactive feature supplementation, mitigating semantic shift and feature erosion during the fusion process. Furthermore, combining layer normalization and feedforward neural network processing not only improves the numerical stability of features but also enhances nonlinear expressiveness and classification performance. The resulting fused and aligned features exhibit good modal consistency, context adaptability, and discriminative ability, providing strong support for multi-task content recognition while maintaining a lightweight model structure suitable for real-time applications. This demonstrates significant practical value and potential for widespread adoption.

[0109] S5, the recognition module, is used to input the fused and aligned features into the multi-task classifier, select Sigmoid or Softmax output according to the task attributes, and optimize the model with the cross-entropy loss function to complete multi-label or single-label recognition.

[0110] Specifically, the fused alignment features are input into a multi-task classifier, which selects either Sigmoid or Softmax output based on the task attributes, and optimizes the model using the cross-entropy loss function to complete multi-label or single-label recognition. The fused alignment features are input into a multi-task classifier (such as Multi-Task Learning), which divides different recognition tasks into single-label or multi-label categories based on task attributes. Each task is configured with an independent output branch. For multi-label tasks, the Sigmoid activation function is used to output probabilities for each category, while for single-label tasks, the Softmax function is used to generate normalized category distribution probabilities.

[0111] Forward inference is performed on each task to obtain a set of prediction results, which are then compared with the real labels. The error of each task is calculated based on the cross-entropy loss function. Binary cross-entropy is used for multi-label tasks, and class cross-entropy is used for single-label tasks to ensure that the loss function matches the task attributes. All task losses are combined into a total loss according to preset weights and used for backpropagation to optimize model parameters. The optimized model parameters refer to the joint backpropagation training of all unfrozen trainable parameters in the dual-track transfer structure, residual attention mechanism, and multi-task classifier to improve the performance of the entire multimodal content recognition system.

[0112] By dynamically selecting activation functions and output structures based on task attributes, the optimal recognition strategy is ensured for each task type, improving the model's generalization ability and adaptability to practical applications. Different cross-entropy loss functions are used to calculate errors for multi-label and single-label tasks, and these errors are merged into a total loss function according to preset task importance weights. This effectively controls the primary and secondary relationships during multi-task training, enhancing system tunability. Backpropagation is used to uniformly optimize and train the dual-track transfer structure, residual attention mechanism, and all trainable parameters in the multi-task classifier, ensuring the co-evolution of modality representation learning and classification decision-making, thus improving the overall recognition performance of the model. The classifier module design supports subsequent expansion to more sub-tasks, and the fusion of representations enhances semantic consistency, helping to maintain robust recognition capabilities in complex contexts.

[0113] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight multimodal content recognition system based on a dual-track transfer framework, characterized in that: include, The data acquisition module is used to simultaneously collect multi-source content of text and images, perform standardization processing and tensor construction to form a fused tensor X; The model building module is used to input the fusion tensor into a dual-track transfer structure built based on the Transformer backbone network. The dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion, and outputs an intermediate representation with modality alignment. The data augmentation module is used to extract spatial and temporal features from the intermediate representation output by the dual-track transfer structure, respectively, in the image modality and fuse them into an image augmentation representation. In the text modality, for existing OOV terms, the module extracts context embeddings to generate semantic completion Prompt vectors and replaces the original OOV term vectors to obtain the augmented text representation. The data fusion module is used to fuse the enhanced image and text representations and construct a joint Prompt vector, guiding the non-additive interaction between the text and image modalities. It completes multimodal fusion through a residual attention mechanism and generates fused aligned features. The recognition module is used to input fused and aligned features into a multi-task classifier, select Sigmoid or Softmax output according to task attributes, and optimize the model with cross-entropy loss function to complete multi-label or single-label recognition.

2. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 1, characterized in that: The dual-track transfer structure achieves task semantic alignment and structural transfer under parameter freezing through Prompt Learning embedding and Adapter-Tuning insertion. The intermediate representation of the output modality alignment includes: Extract the task type from the task context, encode the task instruction text using a pre-trained language model to form a task semantic embedding vector, and copy the task semantic embedding vector l times to form the initial prompt matrix o; The initial prompt matrix o is transformed into a learnable parameter matrix P, which serves as the core guiding variable for the Prompt Learning channel. All elements in matrix P are initialized using the Xavier uniform initialization method. Insert matrix P from the Prompt Learning channel into the beginning of the modality fusion tensor X along the time dimension to form an extended sequence B. Input B into the frozen Transformer backbone, define three sets of linear transformation matrices in the attention mechanism and construct three types of tensors. Based on the constructed three types of tensors, perform Prompt guided attention weight calculation. The fused Prompt context representation is obtained by weighted aggregation of the value tensors using attention weight A. The fused Prompt context representation replaces the corresponding matrix P in the original extended sequence B, while keeping the modal content unchanged, and is then recombined into the updated input sequence B'. The updated input sequence B' is input into the frozen Transformer backbone model. The Transformer backbone keeps the original parameters unchanged and only performs multi-layer Transformer Encoder Layer stacking operation as a feature encoder. After the feedforward submodule of each Encoder Layer in the Transformer backbone, insert a lightweight Adapter module, set the bottleneck dimension of the Adapter, and attach the Adapter module to the standard residual path for each layer in the Transformer Encoder. The output of the Adapter module is residually fused with the original Transformer backbone output to maintain the backbone features while adding compensating perturbations for modal differences. The final output tensor after all Adapter enhancements and residual guidance is denoted as B''; The generated final output tensor B'' is input into the Transformer backbone structure that has been inserted into the Adapter module. All parameters in the Transformer backbone are frozen, and only the trainable parameters involved in the Prompt guidance path and the Adapter compensation path are retained as optimization targets. After setting the parameter freezing strategy, the Adam optimizer is used for iterative updates, and the loss function is set according to the type of downstream task. Gradient updates are performed on the parameters related to Prompt and Adapter to optimize the model's expressive and generalization capabilities on the target task. During training, all frozen parts remain unchanged, and only the parameters in the Prompt and Adapter paths are updated. The fusion sequence is input into the frozen backbone network, and cross-modal fusion representations, i.e., modally aligned intermediate representations, are generated under the synergistic effect of Prompt intervention and Adapter adjustment.

3. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 2, characterized in that: The process of decoupling and extracting spatial and temporal features from image modalities and fusing them into an image enhancement representation refers to receiving the image modal vector in the modality-aligned intermediate representation as input. For each frame of the image, a spatial structure partitioning operation is performed, dividing the image into several fixed-size region blocks. For each region block, linear embedding and position encoding are performed to transform it into a spatial feature vector sequence. A sliding window strategy is used to locally sample the image frame sequence to construct image subsequences with short time intervals. A time interval modeling network is used to extract the change information between consecutive frames of each subsequence and generate a time feature vector sequence. The extracted spatial feature vector sequence is concatenated with the temporal feature vector sequence to form a complete set of image spatiotemporal features. A fusion mapping matrix is ​​constructed to perform a unified mapping operation on the spatial and temporal features, generating an enhanced representation of the image modality.

4. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 3, characterized in that: The process involves extracting context embeddings to generate semantic completion Prompt vectors for existing OOV (Out-of-V) terms in the text modality and replacing the original OOV term vectors to obtain the enhanced text representation. This process involves checking the visibility of each term in the text sequence using the vocabulary in the corpus, marking unrecognizable terms as OOV, and extracting a fixed-length context sequence before and after each identified OOV position. The construction form is as follows: Context fragments, where, and These represent the semantic context to the left and right of the OOV word, respectively, while the [MASK] placeholder is used to guide the language model to generate context-related semantic embeddings; The constructed context fragment is input into the pre-trained language model, which outputs the hidden state vector located at the [MASK] position, denoted as... And as a semantic completion process representation of OOV terms, this hidden state vector is used to replace the initial embedding at the corresponding OOV position in the original text vector to form the updated text modality embedding sequence F, that is, the enhanced text representation.

5. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 4, characterized in that: The fusion and enhancement of the image and text representations and the construction of the joint Prompt vector refer to the concatenation of the enhanced image representation and the enhanced text representation in the sequence dimension to form a unified fused sequence representation; Based on the text position of the identified OOV terms, a set of position masks is constructed to focus on key context in the text modality in the fused sequence. An attention pooling mechanism is applied to perform weighted aggregation of the feature representations of each embedded position in the fused sequence. The attention weights are guided by the OOV position mask to highlight contextual information related to OOV and output a joint Prompt guiding vector.

6. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 5, characterized in that: The non-additive interaction between the guiding text and the image modality is achieved through multimodal fusion via a residual attention mechanism, generating fused alignment features including: Taking the fused sequence as input, perform multi-head self-attention operation within the modality, extract the self-preserving semantic structure of the features within each modality, and output the intra-modal attention representation; Based on the joint Prompt vector as the central query vector, a cross-modal interactive attention mechanism is constructed to extend the interaction results to the same length as the original sequence, forming a cross-feature representation. Residual fusion is performed between intramodal self-attention representation and cross feature representation to generate residual interactive representation; The residual interactive representation is normalized according to the standard and processed by a feedforward network to obtain a fused representation, which serves as the fused alignment feature.

7. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 6, characterized in that: The process involves inputting the fused alignment features into a multi-task classifier, selecting either Sigmoid or Softmax output based on task attributes, and optimizing the model using the cross-entropy loss function to complete multi-label or single-label recognition. This process involves inputting the fused alignment features into the multi-task classifier, which then divides different recognition tasks into single-label or multi-label categories based on task attributes. Each task is configured with an independent output branch. For multi-label tasks, the Sigmoid activation function is used to output probabilities for each category; for single-label tasks, the Softmax function is used to generate normalized category distribution probabilities. Forward inference is performed on each task to obtain a set of prediction results, which are then compared with the true labels. The error of each task is calculated based on the cross-entropy loss function. Binary cross-entropy is used for multi-label tasks, and class cross-entropy is used for single-label tasks. All task losses are combined into a total loss according to preset weights and used for backpropagation to optimize model parameters.

8. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 7, characterized in that: The simultaneous acquisition of multi-source content of text and images refers to continuously extracting frame data from the content source as image modal data through an RGB camera module at a fixed frame rate, with the frame timestamp t as the smallest sampling granularity, and extracting text modal data from the captions, labels and explanatory text paired with the images.

9. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 8, characterized in that: The standardization process and tensor construction refer to the mean and variance normalization of the image modal data, converting the original pixel values ​​into a standardized image tensor, using the WordPiece word segmenter to perform sub-word splitting on the original text sequence and constructing a word segmentation sequence, obtaining the vector representation of each sub-word through a pre-trained word embedding model and performing normalization processing, and constructing a unified text input tensor.

10. The lightweight multimodal content recognition system based on a dual-track transfer framework as described in claim 9, characterized in that: The formation of the fusion tensor X refers to mapping the feature tensors of the image modality and the text modality to a unified dimensional space, and combining the unified tensors of the image modality and the text modality to generate the fusion tensor X.

Images