A lightweight multi-modal representation learning method based on multi-layer attention mechanism

Abstract

The application discloses a lightweight multi-modal representation learning method based on a multi-layer attention mechanism, and belongs to the technical field of multi-modal data processing, and aims to solve the problem that representation capability and computational efficiency are difficult to be considered in multi-modal representation learning. The method comprises the following steps: obtaining original multi-modal data collected by a computing node, and preprocessing to generate basic features of each modality; high-level semantic features of each modality are extracted through a lightweight extractor of a simplified visual Transformer, a lightweight BERT variant and a time convolution network; a low-dimensional cross-modal vector is generated through a hierarchical attention mechanism model containing intra-modal, cross-modal and global fusion attention; a lightweight model is obtained by pruning and optimizing the hierarchical attention mechanism model structure; and after the lightweight model is trained based on the low-dimensional cross-modal vector, the computing node is deployed for multi-modal upper task requirements. The application considers representation capability and computational efficiency, adapts to low-computing-power-demand computing nodes, and guarantees task accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to multimodal data processing, and more specifically to a lightweight multimodal representation learning method based on a multi-layer attention mechanism. Background Technology

[0002] With the development of artificial intelligence, the demand for multimodal interaction and perception is becoming increasingly urgent for computing nodes in low-computing-power scenarios such as mobile phones, in-vehicle terminals, and IoT devices. For example, scenarios like mobile phone "voice + image" intelligent object search and in-vehicle "camera + radar" environmental perception both require real-time local processing of heterogeneous data to reduce cloud dependence and transmission latency. Multimodal representation learning, as a core supporting technology, aims to transform heterogeneous data into low-dimensional vectors in a unified semantic space, providing a foundation for upper-layer tasks. Current cloud solutions are relatively mature, extracting single-modal features through models such as Transformer and BERT and achieving cross-modal fusion based on attention mechanisms, but they still face significant challenges when adapting to low-computing-power nodes.

[0003] The bottlenecks of existing technologies are mainly reflected in the following aspects: First, the computation and storage costs of models are too high. Traditional multimodal models often have hundreds of millions of parameters and computational costs exceeding hundreds of gigabytes, far exceeding the hardware constraints of low-computing-power nodes, resulting in excessively high inference latency or even memory overflow. Second, lightweight strategies do not take into account multimodal representation capabilities. Single pruning and quantization can easily destroy the ability to capture cross-modal associations or intramodal semantic information. Third, the fusion mechanism is mismatched with computational requirements. Single-layer attention or simple concatenation is difficult to effectively associate heterogeneous semantics, and layered attention does not filter intramodal redundancy, resulting in excessively high vector dimensions. Fourth, the training objectives are unbalanced. A single loss function only focuses on output matching and ignores the optimization of intramodal feature quality, making it difficult to balance accuracy and efficiency. Therefore, there is an urgent need for a technical solution that can simultaneously ensure the quality of multimodal representation and inference efficiency under the low-computing-power constraints of low-computing-power nodes. Summary of the Invention

[0004] The purpose of this invention is to provide a lightweight multimodal representation learning method based on a multi-layer attention mechanism to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A lightweight multimodal representation learning method based on a multi-layer attention mechanism includes the following steps:

[0007] Step 1. Acquire the raw multimodal data collected by the computing nodes, and preprocess the raw multimodal data to generate the basic features of each modality;

[0008] Step 2. Use a lightweight feature extractor to extract high-level semantic features from the basic features of each modality data, and construct a hierarchical attention mechanism model. Then, fuse the high-level semantic features of each modality data across modalities through the hierarchical attention mechanism model to generate a low-dimensional cross-modal vector.

[0009] Step 3. Perform lightweight optimization on the hierarchical attention mechanism model to form a lightweight model;

[0010] Step 4. Train the lightweight model using low-dimensional cross-modal vectors as the training benchmark to obtain a lightweight multimodal model;

[0011] Step 5. Deploy the lightweight multimodal model to the compute nodes to meet the multimodal upper-layer task requirements of the compute nodes.

[0012] Preferably, the original multimodal data comprises image data, text data, and speech data; the method for preprocessing the original multimodal data is as follows:

[0013] Image data is normalized for mean and variance using ImageNet statistics to obtain standardized RGB image data. The ImageNet statistics are learned from a large-scale ImageNet dataset and are used to effectively eliminate differences in pixel value distribution caused by lighting and device model in different images. The standardized RGB image data is adaptively scaled to 224×224 resolution through bilinear interpolation to match the input dimension of the simplified visual Transformer of 224×224. Then, Gaussian blur with a kernel size of 3×3 is used to filter high-frequency noise to obtain the basic features of the image data. The basic features of the image data are a standardized matrix of 224×224×3.

[0014] For Chinese scenarios, the jieba lightweight word segmentation mode is used to segment the text data. For multilingual scenarios, the SentencePiece lightweight word segmentation version is used to segment the text data. Both segmentation methods output discrete token sequences. The discrete token sequences are input into a pre-trained word embedding model to be converted into a high-dimensional word vector matrix. Then, a shared linear projection layer is used to perform linear projection dimensionality reduction, reducing the high-dimensional word vectors in the high-dimensional word vector matrix to 256 dimensions. This reduces the computational and memory overhead of subsequent feature processing, and finally outputs the basic features of the text data. The basic features of the text data are a matrix of "token sequence length × 256 dimensions". The token sequence length is uniformly 32. If the token sequence length is less than 32, the short sequence is padded with zeros. If the token sequence length is greater than 32, the long sequence is truncated.

[0015] The shared linear projection layer refers to a linear transformation module whose parameters are reused between feature transformation tasks of two or more modalities in a multimodal feature processing flow.

[0016] The word embedding model is either the basic word embedding layer of DistilBERT or Word2Vec;

[0017] The speech data is subjected to a short-time Fourier transform using a 512-point FFT to extract the spectrum, and the spectrum is converted into Mel frequency cepstral coefficients using a Mel filter bank to output the Vimel spectral features, which are the basic features of the speech data.

[0018] Preferably, the method for extracting high-level semantic features of each modality data is as follows:

[0019] For image data, a simplified visual Transformer is used to extract high-level semantic features; for text data, a lightweight BERT variant is used to extract high-level semantic features; and for speech data, a temporal convolutional network is used to extract high-level semantic features.

[0020] The simplified visual Transformer is optimized based on the original ViT structure and specifically includes an input embedding layer, a simplified encoder layer, and a global pooling layer. The input embedding layer replaces the image blocks and linear projection of the original ViT with 16×16 block convolutions, and removes the clstoken of the original ViT. The simplified encoder layer retains only 4 encoder layers, and each encoder layer consists of a multi-head self-attention layer and a multi-layer perceptron. The number of attention heads in the multi-head self-attention layer is set to 4, and the hidden layer dimension of the multi-layer perceptron is set to 512. 1×1 convolutions combined with GELU activation are used to replace the fully connected layers, and the dropout layer in the multi-layer perceptron of the original ViT is removed.

[0021] The lightweight BERT variant is optimized based on the DistilBERT framework and specifically includes a word embedding layer, a simplified TransformerEncoder layer, and a feature aggregation layer. The word embedding matrix dimension in the word embedding layer is set to 256. The simplified TransformerEncoder layer retains 3 TransformerEncoder layers, and the number of attention heads in the multi-head self-attention layer of each TransformerEncoder layer is set to 4. The hidden layer dimension of the feedforward network is set to 512. At the same time, the layer normalization redundant parameters in the feedforward network of DistilBERT are removed.

[0022] The simplified visual Transformer, the lightweight BERT variant, and the temporal convolutional network are all lightweight feature extractors.

[0023] Preferably, the hierarchical attention mechanism model has a three-level structure, specifically including an intra-modal attention layer, a cross-modal attention layer, and a global fusion attention layer;

[0024] The intramodal attention layer is used to take the high-level semantic features of each modal data as input, filter redundant information in the high-level semantic features of each modal data through spatial attention, word-level self-attention and temporal attention, and output the local features of each modal data. Among them, spatial attention is used to process the high-level semantic features of image data, word-level self-attention is used to process the high-level semantic features of text data, and temporal attention is used to process the high-level semantic features of speech data.

[0025] The cross-modal attention layer is used to project the local features of each modality data into the same semantic space through a shared fully connected layer to obtain image alignment features, text alignment features and speech alignment features. The cosine similarity between the image alignment features and text alignment features, as well as between the speech alignment features and text alignment features, is calculated using the text alignment features as anchor points to generate attention scores. The cross-modal association features are obtained by weighted fusion according to the attention scores.

[0026] The shared fully connected layer refers to a fully connected transformation module in the multimodal feature fusion and transformation process, in which parameters are reused between feature semantic mapping or related tasks of two or more modalities.

[0027] The global fusion attention layer is used to concatenate the local features of each modality data with the cross-modal correlation features, and then filter key information by learning channel weights through a gating network. After dimensionality compression, a low-dimensional cross-modal vector is output. The gating network adopts a 2-layer fully connected and Sigmoid activation structure.

[0028] Preferably, the hierarchical attention mechanism model is optimized for lightweighting through structural pruning, the specific process of which includes:

[0029] The contribution of each attention head in the intramodal attention layer for spatial attention, word-level self-attention, and temporal attention is calculated using the L1 norm. After sorting the attention heads by contribution from largest to smallest, the attention heads with the lowest contribution ranking are removed. At the same time, the output projection fully connected layers of spatial attention, word-level self-attention, and temporal attention are subjected to sensitivity analysis using gradient norm to obtain the gradient norm of the weights of each neuron in the projection fully connected layer. After sorting the gradient norms by gradient norm from largest to smallest, the neurons with the lowest gradient norm ranking in the output projection fully connected layer are removed, i.e., redundant neurons.

[0030] For the gating network with a global fusion attention layer, remove input channels in the input layer of the gating network whose activation value percentage is less than 5%, where the activation value percentage is the ratio of the average activation value of the input channel to the sum of the average activation values ​​of all input channels. At the same time, remove neurons in the output layer of the gating network whose absolute weight value is ≤1e-4. If the absolute weight value of a neuron is ≤1e-4, it means that the neuron's modulatory effect on the channel weight is close to 0, and it is an ineffective neuron.

[0031] Preferably, when training the lightweight model using a low-dimensional cross-modal vector as the training benchmark, a multi-task loss function is used to optimize the lightweight model parameters, and the training termination condition is that the output matching loss decreases by less than 0.001 for three consecutive training cycles and the cosine similarity between the model output vector and the low-dimensional cross-modal vector after training is ≥0.9; the multi-task loss function is composed of a weighted sum of output matching loss and intra-modal feature quality loss, and the weight ratio of output matching loss to intra-modal feature quality loss is 1:(0.8-1.2); the output matching loss is the L2 distance loss between the lightweight model output vector and the low-dimensional cross-modal vector, and the intra-modal feature quality loss is obtained by weighting the image high-level semantic feature classification loss and the text high-level semantic feature masking language model loss in a 1:1 ratio, wherein both the image high-level semantic feature classification loss and the text high-level semantic feature masking language model loss use cross-entropy loss;

[0032] The L2 distance loss is expressed by the formula: Where N is the number of training samples, Let i be the output vector of the lightweight model for the i-th training sample. It is a low-dimensional cross-modal vector. For L2 distance loss;

[0033] The output vector of the trained model is a low-dimensional cross-modal vector obtained by the lightweight multimodal model processing the high-level semantic features of each modality data. The output vector of the lightweight model is a low-dimensional cross-modal vector obtained by the lightweight model processing the high-level semantic features of each modality data during training.

[0034] Preferably, when deploying the lightweight multimodal model to the computing node, the lightweight multimodal model is first converted to .tflite or .onnx format to adapt to the hardware and system environment of the computing node, and then multimodal upper-layer tasks are executed based on the format-converted lightweight multimodal model; the multimodal upper-layer tasks include cross-modal retrieval tasks and multimodal intent recognition tasks, and the inference process of both types of tasks is completed locally on the computing node without relying on cloud computing power.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention realizes the construction of a lightweight multimodal processing system. Targeting the low computing power of low-computing-power computing nodes, it forms a lightweight design for the entire process from feature extraction to model optimization. Lightweight extractors are customized for images, text and speech respectively. Then, the hierarchical attention fusion model is optimized through structured pruning. While ensuring multimodal representation capabilities, it significantly reduces computation and storage overhead and adapts to the hardware constraints of low-computing-power computing nodes.

[0037] 2. To realize a hierarchical attention cross-modal fusion mechanism, this invention innovates a three-level attention structure. First, intra-modal attention filters redundant information from a single modality. Then, text is used as an anchor point to achieve accurate cross-modal feature association. Finally, global fusion attention filters key information and compresses dimensions, solving the problem that traditional fusion is difficult to associate heterogeneous semantics or has excessively high vector dimensions, thus improving the fusion effect.

[0038] 3. This invention achieves multi-task training and adaptation optimization of computing nodes. By designing a multi-task loss function, it simultaneously focuses on the matching degree of model output and the quality of intramodal features to avoid training target imbalance. At the same time, it optimizes the model deployment format, enabling the lightweight model to run upper-layer tasks efficiently on the local computing node, get rid of cloud dependence, reduce transmission latency, and balance accuracy and real-time performance. Attached Figure Description

[0039] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] Examples, such as Figure 1 As shown, a lightweight multimodal representation learning method based on a multi-layer attention mechanism includes the following steps:

[0043] S1. Acquire the raw multimodal data collected by the computing nodes, and preprocess the raw multimodal data to generate the basic features of each modality;

[0044] S2. Use a lightweight feature extractor to extract the high-level semantic features of each modality data from the basic features of each modality data, and construct a hierarchical attention mechanism model to perform cross-modal fusion of the high-level semantic features of each modality data through the hierarchical attention mechanism model to generate a low-dimensional cross-modal vector.

[0045] S3. Perform lightweight optimization on the hierarchical attention mechanism model to form a lightweight model;

[0046] S4. Train the lightweight model using a low-dimensional cross-modal vector as the training benchmark to obtain a lightweight multimodal model;

[0047] S5. Deploy lightweight multimodal models to compute nodes to meet the multimodal upper-layer task requirements of compute nodes.

[0048] Furthermore, the working principle of the present invention will be illustrated below through embodiments:

[0049] This embodiment uses a smartphone as the application platform and specifically implements the method of the present invention for the cross-modal retrieval task of intelligent object search using voice and image on the mobile device.

[0050] A 1080×1920 RGB image of a water cup was captured using the rear camera of a smartphone. The Chinese search keyword "white ceramic water cup" was entered into the input method, and the voice command "search for white water cup" was recorded at a 16kHz sampling rate using the microphone. Image data, text data, and voice data were obtained. The image data was normalized to mean and variance using ImageNet statistics, scaled to 224×224 resolution using bilinear interpolation, and then noise was removed using Gaussian blur with a kernel size of 3×3 to obtain the basic features of the 224×224×3 image data. For text data, the jieba lightweight word segmentation mode is used to divide "white ceramic water cup" into a discrete token sequence of ["white", "ceramic", "water cup"]. This sequence is then input into the pre-trained DistilBERT basic word embedding layer to generate a high-dimensional word vector matrix. The matrix is ​​then reduced to 256 dimensions by a shared linear projection layer, and short sequences are padded with zeros to obtain the basic features of the 32×256 text data. For speech data, the spectrum is extracted using a 512-point FFT short-time Fourier transform and converted into Mel frequency cepstral coefficients using a Mel filter bank to obtain the basic features of the speech data.

[0051] A simplified visual Transformer is used to process the basic features of image data, outputting high-level semantic features of 512-dimensional image data; a lightweight BERT variant is used to process the basic features of text data, outputting high-level semantic features of 512-dimensional text data; a temporal convolutional network is used to process the basic features of speech data, outputting high-level semantic features of 512-dimensional speech data; a hierarchical attention mechanism model is constructed, which filters redundant information in the high-level semantic features of each modality data through intra-modal attention layers, outputting local features of each modality data; through cross-modal attention layers, the local features of each modality data are projected to the same semantic space through a shared fully connected layer, and cosine similarity is calculated using text alignment features as anchor points to generate attention scores, which are then weighted and fused to obtain cross-modal association features; through a global fusion attention layer, the local features of each modality data and cross-modal association features are concatenated, and key information is filtered and the dimensions are compressed through a gating network to generate a 128-dimensional low-dimensional cross-modal vector.

[0052] The hierarchical attention mechanism model was optimized by structural pruning. For the intramodal attention layer, the contribution of each attention head was calculated using the L1 norm and the last 40% of attention heads were removed. The last 20% of redundant neurons in the output projection fully connected layer were removed through gradient norm sensitivity analysis. For the global fusion attention layer gating network, input channels with an activation value ratio of <5% and neurons with an absolute weight value of ≤1e-4 in the output layer were removed. This resulted in a lightweight multimodal model with 42% fewer parameters and a storage footprint reduced from 128MB to 74MB compared to the hierarchical attention mechanism model.

[0053] Using low-dimensional cross-modal vectors as the training benchmark, a lightweight model was trained using the Adam optimizer. A multi-task loss function was used to optimize the parameters of the lightweight model (output matching loss and intra-modal feature quality loss were weighted in a 1:1 ratio, output matching loss was L2 distance loss, and intra-modal feature quality loss was a 1:1 weighted sum of image high-level semantic feature classification loss and text high-level semantic feature masking language model loss). Training was stopped when the output matching loss decreased by less than 0.001 for three consecutive training epochs and the cosine similarity between the model output vector and the low-dimensional cross-modal vector was ≥0.9. The inference latency of the lightweight multimodal model on mobile devices was reduced from 180ms to 85ms.

[0054] The trained lightweight multimodal model is converted to .tflite format and deployed to the target smartphone using the TensorFlowLite framework. When performing the "voice + image intelligent object search" task, after the user takes a picture of the water cup and inputs a voice command, the model processes the data locally to generate a low-dimensional cross-modal vector, matches it with the local object feature library, and returns the result within 1 second, such as "white ceramic water cup, matching degree 92%". The inference process does not rely on cloud computing power, meeting the requirements of task accuracy and real-time performance.

[0055] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them; modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions of some of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A lightweight multimodal representation learning method based on a multi-layer attention mechanism, characterized in that, Includes the following steps: S1. Acquire the raw multimodal data collected by the computing node, and preprocess the raw multimodal data to generate the basic features of each modality. The raw multimodal data includes image data, text data, and speech data. S2. Use a lightweight feature extractor to extract the high-level semantic features of each modality data from the basic features of each modality data, and construct a hierarchical attention mechanism model to perform cross-modal fusion of the high-level semantic features of each modality data through the hierarchical attention mechanism model to generate a low-dimensional cross-modal vector. S3. Lightweight optimization of the hierarchical attention mechanism model to form a lightweight model. The hierarchical attention mechanism model has a three-level structure, specifically including an intra-modal attention layer, a cross-modal attention layer, and a global fusion attention layer. The intramodal attention layer is used to take the high-level semantic features of each modal data as input, filter redundant information in the high-level semantic features of each modal data through spatial attention, word-level self-attention and temporal attention, and output the local features of each modal data. The cross-modal attention layer is used to project the local features of each modality data into the same semantic space through a shared fully connected layer to obtain image alignment features, text alignment features and speech alignment features. The cosine similarity between the image alignment features and text alignment features, as well as between the speech alignment features and text alignment features, is calculated using the text alignment features as anchor points to generate attention scores. The cross-modal association features are obtained by weighted fusion according to the attention scores. The global fusion attention layer is used to concatenate the local features of each modality data with cross-modal correlation features, then filter key information by learning channel weights through a gating network, and output a low-dimensional cross-modal vector after dimensional compression. The hierarchical attention mechanism model is optimized for lightweighting through structural pruning. The specific process includes: calculating the contribution of each attention head in the intramodal attention layer for spatial attention, word-level self-attention, and temporal attention using the L1 norm, removing the attention heads with the lowest contribution ranking of 30%-50%, and performing sensitivity analysis on the output projection fully connected layers of spatial attention, word-level self-attention, and temporal attention using the gradient norm, removing 20% ​​of redundant neurons in the output projection fully connected layers. For the gated network with a global fusion attention layer, remove input channels with an activation value ratio of <5% in the input channels of the input layer of the gated network, and remove neurons with an absolute weight value of ≤1e-4 in the output layer of the gated network. S4. Train the lightweight model using a low-dimensional cross-modal vector as the training benchmark to obtain a lightweight multimodal model; S5. Deploy lightweight multimodal models to compute nodes to meet the multimodal upper-layer task requirements of compute nodes.

2. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 1, characterized in that, The method for preprocessing the raw multimodal data is as follows: ImageNet statistics were used to normalize the mean and variance of the image data to obtain standardized RGB image data. After adaptively scaling the standardized RGB image data to 224×224 resolution by bilinear interpolation, Gaussian blur with kernel size of 3×3 was used to remove image noise and obtain the basic features of the image data. The text data is segmented using the jieba lightweight segmentation mode or the SentencePiece lightweight segmentation version to obtain discrete token sequences. These sequences are then input into a pre-trained word embedding model to obtain a high-dimensional word vector matrix. Finally, the high-dimensional word vector matrix is ​​reduced in dimensionality by linear projection through a shared linear projection layer to obtain the basic features of the text data. The speech data is subjected to short-time Fourier transform to extract the spectrum, and the spectrum is converted into Mel frequency cepstral coefficients through Mel filter bank to output the Vimel spectrum features, which are the basic features of the speech data.

3. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 2, characterized in that, The method for extracting high-level semantic features from each modality of data is as follows: For image data, a simplified visual Transformer is used to extract high-level semantic features; for text data, a lightweight BERT variant is used to extract high-level semantic features; and for speech data, a temporal convolutional network is used to extract high-level semantic features. The simplified visual Transformer, the lightweight BERT variant, and the temporal convolutional network are all lightweight feature extractors.

4. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 3, characterized in that, The lightweight model is trained using a multi-task loss function. This multi-task loss function consists of a weighted sum of output matching loss and intra-modal feature quality loss, with a weight ratio of 1:(0.8-1.2). The output matching loss is the L2 distance loss between the lightweight model's output vector and the low-dimensional cross-modal vector, and the intra-modal feature quality loss is the weighted sum of the image high-level semantic feature classification loss and the text high-level semantic feature masking language model loss.

5. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 4, characterized in that, The termination condition for training the lightweight model using a low-dimensional cross-modal vector as the training benchmark is: when the output matching loss decreases by less than 0.001 for three consecutive training cycles, and the cosine similarity between the model output vector and the low-dimensional cross-modal vector after training is ≥0.

9.

6. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 5, characterized in that, When the lightweight multimodal model is deployed to a computing node, it is converted to .tflite or .onnx format.

7. The lightweight multimodal representation learning method based on a multi-layer attention mechanism according to claim 6, characterized in that, The multimodal upper-layer tasks include cross-modal retrieval tasks and multimodal intent recognition tasks.

Images