Cross-modal attention fusion method and device for multi-modal AI large model

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a cross-modal attention fusion method and device for a multi-modal AI large model. The method comprises the following steps: acquiring a multi-modal data set containing texts, images and audios; generating a dynamic weight distribution matrix based on a cross-modal attention mechanism; performing lightweight fusion processing of parameter pruning, quantization coding and feature dimension reduction on the multi-modal data; inputting the lightweight data and the matrix into a hardware collaborative optimization module to generate cross-modal fusion features; and iteratively adjusting the weight distribution matrix through a cross-modal attention loss function, and outputting a cross-modal attention fusion model. According to the method, the correlation between modes can be accurately mined, the model complexity is reduced through lightweight processing, the calculation efficiency is improved through hardware collaborative optimization, the model adaptability is improved through iterative optimization, and the method has wide application prospects in the fields of intelligent security and protection, education, medical treatment and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a cross-modal attention fusion method and device of a multi-modal AI large model. BACKGROUND

[0002] In the current booming development of artificial intelligence, multi-modal data processing has become an important research direction. Multi-modal data contains various types such as text, image, audio, etc., and each modality contains unique and complementary information. However, the current multi-modal data fusion technology still faces many challenges.

[0003] From the perspective of data fusion, the feature representations of different modal data are quite different. Text modality exists in the form of discrete symbolic language, and the extraction and fusion of semantic encoding vectors and context-related features are complex, and traditional methods are difficult to effectively capture complex semantic dependency relationships in long sequence texts. The data of the image modality is based on a pixel matrix, and the processing of spatial encoding vectors and local texture features needs to consider global and local information. The existing technology consumes a lot of computing resources and the feature extraction is not accurate when processing high-resolution and complex scene images. The audio modality has a time sequence, and the sound spectrum encoding vector and the time sequence beat feature are easily disturbed by environmental noise, and different audio sampling rates and formats also increase the processing difficulty, making it difficult to accurately fuse audio features.

[0004] In terms of model performance, as the scale and complexity of multi-modal data continue to increase, the computational load and storage requirements of the model grow exponentially. Traditional fusion models often have serious parameter redundancy, and in practical applications, the high computational cost limits the deployment range of the model, especially on resource-constrained edge devices, making it difficult to achieve real-time and efficient multi-modal data processing. At the same time, the training efficiency of the model also needs to be improved, and the long training process not only consumes a lot of manpower and computing resources, but also may cause overfitting phenomenon in the training process, making the model lack of generalization ability in practical applications.

[0005] Although the cross-modal attention mechanism improves the multi-modal data fusion effect to some extent, the existing attention calculation method has limitations. When calculating the correlation score between modalities, it is difficult to fully consider the complex nonlinear relationship between modalities, resulting in inaccurate attention weight distribution, which cannot effectively focus on key information, and thus affects the quality of multi-modal data fusion and model performance. Moreover, different modal data have poor computational adaptability on hardware devices, which cannot fully utilize the parallel computing capability of hardware computing units, resulting in waste of hardware resources.

[0006] In the face of these problems, it is urgent to develop a technology that can efficiently fuse multi-modal data, reduce model complexity, improve computing efficiency, and accurately allocate cross-modal attention, which will lay a solid foundation for the widespread application of multi-modal artificial intelligence in intelligent security, intelligent medical care, intelligent education, and many other fields. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal AI large model cross-modal attention fusion method and device to solve the problems raised in the background art.

[0008] To achieve the above-mentioned purpose, the present application provides the following technical solution: a multi-modal AI large model cross-modal attention fusion method, the method comprising:

[0009] Obtain a multi-modal data set; the multi-modal data includes text modal data, image modal data and audio modal data; the text modal data contains semantic encoding vectors and context association features, the image modal data contains spatial encoding vectors and local texture features, and the audio modal data contains sound spectrum encoding vectors and timing beat features;

[0010] Based on the multi-modal data, a dynamic weight distribution matrix is generated through a cross-modal attention mechanism; the dynamic weight distribution matrix contains inter-modal correlation scores and cross-modal attention weights;

[0011] The multi-modal data is subjected to lightweight fusion processing, including parameter pruning, quantization encoding and feature dimension reduction;

[0012] The multi-modal data after lightweight processing and the dynamic weight distribution matrix are input into a pre-set hardware co-optimization module to generate cross-modal fusion features;

[0013] Based on the cross-modal fusion features, the weight distribution matrix is iteratively adjusted through a cross-modal attention loss function, and a cross-modal attention fusion model is output.

[0014] Preferably, the cross-modal attention mechanism generates a dynamic weight distribution matrix, comprising:

[0015] The text modal data, image modal data and audio modal data are respectively subjected to modal feature encoding to generate text feature tensors, image feature tensors and audio feature tensors;

[0016] The cross-modal correlation scores between the modal feature tensors are calculated based on a multi-head attention mechanism;

[0017] The cross-modal attention weights are generated according to the correlation scores and a pre-set dynamic weight threshold;

[0018] The cross-modal attention weight is matrix-spliced with each modal feature tensor to form a dynamic weight distribution matrix.

[0019] Preferably, the parameter pruning in the lightweight fusion processing includes:

[0020] A redundant parameter in the model is identified based on gradient sensitivity analysis, and a parameter pruning mask is generated.

[0021] The model weight matrix is sparsified according to the parameter pruning mask, and a pruned weight matrix is generated.

[0022] The pruned weight matrix is quantized and encoded with a low bit to generate lightweight weight parameters.

[0023] Preferably, the processing process of the hardware cooperative optimization module includes:

[0024] The multi-modal data after lightweight processing is divided into parallel computing subgraphs according to the modal type;

[0025] The subgraphs are dynamically distributed to GPUs or TPUs for parallel computing according to the types of hardware computing units.

[0026] The parallel computing results are tensor-spliced and normalized to generate cross-modal fusion features.

[0027] Preferably, the cross-modal correlation score between each modal feature tensor is calculated based on the multi-head attention mechanism, including:

[0028] The text feature tensor, the image feature tensor and the audio feature tensor are linearly projected to generate query vectors, key vectors and value vectors, respectively.

[0029] The dot product correlation score of each modal query vector and key vector is calculated through the multi-head attention score.

[0030] The dot product correlation score is scaled and normalized to generate a cross-modal attention distribution map.

[0031] Preferably, the implementation of the modal feature encoding includes:

[0032] The hierarchical Transformer encoder is used to extract semantic encoding vectors for text modal data, the convolution-attention hybrid network is used to extract spatial encoding vectors for image modal data, and the time-frequency transform encoder is used to extract sound spectrum encoding vectors for audio modal data.

[0033] Preferably, the adjustment method of the dynamic weight distribution includes:

[0034] An initial weight distribution matrix is generated according to the cross-modal attention distribution map and a preset task objective function.

[0035] calculating the gradient sensitivity of the weight distribution matrix through a back propagation algorithm;

[0036] dynamically updating the weight distribution matrix based on the gradient sensitivity to generate an optimized dynamic weight distribution matrix.

[0037] Preferably, the implementation of the low-bit quantization coding comprises: segmenting linear quantization of the pruned weight matrix to generate integer weight parameters, performing precision reservation processing on high sensitivity weights through a non-uniform quantization algorithm, and dynamically aligning the quantized weight parameters with the activation function to generate light-weight model parameters.

[0038] Preferably, the generation method of the parallel computing subgraph comprises: dividing independent computing subgraphs according to the calculation dependency relationship of the multi-modal data, performing operator fusion and memory access optimization on each subgraph, and generating an adaptive parallel computing instruction set.

[0039] Preferably, the application further comprises a cross-modal attention fusion device of a multi-modal AI large model, the device comprising:

[0040] a data acquisition module for acquiring a multi-modal data set, the multi-modal data comprising text modal data, image modal data and audio modal data, wherein the text modal data contains semantic encoding vectors and context association features, the image modal data contains spatial encoding vectors and local texture features, and the audio modal data contains sound spectrum encoding vectors and timing beat features;

[0041] a weight generation module connected to the data acquisition module, for generating a dynamic weight distribution matrix based on the multi-modal data through a cross-modal attention mechanism, the dynamic weight distribution matrix comprising inter-modal correlation scores and cross-modal attention weights;

[0042] a light-weight processing module connected to the data acquisition module, for performing light-weight fusion processing on the multi-modal data, including parameter pruning, quantization coding and feature dimension reduction;

[0043] a fusion processing module connected to the light-weight processing module and the weight generation module, for inputting the light-weight processed multi-modal data and the dynamic weight distribution matrix into a preset hardware co-optimization module to generate cross-modal fusion features;

[0044] a model optimization module connected to the fusion processing module and the weight generation module, for iteratively adjusting the weight distribution matrix based on the cross-modal fusion features through a cross-modal attention loss function, and outputting a cross-modal attention fusion model.

[0045] Compared with the prior art, the application has the following advantages:

[0046] In terms of data fusion, the innovative cross-modal attention mechanism can deeply explore the complex correlations among text, image, and audio modalities. Specifically, each modality data is first encoded with a targeted feature encoder, such as a hierarchical Transformer encoder for text semantic encoding vectors, a convolution-attention hybrid network for image spatial encoding vectors, and a time-frequency transform encoder for audio spectrum encoding vectors, to ensure accurate feature extraction. Then, based on the multi-head attention mechanism, cross-modal correlation scores are calculated, and after scaling and normalization, cross-modal attention distribution maps are obtained. The cross-modal attention weights generated from these maps can accurately focus on key information, enabling efficient collaboration among modalities. For example, in an intelligent security scenario, when an abnormal situation occurs in the monitoring screen, the model can analyze the human actions in the video image (image modality), the on-site environmental sound (audio modality), and the related event description text (text modality), accurately determine the nature of the event, and greatly improve the accuracy and completeness of multi-modal data fusion, avoiding misjudgment due to insufficient single-modal information.

[0047] From the perspective of model optimization, lightweight fusion processing plays a key role. In the parameter pruning stage, redundant parameters are identified based on gradient sensitivity analysis, a pruning mask is generated to sparsify the model weight matrix, unnecessary parameters are removed, and the model complexity is reduced. Then, the pruned weight matrix is quantized with low bits, reducing storage requirements and computational load while preserving key information. When deploying multi-modal models on mobile devices, the lightweight processed models significantly reduce memory usage and significantly improve running speed, while still maintaining high accuracy, achieving efficient compression and acceleration of models, and improving the practicality and scalability of models.

[0048] The hardware cooperative optimization module further improves the computing efficiency. It divides the lightweight processed multi-modal data into parallel computing subgraphs according to the modality type, dynamically allocates tasks according to the types of hardware computing units such as GPU or TPU, and fully utilizes the parallel computing capabilities of hardware. For example, when processing large-scale image and audio data, GPU is good at parallel computing of image data, and TPU has advantages in tensor operations. By reasonably allocating computing tasks, optimal allocation of computing resources can be achieved. The parallel computing results are processed through tensor splicing and normalization to generate high-quality cross-modal fusion features, significantly improving the processing speed of the model without increasing hardware costs, meeting the needs of application scenarios with high real-time requirements.

[0049] The weight distribution matrix can be continuously optimized by iterative adjustment through the cross-modal attention loss function. As the training data increases and the task requirements change, the model can continuously optimize the dynamic weight distribution according to the cross-modal fusion features, improving the adaptability and generalization ability of the model. In the field of intelligent education, different students have different learning habits and knowledge mastery levels. The model can dynamically adjust the attention weight according to the multi-modal data generated by students during the learning process (such as learning videos, voice answers, and text assignments), accurately analyze the learning situation of students, provide personalized learning suggestions and guidance, and improve the educational effect. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The working principle diagram of the cross-modal attention fusion method of the multi-modal AI large model is shown in the figure.

[0051] Figure 2 The flowchart of parameter pruning in lightweight fusion processing is shown in the figure.

[0052] Figure 3 The flowchart of hardware cooperative optimization module processing is shown in the figure.

[0053] Figure 4 The flowchart of low-bit quantization encoding implementation is shown in the figure. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] Please refer to Figures 1-4 The present application relates to a cross-modal attention fusion method of a multi-modal AI large model, and the specific embodiments are described in detail below.

[0056] A multi-modal data set containing text modal data, image modal data and audio modal data is obtained. The text modal data contains semantic encoding vectors and context association features, which are used to express the meaning of the text and the relationship between the parts of the text. The image modal data contains spatial encoding vectors and local texture features. The spatial encoding vectors reflect the position and layout information of the objects in the image, and the local texture features represent the detailed features of the local regions in the image. The audio modal data contains sound spectrum encoding vectors and timing beat features. The sound spectrum encoding vectors can represent the energy distribution of the audio signal at different frequencies, and the timing beat features record the rhythm and time sequence information of the audio.

[0057] A dynamic weight distribution matrix is generated based on the acquired multi-modal data using a cross-modal attention mechanism. This matrix contains inter-modal correlation scores and cross-modal attention weights, which measure the degree of association between different modal data and determine the importance of each modal data in the fusion process.

[0058] The multi-modal data is subjected to lightweight fusion processing, including parameter pruning, quantization encoding, and feature dimension reduction. Parameter pruning removes redundant parameters in the model, reducing the computational load and storage requirements. Quantization encoding converts parameters into low-bit representations, further reducing computational complexity. Feature dimension reduction reduces the dimensionality of the data while preserving key information, improving processing efficiency.

[0059] The lightweight processed multi-modal data and the generated dynamic weight distribution matrix are input into a pre-set hardware co-optimization module. This module utilizes the parallel computing capabilities of hardware to process the data and generate cross-modal fusion features that integrate the strengths of different modal data.

[0060] Based on the generated cross-modal fusion features, a cross-modal attention loss function is used to evaluate the performance of the current model. By iteratively adjusting the weight distribution matrix, the model is continuously optimized, and finally a cross-modal attention fusion model is output. This model can more effectively process multi-modal data, achieving more accurate task prediction and analysis.

[0061] The following examples 1 to 5 further illustrate the application:

[0062] Example 1:

[0063] In this example, the process of generating a dynamic weight distribution matrix using a cross-modal attention mechanism is detailed. First, the text modal data, image modal data, and audio modal data are respectively subjected to modal feature encoding. For text modal data, a hierarchical Transformer encoder is used to extract semantic encoding vectors. The hierarchical Transformer encoder can extract multi-level features from text, from word-level features to sentence-level features, thus capturing more comprehensive semantic information. For image modal data, a convolution-attention hybrid network is used to extract spatial encoding vectors. The convolution network can extract local features of the image, while the attention mechanism can focus on important regions of the image, enhancing the ability to extract key information. For audio modal data, a time-frequency transform encoder is used to extract spectrogram encoding vectors. The time-frequency transform encoder converts audio signals from the time domain to the frequency domain, extracting features at different frequencies and times to form spectrogram encoding vectors.

[0064] After modal feature encoding, the text feature tensor , image feature tensor and audio feature tensor The cross-modal correlation score between feature tensors of different modalities is calculated based on a multi-head attention mechanism. The specific steps are as follows: [The text feature tensor is then processed]. Image feature tensor and audio feature tensor Perform linear projections separately to generate query vectors. Key vector and value vector Assume the text feature tensor The dimension is ,in Indicates batch size. Indicates the length of the text sequence. Representing feature dimension; image feature tensor The dimension is , Represents the number of image features; audio feature tensor The dimension is , Represents the number of audio features. This is achieved through a linear projection matrix. , , To each , , Project the vector to obtain the query vector. key vector value vector .here , , The dimension is determined based on the target dimension of the projection.

[0065] Next, the dot product relevance score between the query vector and the key vector for each modality is calculated using multi-head attention scores. The multi-head attention mechanism, by calculating attention scores in parallel across multiple heads, can capture feature relationships from different perspectives. The formula for calculating the dot product relevance score is: ,in Indicates the first The query vector and the first The dot product correlation score of the key vectors Indicates the first A query vector, Indicates the first A key vector, The dimension of the key vector is represented. The dot product correlation scores are scaled and normalized to generate a cross-modal attention distribution map. The scaling operation is to avoid the dot product result being too large, which would lead to gradient instability, while the normalization operation ensures that the attention scores can represent the probability distribution.

[0066] The cross-modal attention weight is generated according to the correlation score and a preset dynamic weight threshold. When the correlation score is greater than the preset threshold, it is considered that the correlation between the modal pairs is high, and the corresponding attention weight is increased. The cross-modal attention weight is spliced with each modal feature tensor to form a dynamic weight distribution matrix. In this way, the dynamic weight distribution matrix can reflect the correlation degree and importance between different modalities, providing a basis for subsequent fusion processing.

[0067] Embodiment 2

[0068] This embodiment focuses on the parameter pruning link in lightweight fusion processing. First, based on gradient sensitivity analysis, the redundant parameters in the model are identified, and a parameter pruning mask is generated. During model training, the gradient of a parameter can reflect its influence on the model output. Parameters with high gradient sensitivity have a greater impact on model performance, while parameters with low gradient sensitivity may be redundant. By calculating the gradient sensitivity of each parameter, a threshold is set. When the gradient sensitivity of a parameter is lower than the threshold, it is considered to be redundant, and the corresponding position in the parameter pruning mask is set to 0, otherwise it is set to 1. Assuming that the parameter matrix of the model is , the parameter pruning mask is , where represents the mask value corresponding to the element in the row and column of the parameter matrix .

[0069] According to the parameter pruning mask, the model weight matrix is sparsified to generate a pruned weight matrix. The specific operation is to multiply the parameter pruning mask and the model weight matrix element by element, that is, , to obtain the pruned weight matrix . After sparsification, many redundant parameters in the pruned weight matrix are set to 0, reducing the computational complexity and storage requirements of the model.

[0070] The pruned weight matrix is low-bit quantization encoded to generate lightweight weight parameters. Low-bit quantization encoding is to convert the parameters in the pruned weight matrix to low-bit representation to further reduce the computational complexity. The specific implementation includes segmenting the pruned weight matrix for linear quantization to generate integer weight parameters. Assuming that the weight value range is divided into intervals, each interval corresponds to a quantization value . For a weight value , its quantization value is obtained by the formula Determination. The high sensitivity weight is processed by precision reservation through a non-uniform quantization algorithm. For weights with high gradient sensitivity, higher precision is given during quantization to minimize information loss caused by quantization. The quantized weight parameters are aligned with the activation function in dynamic range to generate lightweight model parameters. Through dynamic range alignment, the quantized weight parameters are ensured to match the input range of the activation function, ensuring the normal operation of the model.

[0071] Embodiment 3

[0072] This embodiment details the processing process of the hardware cooperative optimization module. First, the multi-modal data after lightweight processing is divided into parallel computing subgraphs according to the modal type. According to the calculation dependency relationship of multi-modal data, independent computing subgraphs are divided, such as word vector calculation in text modal data, convolution calculation in image modal data, and time-frequency transformation calculation in audio modal data. These calculation operations are independent of each other and can be divided into different subgraphs. Each subgraph is subjected to operator fusion and memory access optimization to generate an adaptive parallel computing instruction set. Operator fusion is to combine multiple adjacent calculation operators into a more efficient operator to reduce data transmission and intermediate result storage in the calculation process. Memory access optimization improves the efficiency of data access by reasonably arranging the storage layout of data in memory.

[0073] According to the type of hardware computing unit, the subgraphs are dynamically allocated to GPU or TPU for parallel computing. GPU (Graphics Processing Unit) has strong parallel computing capability and is suitable for processing large-scale matrix operations such as convolution calculation and matrix multiplication; TPU (Tensor Processing Unit) is optimized for tensor operations in deep learning and can more efficiently handle neural network computing tasks. For convolution calculation subgraphs in image modal data, they can be allocated to GPU for calculation; for Transformer calculation subgraphs in text modal data, they can be allocated to GPU or TPU for calculation according to specific conditions. By dynamically allocating subgraphs, the advantages of different hardware computing units can be fully utilized to improve computing efficiency.

[0074] The parallel computing results are subjected to tensor splicing and normalization processing to generate cross-modal fusion features. After the completion of calculation on different hardware computing units, the results obtained by different subgraphs are multiple tensors. These tensors are spliced in a certain order to form a complete tensor. Then the spliced tensor is subjected to normalization processing to make data of different modalities on the same scale for fusion. Normalization processing can adopt various methods, such as L2 normalization, the formula of which is: where represents the input tensor, represents the normalized tensor, represents the tensor in the The cross-modal fusion features are generated by tensor splicing and normalization processing, which fuse the advantage information of different modal data, and provide more effective data for subsequent model training and task prediction.

[0075] Embodiment 4:

[0076] In this embodiment, the adjustment method of dynamic weight distribution is discussed in depth. According to the cross-modal attention distribution map and the preset task objective function, an initial weight distribution matrix is generated. The cross-modal attention distribution map reflects the correlation degree between different modal features, and the task objective function is determined according to the specific task requirements, such as the cross-entropy loss function of the classification task and the mean square error loss function of the regression task. By combining the cross-modal attention distribution map and the task objective function, the initial weight distribution matrix can be determined, so that the model can perform weight distribution according to the task requirements and the correlation degree of the modal in the initial stage.

[0077] The gradient sensitivity of the weight distribution matrix is calculated by the back propagation algorithm. The back propagation algorithm is a common optimization algorithm in deep learning, which updates the model parameters by calculating the gradient of the loss function with respect to the model parameters. In this invention, the gradient of the cross-modal attention loss function with respect to the weight distribution matrix is calculated to determine the gradient sensitivity of each weight. Assuming that the cross-modal attention loss function is , the weight distribution matrix is , and the gradient calculated by the back propagation algorithm is , which reflects the gradient sensitivity of each element in the weight distribution matrix.

[0078] Based on the gradient sensitivity, the weight distribution matrix is dynamically updated to generate an optimized dynamic weight distribution matrix. According to the calculated gradient sensitivity, the weight distribution matrix is updated. For the weights with high gradient sensitivity, a larger step size is given in the update to speed up the convergence of the model; for the weights with low gradient sensitivity, the update step size is appropriately reduced to avoid over-adjustment. The update formula can be expressed as: , where represents the updated weight distribution matrix, represents the current weight distribution matrix, represents the learning rate, which is used to control the size of the update step. By continuously iterating the weight distribution matrix, the model can better adapt to the task requirements and improve the performance of the model.

[0079] Embodiment 5:

[0080] The embodiment focuses on the detailed implementation process of low-bit quantization coding. When performing low-bit quantization coding on the pruned weight matrix, first, segment linear quantization is performed to generate integer weight parameters. Assume that the weight value range of the pruned weight matrix is divided into intervals , and each interval corresponds to a quantization value . For the weight value , its quantization value is determined by the formula . For example, if the weight value falls within the interval , and the quantization value corresponding to the interval is , then the quantization value of is . In this way, continuous weight values are converted into discrete integer weight parameters.

[0081] The high sensitivity weights are processed by the non-uniform quantization algorithm for precision retention. Based on the previous gradient sensitivity analysis to identify redundant parameters, for weights with high gradient sensitivity, the non-uniform quantization algorithm is used to give higher precision during quantization. The non-uniform quantization algorithm can adjust the quantization interval according to the importance of the weight. For important weights, finer quantization intervals are divided to reduce quantization error.

[0082] The quantized weight parameters are aligned with the activation function for dynamic range, and the lightweight model parameters are generated. The activation function plays an important role in neural networks, as it can introduce non-linear factors and enhance the model's expression ability. Different activation functions have different input and output ranges. In order to ensure that the quantized weight parameters match the input range of the activation function, dynamic range alignment is needed. For example, for the Sigmoid activation function, the input range is usually , and the output range is . By scaling and shifting the quantized weight parameters, they can be adapted to the input range of the Sigmoid activation function, thereby generating lightweight model parameters and ensuring that the model can maintain good performance under low-bit quantization.

[0083] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0084] While the embodiments of the application have been shown and described herein, it will be understood by those of ordinary skill in the art that various changes, modifications, alternatives, and variations can be made to the embodiments without departing from the spirit and scope of the application, which is defined by the appended claims and their equivalents.

Claims

1. A cross-modal attention fusion method for a large multimodal AI model, characterized in that, include: Obtain a multimodal data set; the multimodal data includes text modal data, image modal data and audio modal data; the text modal data contains semantic encoding vectors and contextual features, the image modal data contains spatial encoding vectors and local texture features, and the audio modal data contains spectrogram encoding vectors and temporal beat features; Based on the multimodal data, a dynamic weight allocation matrix is ​​generated through a cross-modal attention mechanism; the dynamic weight allocation matrix includes intermodal correlation scores and cross-modal attention weights. The multimodal data is subjected to lightweight fusion processing, including parameter pruning, quantization encoding, and feature dimensionality reduction; The lightweight multimodal data and dynamic weight allocation matrix are input into a preset hardware co-optimization module to generate cross-modal fusion features. Based on the cross-modal fusion features, the weight allocation matrix is ​​iteratively adjusted through the cross-modal attention loss function to output the cross-modal attention fusion model.

2. The cross-modal attention fusion method for a large multimodal AI model according to claim 1, characterized in that, The cross-modal attention mechanism generates a dynamic weight allocation matrix, including: Modal feature encoding is performed on the text modal data, image modal data, and audio modal data respectively to generate text feature tensors, image feature tensors, and audio feature tensors; The cross-modal correlation score between feature tensors of each modality is calculated based on the multi-head attention mechanism; Cross-modal attention weights are generated based on the correlation scores and preset dynamic weight thresholds; The cross-modal attention weights are concatenated with the feature tensors of each modality to form a dynamic weight allocation matrix.

3. The cross-modal attention fusion method for a large multimodal AI model according to claim 1, characterized in that, The parameter pruning in the lightweight fusion process includes: Based on gradient sensitivity analysis, redundant parameters in the model are identified, and parameter pruning masks are generated. The model weight matrix is ​​sparsified according to the parameter pruning mask to generate the pruned weight matrix. The pruned weight matrix is ​​low-bit quantization encoded to generate lightweight weight parameters.

4. The cross-modal attention fusion method for a large multimodal AI model according to claim 1, characterized in that, The processing steps of the hardware collaborative optimization module include: The lightweighted multimodal data is divided into parallel computing subgraphs according to modality type; Dynamically allocate subgraphs to GPUs or TPUs for parallel computing based on the type of hardware computing unit; The parallel computation results are subjected to tensor splicing and normalization to generate cross-modal fusion features.

5. The cross-modal attention fusion method for a large multimodal AI model according to claim 2, characterized in that, The calculation of cross-modal correlation scores between feature tensors of each modality based on the multi-head attention mechanism includes: Linear projection is performed on the text feature tensor, image feature tensor, and audio feature tensor respectively to generate query vector, key vector, and value vector; The dot product relevance score between the query vector and the key vector for each modality is calculated using multi-head attention scores. The dot product correlation scores are scaled and normalized to generate a cross-modal attention distribution map.

6. The cross-modal attention fusion method for a large multimodal AI model according to claim 2, characterized in that, The implementation of the modal feature encoding includes: A hierarchical Transformer encoder is used to extract semantic coding vectors for text modal data, a convolutional-attention hybrid network is used to extract spatial coding vectors for image modal data, and a time-frequency transform encoder is used to extract spectrogram coding vectors for audio modal data.

7. The cross-modal attention fusion method for a large multimodal AI model according to claim 2, characterized in that, The adjustment method for the dynamic weight allocation includes: An initial weight allocation matrix is ​​generated based on the cross-modal attention distribution map and the preset task objective function; The gradient sensitivity of the weight assignment matrix is ​​calculated using the backpropagation algorithm; The weight allocation matrix is ​​dynamically updated based on gradient sensitivity to generate an optimized dynamic weight allocation matrix.

8. The cross-modal attention fusion method for a large multimodal AI model according to claim 3, characterized in that, The implementation of the low-bit quantization encoding includes: performing piecewise linear quantization on the pruned weight matrix to generate integer weight parameters; performing precision preservation processing on the high-sensitivity weights through a non-uniform quantization algorithm; and dynamically aligning the quantized weight parameters with the activation function to generate lightweight model parameters.

9. The cross-modal attention fusion method for a large multimodal AI model according to claim 4, characterized in that, The method for generating the parallel computing subgraph includes: dividing the independent computing subgraph according to the computing dependencies of multimodal data, performing operator fusion and memory access optimization on each subgraph, and generating an adapted parallel computing instruction set.

10. A cross-modal attention fusion device for a large multimodal AI model, characterized in that, include: The data acquisition module is used to acquire a multimodal data set, which includes text modal data, image modal data and audio modal data. The text modal data includes semantic encoding vectors and contextual features, the image modal data includes spatial encoding vectors and local texture features, and the audio modal data includes spectrogram encoding vectors and temporal beat features. The weight generation module is connected to the data acquisition module. Based on the multimodal data, it generates a dynamic weight allocation matrix through a cross-modal attention mechanism. The dynamic weight allocation matrix includes intermodal correlation scores and cross-modal attention weights. The lightweight processing module is connected to the data acquisition module and performs lightweight fusion processing on the multimodal data, including parameter pruning, quantization encoding, and feature dimensionality reduction. The fusion processing module connects the lightweight processing module and the weight generation module, and inputs the lightweight processed multimodal data and dynamic weight allocation matrix into the preset hardware collaborative optimization module to generate cross-modal fusion features. The model optimization module connects the fusion processing module and the weight generation module. Based on the cross-modal fusion features, it iteratively adjusts the weight allocation matrix through the cross-modal attention loss function and outputs the cross-modal attention fusion model.

Images