A Multimodal Parallel Optimization Computation Method and Apparatus Based on Feature Dimensionality Reduction and Aggregation

Abstract

This application relates to a multimodal parallel optimization computation method and apparatus based on feature dimensionality reduction and aggregation. The method includes: obtaining a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix; performing dimensionality reduction and attention computation on the first and second modality embedding layer parameter matrices to obtain the importance matrix of all attention heads; selecting a preset number of attention heads to be retained from all attention heads according to the importance matrix of each attention head, a preset attention head discard rate, and a preset attention head performance threshold to obtain a set of retained attention heads; performing intermodal fusion, intermodal similar feature aggregation computation, and intermodal attention decision processing based on the set of retained attention heads to obtain a mask matrix of the attention score matrix; and reorganizing the high-precision attention computation based on the mask matrix and extracting the feature vectors from the corresponding second modality key matrix for floating-point calculation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimodal parallel optimization computation method, apparatus, computer equipment, storage medium, and computer program product based on feature dimensionality reduction and aggregation. Background Technology

[0002] A multimodal model is a machine learning or deep learning model capable of processing data from multiple different modalities (such as text, images, audio, etc.). In the real world, many problems involve multiple data sources or data types. For example, social media content may contain text, images, and videos, while medical diagnosis may involve medical images and clinical data. Multimodal models can handle complex multi-source data and capture the inherent correlations between data from different modalities. They have already been widely applied in many fields (such as text-to-text retrieval, text-to-text question answering, and sentiment recognition).

[0003] Specifically, commonly used multimodal models generally include two main components: intramodal encoding and intermodal fusion. Intramodal encoding refers to encoding the information within each data modality and mapping this information to a common representation space for subsequent fusion and analysis. Intermodal fusion combines information from different modalities to obtain a more comprehensive and richer feature representation, thereby improving the model's performance and generalization ability for various downstream tasks with multimodal features. In the intermodal fusion stage, the model considers the correlation and importance between different modalities to effectively fuse this information. Attention mechanisms play a crucial role in both intramodal encoding and intermodal fusion. In intramodal encoding, attention mechanisms within a modality (such as text, images, etc.) help capture the relationships between different features, thus better representing the information within that modality. In intermodal fusion, it helps the model dynamically focus on relevant information between different modalities, assigning different attention weights based on the importance of each modality, thereby weighted fusion of information from different modalities. However, the significant performance improvement of multimodal models comes at the cost of their high computational cost. Although multimodal models perform well in a variety of scenarios, their computational performance faces significant challenges during the inference and deployment phase. The number of parameters in multimodal models is generally in the billions or even tens of billions, which places enormous demands on computing resources and often makes it difficult to meet the requirements of online tasks for low latency and high throughput.

[0004] Traditional techniques have rarely been studied in terms of optimizing and accelerating the computation of multimodal models, and there is currently no efficient solution. Summary of the Invention

[0005] Therefore, it is necessary to provide an efficient multimodal parallel optimization computation method, apparatus, computer equipment, storage medium, and computer program product based on feature dimensionality reduction and aggregation to address the aforementioned technical problems.

[0006] Firstly, this application provides a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation. The method includes:

[0007] Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer;

[0008] Dimensionality reduction and attention calculation are performed on the parameter matrices of the first and second modality embedding layers to obtain the importance matrices of all attention heads;

[0009] Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads;

[0010] Based on the retained set of attention heads, intermodal fusion, intermodal similarity feature aggregation calculation, and intermodal attention decision processing are performed to obtain the mask matrix of the attention score matrix;

[0011] Based on the mask matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modal key matrix are extracted for floating-point calculation.

[0012] In one embodiment, the dimensionality reduction and attention calculation of the first and second modality embedding layer parameter matrices to obtain the importance matrix of all attention heads includes:

[0013] The parameter matrices of the first modality embedding layer and the second modality embedding layer are reduced in dimension based on a preset number of attention heads and a preset dimension reduction dimension, respectively, to obtain the dimension-reduced matrix representations.

[0014] Attention is calculated on the matrix representation after dimensionality reduction for each attention head, and the attention score output of each word of each attention head after dimensionality reduction is accumulated to obtain the importance matrix of all attention heads.

[0015] In one embodiment, the dimensionality reduction of the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, to obtain the dimensionality-reduced matrix representation includes:

[0016] Obtain the preset number of attention heads h and the preset dimensionality reduction dimension;

[0017] The first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively fed into N and M transformer blocks to start intramodal encoding, and each transformer block has h attention heads;

[0018] For each attention head, a dimensionality reduction dimension is selected based on the preset dimensionality reduction dimension, and a dimensionality reduction operation is performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the dimensionality-reduced matrix representation.

[0019] In one embodiment, the mask matrix for obtaining the attention score matrix based on the retained attention head set includes:

[0020] Based on the retained set of attention heads, obtain the corresponding retained first modality feature vector and second modality feature vector;

[0021] Perform matrix transformations on the first modality eigenvector and the second modality eigenvector respectively;

[0022] An attention-cross computation method is used to aggregate intermodal similarity features.

[0023] Based on the aggregation calculation results of inter-modal similarity features and the preset attention score matrix threshold, inter-modal attention decision is performed to obtain the mask matrix of the attention score matrix.

[0024] In one embodiment, the inter-modal similarity feature aggregation calculation using attention cross-computation includes:

[0025] The attention score matrix is ​​obtained by multiplying the query matrix after transforming the first modality feature vector matrix with the key matrix after transforming the second modality feature vector matrix.

[0026] Based on the attention score matrix, intermodal similarity feature aggregation calculation is performed to group vectors with similar features within a modality into one class.

[0027] In one embodiment, the inter-modal attention decision based on the aggregation calculation result of inter-modal similarity features and a preset attention score matrix threshold, to obtain the mask matrix of the attention score matrix, includes:

[0028] For each category aggregated by the second modality, int4 fixed-point computation is performed using the corresponding category center and the query vector of the first modality to obtain the fixed-point computation result;

[0029] Based on the preset attention score matrix threshold and the fixed-point calculation results, a mask matrix of the attention score matrix is ​​obtained. In the mask matrix, 1 represents the target point for which the attention score matrix needs to be calculated, and 0 represents the target point for which the calculation can be ignored.

[0030] Secondly, this application also provides a multimodal parallel optimization computing device based on feature dimensionality reduction and aggregation. The device includes:

[0031] The parameter acquisition module is used to acquire the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer.

[0032] The attention calculation module is used to perform dimensionality reduction and attention calculation on the parameter matrices of the first and second modality embedding layers to obtain the importance matrix of all attention heads.

[0033] The filtering module is used to select the first preset number of attention heads to be retained from all attention heads based on the importance matrix of each attention head, the preset attention head rejection rate, and the preset attention head performance threshold, so as to obtain the set of retained attention heads.

[0034] The processing module is used to perform inter-modal fusion, inter-modal similarity feature aggregation calculation, and inter-modal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix;

[0035] The floating-point calculation module is used to recover and reorganize the high-precision attention calculation based on the mask matrix, and extract the feature vectors in the corresponding second modal key matrix for floating-point calculation.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0037] Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer;

[0038] Dimensionality reduction and attention calculation are performed on the parameter matrices of the first and second modality embedding layers to obtain the importance matrices of all attention heads;

[0039] Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads;

[0040] Based on the retained set of attention heads, intermodal fusion, intermodal similarity feature aggregation calculation, and intermodal attention decision processing are performed to obtain the mask matrix of the attention score matrix;

[0041] Based on the mask matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modal key matrix are extracted for floating-point calculation.

[0042] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0043] Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer;

[0044] Dimensionality reduction and attention calculation are performed on the parameter matrices of the first and second modality embedding layers to obtain the importance matrices of all attention heads;

[0045] Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads;

[0046] Based on the retained set of attention heads, intermodal fusion, intermodal similarity feature aggregation calculation, and intermodal attention decision processing are performed to obtain the mask matrix of the attention score matrix;

[0047] Based on the mask matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modal key matrix are extracted for floating-point calculation.

[0048] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0049] Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer;

[0050] Dimensionality reduction and attention calculation are performed on the parameter matrices of the first and second modality embedding layers to obtain the importance matrices of all attention heads;

[0051] Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads;

[0052] Based on the retained set of attention heads, intermodal fusion, intermodal similarity feature aggregation calculation, and intermodal attention decision processing are performed to obtain the mask matrix of the attention score matrix;

[0053] Based on the mask matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modal key matrix are extracted for floating-point calculation.

[0054] The aforementioned multimodal parallel optimization computation method, apparatus, computer device, storage medium, and computer program product based on feature dimensionality reduction and aggregation obtains the parameter matrices of the first and second modality embedding layers; performs dimensionality reduction and attention computation on the first and second modality embedding layer parameter matrices to obtain the importance matrices of all attention heads; selects a preset number of attention heads to retain from all attention heads based on the importance matrix of each attention head, a preset attention head discard rate, and a preset attention head performance threshold, to obtain a set of retained attention heads; performs intermodal fusion, intermodal similar feature aggregation computation, and intermodal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix; reorganizes the high-precision attention computation based on the mask matrix, and extracts the feature vectors from the corresponding second modality key matrix for floating-point calculation. Throughout the process, the number of parameters and computation time required for intramodal encoding are greatly reduced by dimensionality reduction, attention computation is reduced by attention head filtering, and attention cross-computation is used to make full use of the interaction information between modalities to perform intermodal similar feature aggregation computation and intermodal attention decision-making. This can significantly reduce the number of processes required, thereby achieving efficient multimodal parallel optimization computation. Attached Figure Description

[0055] Figure 1 This is an application environment diagram of a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation in one embodiment;

[0056] Figure 2 This is a flowchart illustrating a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation in one embodiment.

[0057] Figure 3 This is a flowchart illustrating a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation in another embodiment.

[0058] Figure 4 This is a schematic diagram of a sub-process of S400 in one embodiment;

[0059] Figure 5 This is a flowchart illustrating a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation in a specific application embodiment.

[0060] Figure 6 This is a block diagram of a multimodal parallel optimization computing device based on feature dimensionality reduction and aggregation in one embodiment;

[0061] Figure 7This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 sends a multimodal parallel optimization computation request to server 104. Server 104 obtains the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix; it performs dimensionality reduction and attention computation on the first and second modality embedding layer parameter matrices to obtain the importance matrix of all attention heads; based on the importance matrix of each attention head, a preset attention head discard rate, and a preset attention head performance threshold, it selects a preset number of attention heads to retain from all attention heads to obtain a set of retained attention heads; based on the set of retained attention heads, it performs intermodal fusion, intermodal similarity feature aggregation computation, and intermodal attention decision processing to obtain a mask matrix of the attention score matrix; it reconstructs and reorganizes the high-precision attention computation based on the mask matrix, and extracts the feature vectors from the corresponding second modality key matrix for floating-point calculation. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0064] In one embodiment, such as Figure 2 As shown, a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation is provided, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0065] S100: Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer.

[0066] The first modality embedding layer parameter matrix specifically refers to the embedding layer parameter matrix of modality 1. The second modality embedding layer parameter matrix specifically refers to the embedding layer parameter matrix of modality 2. The original data of modality 1 and modality 2 belong to two different types of data; for example, the original data of modality 1 can be text data, and the original data of modality 2 can be image data. Specifically, the embedding layer parameter matrix can be obtained by preprocessing the original data of modality 1 and modality 2. In addition, the multimorphic model weight parameters can also be obtained here. The specific model includes two parts: intra-modality encoding and inter-modality fusion. The modality can be image, text, audio, etc. The multimorphic model weight parameters include: the modality 1 encoder weight parameter W. mod1 Modal 2 encoder weight parameters W mod2 The weight parameters W in the modal fusion part fusion The model output includes some weight parameters W. output Mode 1 original Embedding parameter E mod1 Mode 2 original Embedding parameter E mod2 .

[0067] S200: Perform dimensionality reduction and attention calculation on the parameter matrices of the first and second modal embedding layers to obtain the importance matrix of all attention heads.

[0068] As previously mentioned, the significant performance improvements of multimodal models have come at the cost of their high computational cost. While multimodal models perform well in various scenarios, their computational performance faces significant challenges during the inference and deployment phase. Multimodal models typically have billions or even tens of billions of parameters, placing enormous demands on computational resources and often failing to meet the low latency and high throughput requirements of online tasks. Here, we discuss the parameter matrix E of the first modality embedding layer. mod1 Second mode embedding layer parameter matrix E mod2 Dimensionality reduction is performed to reduce the amount of data processed.

[0069] In addition to dimensionality reduction, this process also performs attention calculations to determine the importance of each attention head, resulting in an importance matrix for all attention heads. Specifically, attention scores can be output for each word in each head after dimensionality reduction, and then the attention of each head can be calculated to obtain the importance matrix of the attention heads.

[0070] S300: Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads.

[0071] Attention drop-off rate p hAttention head performance threshold hp These are all pre-defined. Here, the performance of the attention heads is evaluated to determine whether they have reached a performance threshold, based on the importance matrix A′ of each head. h Attention drop-off rate p h and attention head performance threshold hp Select the top-k heads to be retained from the h heads to obtain the set of retained attention heads H′.

[0072] S400: Based on the retained set of attention heads, perform inter-modal fusion, inter-modal similarity feature aggregation calculation, and inter-modal attention decision processing to obtain the mask matrix of the attention score matrix.

[0073] The attention heads in the retained attention head set continue to participate in subsequent inter-modal similarity feature aggregation calculation and inter-modal attention decision-making, ultimately obtaining the mask matrix of the attention score matrix.

[0074] S500: Reorganize the high-precision attention calculation based on the mask matrix, and extract the eigenvectors from the corresponding second modal key matrix for floating-point calculation.

[0075] Based on the mask matrix of the attention score matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modality are extracted for high-precision floating-point calculation to complete the subsequent attention calculation and inference tasks.

[0076] The aforementioned multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation obtains the parameter matrices of the first and second modality embedding layers. Dimensionality reduction and attention calculations are performed on these matrices to obtain the importance matrices of all attention heads. Based on the importance matrix of each attention head, a preset attention head discard rate, and a preset attention head performance threshold, a preset number of attention heads are selected to be retained, resulting in a set of retained attention heads. Based on this set, intermodal fusion, intermodal similarity feature aggregation, and intermodal attention decision processing are performed to obtain a mask matrix for the attention score matrix. The high-precision attention calculations are then reorganized using the mask matrix, and the feature vectors from the corresponding second modality key matrix are extracted for floating-point calculations. Throughout this process, dimensionality reduction significantly reduces the number of parameters and computation time required for intramodal encoding, attention head selection reduces attention computation, and attention cross-computation fully utilizes the interaction information between modalities for intermodal similarity feature aggregation and intermodal attention decision processing, thus significantly reducing the required processing quantity and achieving efficient multimodal parallel optimization computation.

[0077] like Figure 3 As shown, in one embodiment, S200 includes:

[0078] S220: Based on the preset number of attention heads and the preset dimensionality reduction dimension, the parameter matrices of the first modality embedding layer and the parameter matrices of the second modality embedding layer are reduced in dimension to obtain the dimensionality-reduced matrix representation.

[0079] Number of attention heads h and dimensionality reduction d k These are all pre-set, but can be configured by the user based on the actual application. Specifically, during dimensionality reduction, the first modality embedding layer parameter matrix E can be... mod1 Second mode embedding layer parameter matrix E mod2 The input is fed into a transformer block to begin intramodal encoding. Each transformer block has h attention heads. After encoding, the query matrix, key matrix, and value matrix are obtained. Then, for each attention head h, d is selected. k Dimensionality reduction operations are performed on its queries, keys, and values.

[0080] In one embodiment, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are dimensionality reduced based on a preset number of attention heads and a preset dimensionality reduction dimension to obtain the dimensionality-reduced matrix representation. This includes: obtaining a preset number of attention heads h and a preset dimensionality reduction dimension; feeding the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix into N and M transformer blocks respectively to start intramodal encoding, with each transformer block having h attention heads; for each attention head, selecting a dimensionality reduction dimension based on the preset dimensionality reduction dimension, and performing a dimensionality reduction operation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the dimensionality-reduced matrix representation.

[0081] Specifically, E mod1 E mod2 Intramodal encoding is performed in N and M transformer blocks respectively, with each transformer block having h attention heads. The original query matrix is ​​as follows: The key matrix (Key) is And the value matrix (Value) is It is divided into h blocks by h attention heads, where the query matrix, key matrix, and value matrix of one head are respectively... and n is the length of the modal sequence (such as the word length of a sentence, the number of pixels in an image, etc.). Dimensionality reduction d k <d. For each attention head h, select d. kWe perform dimensionality reduction on its query, key, and value to obtain the dimensionality-reduced representation:

[0082] Q′ h =PCA(Q h d k )

[0083] K′ h =PCA(K h ,d k )

[0084] V′ h =PCA(V h ,d k ).

[0085] S240: Perform attention calculation on the matrix representation after dimensionality reduction for each attention head, and accumulate the attention score output for each word of each attention head after dimensionality reduction to obtain the importance matrix of all attention heads.

[0086] Here, we perform performance evaluation and discard decisions for intramodal attention heads. Specifically, for each attention head h, we perform dimensionality-reduced attention calculation and accumulate the attention score output for each word of each head after dimensionality reduction. The score for each head can be obtained by summing or averaging certain dimensions of the score matrix. For simplicity, we use absolute value summation to obtain the importance matrix A′ of the attention heads. h as follows:

[0087]

[0088] Furthermore, it can also be based on the importance matrix A′ h Implement an attention head discarding strategy. Specifically, this can be done based on the importance matrix A′ of each head. h Attention drop-off rate p h Attention head performance threshold hp Select the top-k heads to be retained from the h heads to obtain the retained attention head set H′. Use the retained attention head set H′ to perform attention calculation and subsequent inference tasks.

[0089] H = top-k(A, h*(1-p)) h threshold hp ).

[0090] like Figure 4 As shown, in one embodiment, S400 includes:

[0091] S420: Based on the retained attention head set, obtain the corresponding retained first modality feature vector and second modality feature vector.

[0092] Based on the retained attention head set H′, the corresponding retained first modality feature vector and second modality feature vector are obtained to continue the subsequent attention calculation and inference tasks.

[0093] S440: Perform matrix transformations on the eigenvectors of the first mode and the eigenvectors of the second mode, respectively.

[0094] When entering the intermodal fusion stage, the feature vector of modality 1 is... and mode 2 eigenvectors (d is the dimension of the feature vector) respectively passed through their respective W Q W k W v Matrix transformations are performed to obtain matrices Q1, K1, V1 and Q2, K2, V2, which are then used to calculate the attention score matrix.

[0095] S460: Employs attention-based cross-calculation to aggregate similar features between modalities.

[0096] When performing intermodal fusion in a multimodal model, attention cross-calculation is used to aggregate intermodal similarity features. Specifically, the query matrix transformed from the first modality feature vector matrix is ​​multiplied by the key matrix transformed from the second modality feature vector matrix to obtain the attention score matrix. Based on the attention score matrix, intermodal similarity feature aggregation is performed to group vectors with similar features within a modality into one category. Furthermore, the score matrix can be obtained by multiplying the query matrix of modality 1 and the key matrix of modality 2. Taking the query performed by modality 1 in modality 2 as an example, the reverse is also true. First, the query vector of modality 1 is extracted. Extract the modal 2-bond matrix Aggregate the high-dimensional feature representations within modality 2. If two vectors are similar in the original data space, they are classified into the same class. In other words, vectors with similar features within the modality are grouped into the same class.

[0097] Furthermore, the specific aggregation process for each high-dimensional feature representation within mode 2 includes the following sub-steps:

[0098] 1) Input the dataset to be aggregated: K = {k1, k2, ... k} m},in Specify the number of categories to be aggregated N, the number of subspaces L, each subspace corresponds to a hyperplane, and the number of hash functions K, each subspace corresponds to a hash function.

[0099] 2) Selecting a hyperplane. For each subspace l = 1 to L, randomly generate a hyperplane:

[0100] h l(x)=sign(w l *k+b l )

[0101] Where w l b is the normal vector of the hyperplane, randomly generated from a Gaussian distribution with a mean of 0; l The intercept of the hyperplane is randomly selected from the interval [0, R], where R is a sufficiently large value.

[0102] 3) Projection. For each data point k i For each subspace l, calculate the projection results:

[0103] p il =w l ·k i .

[0104] 4) Mapping to binary hash codes. For each subspace l, K hash functions are used to map the projection results to binary hash codes:

[0105]

[0106] in, It is the threshold of the hash function, which is randomly selected.

[0107] 5) Construct a hash table. Construct a hash table for each subspace l, storing the hash codes... As the key, the corresponding data point index i is stored in the hash table.

[0108] 6) Iterative Clustering. Initialize N cluster centers. In each iteration, for each data point k... i Perform the same projection and hash operations as in steps 3) and 4) to obtain the hash code. For each cluster center c k Calculate its hash code

[0109] Calculate data point k i With each cluster center c k Hamming weights between:

[0110]

[0111] in, A function that indicates whether two bits are equal.

[0112] 7) Assign data points. Assign each data point to the cluster center with the smallest Hamming weight. Update each cluster center to the average value of the data points belonging to that cluster based on the current clustering. Then recalculate the Hamming distance between each data point and the updated cluster center, and reassign the data point to the nearest cluster center.

[0113] 8) Repeat steps 6) to 7) until the cluster centers no longer change or the predetermined number of iterations is reached.

[0114] S480: Based on the aggregation calculation results of inter-modal similarity features and the preset attention score matrix threshold, inter-modal attention decision is performed to obtain the mask matrix of the attention score matrix.

[0115] Based on the intermodal similarity feature aggregation calculation results obtained from S460 and the preset attention score matrix threshold, score This yields a mask matrix `score′` for the attention score matrix `score`. In the mask matrix `score′`, 1 represents the target point for which the score needs to be calculated, and 0 represents the target point for which the calculation can be ignored.

[0116] Specifically, the above process includes performing int4 fixed-point calculation on each category aggregated by the second modality using the corresponding category center and the query vector of the first modality to obtain the fixed-point calculation result; and obtaining the mask matrix of the attention score matrix based on the preset attention score matrix threshold and the fixed-point calculation result.

[0117] This means that for each category aggregated from modality 2, low-precision int4 fixed-point calculations can be performed using the category centroids in the corresponding hash buckets and the query vector q from modality 1, based on the threshold. score This yields a mask matrix score′ for the score matrix, where 1 represents the target point for which the score needs to be calculated, and 0 represents the target point for which the calculation can be ignored.

[0118] In subsequent processing, based on the score′, the high-precision attention calculation can be reconstructed and reorganized. The feature vectors from the corresponding modality 2-key matrix K are extracted, and then high-precision floating-point calculations are performed to complete the subsequent attention calculations and inference tasks. Finally, performance evaluation and fine-tuning are conducted. Specifically, based on the above process, the hyperparameters involved in model feature dimensionality reduction or aggregation may need to be fine-tuned or recalculated to restore or maintain performance.

[0119] In specific application examples, such as Figure 5 As shown, the multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation in this application includes the following steps:

[0120] (1) Input the multimodal model weight parameters (the model includes two parts: intramodal encoding and intermodal fusion. The modality can be an image, text, audio, etc.) and the original embedding layer parameter matrix of modality 1 and modality 2 after data preprocessing.

[0121] (2) Set the number of attention heads h and the attention head rejection rate p. h Attention head performance threshold hp Attention score mask threshold hs Dimensionality reduction d k ;

[0122] (3) Dimensionality Reduction Preparation and Intra-Modal Dimensionality Reduction. The embedding layer parameter matrices of Modality 1 and Modality 2 are respectively fed into N and M transformer blocks for intra-modal encoding. Each transformer block has h attention heads. The original query matrix, key matrix, and value matrix are divided into h blocks by the h attention heads. For each attention head h, d is selected. k We perform dimensionality reduction on its query matrix, key matrix, and values ​​to obtain a dimensionality-reduced matrix representation.

[0123] (4) Perform attention calculation on the matrix representation of each attention head h after dimensionality reduction, and accumulate the attention score output of each word of each head after dimensionality reduction. The score of each head can be obtained by summing or averaging some dimensions of the score matrix, thereby obtaining the importance matrix of all attention heads.

[0124] (5) Based on the importance matrix of each head and the attention head rejection rate p h Attention head performance threshold hp Select the top-k heads to be retained from the h heads to obtain the retained attention head set H′. Use the retained attention head set H′ to perform attention calculation and subsequent inference tasks.

[0125] (6) Entering the intermodal fusion stage. Input modality 1 and modality 2 are processed by encoder 1 and encoder 2 respectively, resulting in modality 1 feature vector and modality 2 feature vector. Then, the modality 1 feature vector and modality 2 feature vector undergo matrix transformations before the attention score matrix is ​​calculated;

[0126] (7) Execute the intermodal similarity feature aggregation algorithm. When performing intermodal fusion in a multimodal model, attention cross-calculation is required. The query matrix of modality 1 and the key matrix of modality 2 are multiplied to obtain the score matrix. For example, modality 1 is the query modality and modality 2 is the queried modality. First, the query vector of modality 1 is extracted, and the high-dimensional feature representations in modality 2 are aggregated. If the two vectors are similar in the original data space, they are classified into the same class. That is, vectors with similar features in the modality are grouped into the same class, which has a large similarity.

[0127] (8) Intermodal attention decision. For each category aggregated by modality 2, low-precision int4 fixed-point calculation is performed using the corresponding category centroid and the query vector q of modality 1, based on the threshold. score The mask matrix score′ is obtained, where 1 represents the target point for which the score needs to be calculated, and 0 represents the target point for which the calculation can be ignored.

[0128] (9) Based on score′, reconstruct and reorganize the high-precision attention calculation, and extract the feature vectors from the corresponding modality 2-key matrix K. Perform high-precision floating-point calculations to complete the subsequent attention calculation and inference tasks.

[0129] (10) Performance evaluation and fine-tuning. Based on the above process, the hyperparameters involved in model feature dimensionality reduction or aggregation may need to be fine-tuned or recalculated in order to restore or maintain performance.

[0130] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0131] Based on the same inventive concept, this application also provides a feature-dimension reduction aggregation-based multimodal parallel optimization computing device for implementing the feature-dimension reduction aggregation-based multimodal parallel optimization computing method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more feature-dimension reduction aggregation-based multimodal parallel optimization computing device embodiments provided below can be found in the limitations of the feature-dimension reduction aggregation-based multimodal parallel optimization computing method described above, and will not be repeated here.

[0132] In one embodiment, such as Figure 6 As shown, a multimodal parallel optimization computing device based on feature dimensionality reduction and aggregation is provided, comprising:

[0133] The parameter acquisition module 100 is used to acquire the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer;

[0134] The attention calculation module 200 is used to perform dimensionality reduction and attention calculation on the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer to obtain the importance matrix of all attention heads;

[0135] The filtering module 300 is used to select the first preset number of attention heads to be retained from all attention heads based on the importance matrix of each attention head, the preset attention head rejection rate, and the preset attention head performance threshold, so as to obtain the set of retained attention heads.

[0136] The processing module 400 is used to perform inter-modal fusion, inter-modal similarity feature aggregation calculation, and inter-modal attention decision processing based on the retained attention head set, so as to obtain the mask matrix of the attention score matrix;

[0137] The floating-point calculation module 500 is used to reorganize the high-precision attention calculation based on the mask matrix and extract the feature vectors from the corresponding second modal key matrix for floating-point calculation.

[0138] In one embodiment, the attention calculation module 200 is further configured to reduce the dimensionality of the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, to obtain a dimensionality-reduced matrix representation; perform attention calculation on the dimensionality-reduced matrix representation of each attention head, and accumulate the attention score output of each word of each attention head after dimensionality reduction to obtain the importance matrix of all attention heads.

[0139] In one embodiment, the attention calculation module 200 is further configured to obtain a preset number of attention heads h and a preset dimensionality reduction dimension; to feed the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix into N and M transformer blocks respectively to start intramodal encoding, each transformer block having h attention heads; for each attention head, to select a dimensionality reduction dimension based on the preset dimensionality reduction dimension, and to perform dimensionality reduction operation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain a dimensionality-reduced matrix representation.

[0140] In one embodiment, the processing module 400 is further configured to obtain the corresponding retained first modality feature vector and second modality feature vector based on the retained attention head set; perform matrix transformation on the first modality feature vector and the second modality feature vector respectively; perform inter-modality similarity feature aggregation calculation using attention cross-calculation; and perform inter-modality attention decision based on the inter-modality similarity feature aggregation calculation result and a preset attention score matrix threshold to obtain the mask matrix of the attention score matrix.

[0141] In one embodiment, the processing module 400 is further configured to multiply the query matrix after the transformation of the first modality feature vector matrix with the key matrix after the transformation of the second modality feature vector matrix to obtain an attention score matrix; and perform inter-modal similarity feature aggregation calculation based on the attention score matrix to classify vectors with similar features within a modality into one class.

[0142] In one embodiment, the processing module 400 is further configured to perform int4 fixed-point calculation on each category aggregated by the second modality using the corresponding category center and the query vector of the first modality to obtain the fixed-point calculation result; and obtain a mask matrix of the attention score matrix according to the preset attention score matrix threshold and the fixed-point calculation result, wherein 1 in the mask matrix is ​​the target point for which the attention score matrix needs to be calculated, and 0 is the target point for which the calculation can be ignored.

[0143] Each module in the aforementioned multimodal parallel optimization computing device based on feature dimensionality reduction and aggregation can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0144] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores preset data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation.

[0145] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation.

[0147] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal parallel optimization computation method based on feature dimensionality reduction and aggregation, characterized in that, The method includes: Obtain the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer; the first modality is text type; the second modality is image type; Dimensionality reduction and attention calculation are performed on the parameter matrices of the first and second modality embedding layers to obtain the importance matrices of all attention heads; Based on the importance matrix of each attention head, the preset attention head discard rate, and the preset attention head performance threshold, select the first preset number of attention heads to be retained from all attention heads to obtain the set of retained attention heads; Based on the retained set of attention heads, intermodal fusion, intermodal similarity feature aggregation calculation, and intermodal attention decision processing are performed to obtain the mask matrix of the attention score matrix; Based on the mask matrix, the high-precision attention calculation is reorganized, and the feature vectors in the corresponding second modal key matrix are extracted for floating-point calculation.

2. The method according to claim 1, characterized in that, The dimensionality reduction and attention calculation performed on the parameter matrices of the first and second modality embedding layers yields the importance matrices of all attention heads, including: The parameter matrices of the first modality embedding layer and the second modality embedding layer are reduced in dimension based on a preset number of attention heads and a preset dimension reduction dimension, respectively, to obtain the dimension-reduced matrix representations. Attention is calculated on the matrix representation after dimensionality reduction for each attention head, and the attention score output of each word of each attention head after dimensionality reduction is accumulated to obtain the importance matrix of all attention heads.

3. The method according to claim 2, characterized in that, The first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are dimensionality reduced based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, to obtain the dimensionality-reduced matrix representation, including: Obtain the preset number of attention heads h and the preset dimensionality reduction dimension; The first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively fed into N and M transformer blocks to start intramodal encoding, and each transformer block has h attention heads; For each attention head, a dimensionality reduction dimension is selected based on the preset dimensionality reduction dimension, and a dimensionality reduction operation is performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the dimensionality-reduced matrix representation.

4. The method according to claim 1, characterized in that, The mask matrix for obtaining the attention score matrix based on the retained attention head set includes: inter-modal fusion, inter-modal similarity feature aggregation calculation, and inter-modal attention decision processing. Based on the retained set of attention heads, obtain the corresponding retained first modality feature vector and second modality feature vector; Perform matrix transformations on the first modality eigenvector and the second modality eigenvector respectively; An attention-cross computation method is used to aggregate intermodal similarity features. Based on the aggregation calculation results of inter-modal similarity features and the preset attention score matrix threshold, inter-modal attention decision is performed to obtain the mask matrix of the attention score matrix.

5. The method according to claim 4, characterized in that, The method of using attention cross-computation to aggregate intermodal similarity features includes: The attention score matrix is ​​obtained by multiplying the query matrix after transforming the first modality feature vector matrix with the key matrix after transforming the second modality feature vector matrix. Based on the attention score matrix, intermodal similarity feature aggregation calculation is performed to group vectors with similar features within a modality into one class.

6. The method according to claim 5, characterized in that, The inter-modal attention decision-making based on the aggregation calculation results of inter-modal similarity features and the preset attention score matrix threshold, resulting in a mask matrix for the attention score matrix, includes: For each category aggregated by the second modality, int4 fixed-point computation is performed using the corresponding category center and the query vector of the first modality to obtain the fixed-point computation result; Based on the preset attention score matrix threshold and the fixed-point calculation results, a mask matrix of the attention score matrix is ​​obtained. In the mask matrix, 1 represents the target point for which the attention score matrix needs to be calculated, and 0 represents the target point for which the calculation can be ignored.

7. A multimodal parallel optimization computing device based on feature dimensionality reduction and aggregation, characterized in that, The device includes: The parameter acquisition module is used to acquire the parameter matrix of the first modality embedding layer and the parameter matrix of the second modality embedding layer; the first modality is text type; the second modality is image type. The attention calculation module is used to perform dimensionality reduction and attention calculation on the parameter matrices of the first and second modality embedding layers to obtain the importance matrix of all attention heads. The filtering module is used to select the first preset number of attention heads to be retained from all attention heads based on the importance matrix of each attention head, the preset attention head rejection rate, and the preset attention head performance threshold, so as to obtain the set of retained attention heads. The processing module is used to perform inter-modal fusion, inter-modal similarity feature aggregation calculation, and inter-modal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix; The floating-point calculation module is used to recover and reorganize the high-precision attention calculation based on the mask matrix, and extract the feature vectors in the corresponding second modal key matrix for floating-point calculation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images