Multi-modal parallel optimization calculation method and device based on feature dimension reduction aggregation

Abstract

The invention relates to a multi-modal parallel optimization calculation method and device based on feature dimension reduction aggregation, and the method comprises the steps: obtaining a first modal embedding layer parameter matrix and a second modal embedding layer parameter matrix; carrying out dimensionality reduction and attention calculation on the first modal embedding layer parameter matrix and the second modal embedding layer parameter matrix to obtain an importance matrix of all attention heads; according to the importance matrix of each attention head, a preset attention head abandoning rate and a preset attention head performance threshold value, a preset number of heads needing to be reserved are selected from all the attention heads, and a reserved attention head set is obtained; based on the reserved attention head set, performing inter-modal fusion, inter-modal similar feature aggregation calculation and inter-modal attention decision processing to obtain a mask matrix of an attention score matrix; high-precision attention calculation is recovered and reorganized according to the mask matrix, and feature vectors in the corresponding second modal key matrix are taken out for floating-point number calculation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal parallel optimization calculation method, apparatus, computer equipment, storage medium and computer program product based on feature dimension reduction and aggregation. Background Art

[0002] A multimodal model is a machine learning or deep learning model that can process data from multiple different modalities (such as text, images, audio, etc.). In the real world, many problems involve multiple data sources or data types. For example, social media content may contain text, images, and videos, and medical diagnosis may involve medical images and clinical data. Multimodal models can process complex multi-source data and capture the inherent correlation between data in different modalities. They have been widely used in many fields (such as image and text retrieval, image and text question and answer, and emotion recognition, etc.).

[0003] Specifically, the commonly used multimodal models generally include two main components: intra-modal encoding and inter-modal fusion. Intra-modal encoding refers to encoding the information within each data modality and mapping the information within each modality to a common representation space for subsequent fusion and analysis; inter-modal fusion refers to combining information from different modalities to obtain a more comprehensive and richer feature representation, thereby improving the performance and generalization ability of the model for various downstream tasks with multimodal features. In the inter-modal fusion stage, the model considers the correlation and importance between different modalities to effectively fuse this information. Whether it is intra-modal encoding or inter-modal fusion, the attention mechanism plays an important role. In intra-modal encoding, the attention mechanism within the modality (such as text, image, etc.) helps to capture the relationship between different features, thereby better representing the information within the modality; in inter-modal fusion, it can help the model dynamically pay attention to the relevant information between different modalities, assign different attention weights according to the importance of each modality, and thus perform weighted fusion of information from different modalities. However, the significant improvement in the performance of multimodal models comes at the cost of its expensive computational cost. Although multimodal models have excellent performance in many scenarios, their computing performance faces huge challenges during the inference and deployment phase. The number of multimodal model parameters is generally in the billions or even tens of billions, which places huge demands on computing resources and often fails to meet the requirements of online tasks for low latency and high throughput.

[0004] There is very little research in traditional technology on optimizing and accelerating the calculation of multimodal models, and there is currently no efficient solution. Summary of the invention

[0005] Based on this, it is necessary to provide an efficient multimodal parallel optimization calculation method, device, computer equipment, storage medium and computer program product based on feature dimensionality reduction and aggregation to address the above technical problems.

[0006] In a first aspect, the present application provides a multi-modal parallel optimization calculation method based on feature dimension reduction aggregation. The method comprises:

[0007] Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0008] Performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0009] According to the importance matrix of each attention head, the preset attention head discard rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads;

[0010] Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix;

[0011] The high-precision attention calculation is restored and reorganized according to the mask matrix, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation.

[0012] In one embodiment, the performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the importance matrix of all attention heads includes:

[0013] Based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are reduced in dimension to obtain a matrix representation after dimensionality reduction;

[0014] Attention calculation is performed on the matrix representation of each attention head after dimensionality reduction, and the attention score output of each word unit of each attention head after dimensionality reduction is accumulated to obtain the importance matrix of all attention heads.

[0015] In one embodiment, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively reduced in dimension based on a preset number of attention heads and a preset dimensionality reduction dimension, and the matrix representation after dimensionality reduction is obtained includes:

[0016] Get the preset number of attention heads h and the preset dimensionality reduction dimension;

[0017] Sending the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to N and M transformer blocks respectively to start intra-modality encoding, each of the transformer blocks having h attention heads;

[0018] For each of the attention heads, a dimension reduction dimension is selected based on the preset dimension reduction dimension, and a dimension reduction operation is performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain a matrix representation after dimension reduction.

[0019] In one embodiment, the inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed based on the retained attention head set to obtain a mask matrix of the attention score matrix including:

[0020] Based on the reserved attention head set, obtaining a corresponding reserved first modality feature vector and a second modality feature vector;

[0021] Performing matrix transformation on the first modal eigenvector and the second modal eigenvector respectively;

[0022] Use attention cross calculation to aggregate similar features between modalities;

[0023] Based on the aggregation calculation results of similar features between modalities and the preset attention score matrix threshold, inter-modal attention decisions are made to obtain the mask matrix of the attention score matrix.

[0024] In one embodiment, the method of performing inter-modal similar feature aggregation calculation in the form of attention cross calculation includes:

[0025] Multiplying the query matrix transformed by the first modal feature vector matrix and the key matrix transformed by the second modal feature vector matrix to obtain an attention score matrix;

[0026] Based on the attention score matrix, similar feature aggregation calculation between modalities is performed to classify vectors with similar features within the modality into one category.

[0027] In one embodiment, the inter-modal attention decision is performed based on the inter-modal similar feature aggregation calculation result and a preset attention score matrix threshold, and the mask matrix of the attention score matrix is ​​obtained, including:

[0028] For each category aggregated from the second modality, use the corresponding category center and the query vector of the first modality to perform int4 fixed-point calculation to obtain the fixed-point calculation result;

[0029] According to the preset attention score matrix threshold and the fixed-point calculation result, a mask matrix of the attention score matrix is ​​obtained, in which 1 is the target point for which the attention score matrix needs to be calculated, and 0 is the target point for which the calculation can be ignored.

[0030] In a second aspect, the present application also provides a multi-modal parallel optimization computing device based on feature dimension reduction aggregation. The device comprises:

[0031] A parameter acquisition module, used to acquire a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0032] An attention calculation module, used to perform dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0033] A screening module is used to select a preset number of heads to be retained from all attention heads according to the importance matrix of each attention head, a preset attention head rejection rate, and a preset attention head performance threshold, to obtain a set of retained attention heads;

[0034] A processing module, configured to perform inter-modal fusion, inter-modal similar feature aggregation calculation, and inter-modal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix;

[0035] The floating point calculation module is used to restore and reorganize the high-precision attention calculation according to the mask matrix, and take out the eigenvector in the corresponding second modal key matrix for floating point calculation.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0038] Performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0039] According to the importance matrix of each attention head, the preset attention head discard rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads;

[0040] Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix;

[0041] The high-precision attention calculation is restored and reorganized according to the mask matrix, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0043] Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0044] Performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0045] According to the importance matrix of each attention head, the preset attention head discard rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads;

[0046] Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix;

[0047] The high-precision attention calculation is restored and reorganized according to the mask matrix, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation.

[0048] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0049] Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0050] Performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0051] According to the importance matrix of each attention head, the preset attention head discard rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads;

[0052] Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix;

[0053] The high-precision attention calculation is restored and reorganized according to the mask matrix, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation.

[0054] The above-mentioned multimodal parallel optimization calculation method, device, computer equipment, storage medium and computer program product based on feature dimensionality reduction and aggregation obtain the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix; perform dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the importance matrix of all attention heads; according to the importance matrix of each attention head, the preset attention head rejection rate and the preset attention head performance threshold, select the first preset number of heads to be retained from all attention heads to obtain a set of retained attention heads; based on the retained set of attention heads, perform inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing to obtain a mask matrix of the attention score matrix; restore and reorganize the high-precision attention calculation according to the mask matrix, and take out the eigenvectors in the corresponding second modality key matrix for floating-point calculation. During the whole process, dimension reduction is used to greatly reduce the number of parameters and computing time required for intra-modal encoding, and attention head screening is used to reduce attention calculation. Then, through attention cross-calculation, the interactive information between modalities is fully utilized to perform inter-modal similar feature aggregation calculation and inter-modal attention decision-making, which can significantly reduce the number of required processing and achieve efficient multi-modal parallel optimization calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is an application environment diagram of a multi-modal parallel optimization computing method based on feature dimension reduction and aggregation in one embodiment;

[0056] Figure 2 It is a flowchart of a multi-modal parallel optimization calculation method based on feature dimension reduction and aggregation in one embodiment;

[0057] Figure 3 It is a flowchart of a multi-modal parallel optimization calculation method based on feature dimension reduction and aggregation in another embodiment;

[0058] Figure 4 is a schematic diagram of a sub-process of S400 in one embodiment;

[0059] Figure 5 It is a flowchart of a multi-modal parallel optimization calculation method based on feature dimension reduction and aggregation in a specific application embodiment;

[0060] Figure 6 It is a structural block diagram of a multi-modal parallel optimization computing device based on feature dimension reduction and aggregation in one embodiment;

[0061] Figure 7FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The multi-modal parallel optimization calculation method based on feature dimension reduction aggregation provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends a multimodal parallel optimization calculation request to the server 104, and the server 104 obtains the first modal embedding layer parameter matrix and the second modal embedding layer parameter matrix; the first modal embedding layer parameter matrix and the second modal embedding layer parameter matrix are reduced in dimension and attention is calculated to obtain the importance matrix of all attention heads; according to the importance matrix of each attention head, the preset attention head rejection rate and the preset attention head performance threshold, the preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads; based on the reserved attention head set, inter-modal fusion, inter-modal similar feature aggregation calculation and inter-modal attention decision processing are performed to obtain the mask matrix of the attention score matrix; according to the mask matrix, the high-precision attention calculation is restored and reorganized, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0064] In one embodiment, Figure 2 As shown in the figure, a multi-modal parallel optimization calculation method based on feature dimension reduction aggregation is provided. Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0065] S100: Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix.

[0066] The first modality embedding layer parameter matrix specifically refers to the embedding layer parameter matrix of modality 1. The second modality embedding layer parameter matrix specifically refers to the embedding layer parameter matrix of modality 2. Modality 1 original data and modality 2 original data belong to two different types of data. For example, modality 1 original data can be text type data, and modality 2 original data can be image type data. Specifically, the original data of modality 1 and modality 2 can be subjected to data preprocessing to obtain the embedding layer Embedding parameter matrix. In addition, the weight parameters of the multi-state model can also be obtained here. The specific model includes two parts: intra-modality encoding and inter-modality fusion. The modality can be pictures, text, audio, etc. The multi-state model weight parameters include: modality 1 encoder weight parameter W mod1 , Mode 2 encoder weight parameter W mod2 , weight parameter W of the modality fusion part fusion , the model output part weight parameter W output , Mode 1 original Embedding parameter E mod1 , Mode 2 original Embedding parameter E mod2 .

[0067] S200: Perform dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the importance matrix of all attention heads.

[0068] As mentioned before, the significant improvement in the performance of multimodal models comes at the cost of expensive computational costs. Although multimodal models have excellent performance in many scenarios, their computational performance is greatly challenged during the inference and deployment phase. The number of multimodal model parameters is generally in the billions or even tens of billions, and the requirements for computing resources are extremely huge, often unable to meet the requirements of online tasks for low latency and high throughput. Here, the first modal embedding layer parameter matrix E mod1 and the second modality embedding layer parameter matrix E mod2 Perform dimensionality reduction to reduce the amount of data processing.

[0069] In addition to the dimensionality reduction calculation, attention calculation is also performed here to determine the importance of the attention head and obtain the importance matrix of all attention heads. Specifically, the attention score can be output for each word unit of each head after dimensionality reduction, and then the attention of each head can be calculated to obtain the importance matrix of the attention head.

[0070] S300: According to the importance matrix of each attention head, the preset attention head rejection rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads.

[0071] Attention head abandonment rate p h, attention head performance threshold hp They are all pre-set. Here, the performance of the attention head is evaluated to determine whether the attention head reaches the performance threshold. According to the importance matrix A′ of each head h , attention head abandonment rate p h And the attention head performance threshold threshld hp , select the top-k heads to be retained from the h heads and obtain the retained attention head set H′.

[0072] S400: Based on the retained set of attention heads, inter-modal fusion, inter-modal similar feature aggregation calculation, and inter-modal attention decision processing are performed to obtain a mask matrix of the attention score matrix.

[0073] The attention heads in the retained attention head set continue to participate in the subsequent inter-modal similar feature aggregation calculation and inter-modal attention decision, and finally obtain the mask matrix of the attention score matrix.

[0074] S500: Reorganize the high-precision attention calculation according to the mask matrix recovery, and take out the eigenvector in the corresponding second modal key matrix for floating-point calculation.

[0075] According to the mask matrix of the attention score matrix, the high-precision attention calculation is restored and reorganized, and the corresponding feature vector in the second mode is taken out for high-precision calculation of float32 floating-point numbers to complete the subsequent attention calculation and reasoning tasks.

[0076] The multimodal parallel optimization calculation method based on feature dimension reduction and aggregation obtains the first modal embedding layer parameter matrix and the second modal embedding layer parameter matrix; performs dimension reduction and attention calculation on the first modal embedding layer parameter matrix and the second modal embedding layer parameter matrix to obtain the importance matrix of all attention heads; according to the importance matrix of each attention head, the preset attention head abandonment rate and the preset attention head performance threshold, selects the preset number of heads to be retained from all attention heads to obtain the retained attention head set; based on the retained attention head set, performs inter-modal fusion, inter-modal similar feature aggregation calculation and inter-modal attention decision processing to obtain the mask matrix of the attention score matrix; according to the mask matrix, the high-precision attention calculation is restored and reorganized, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point number calculation. In the whole process, the number of parameters and calculation time required for intra-modal encoding are greatly reduced by dimension reduction, and the attention calculation is reduced by attention head screening, and then the interactive information between the modalities is fully utilized to perform inter-modal similar feature aggregation calculation and inter-modal attention decision through attention cross calculation, which can significantly reduce the number of required processing, thereby realizing efficient multimodal parallel optimization calculation.

[0077] like Figure 3 As shown, in one embodiment, S200 includes:

[0078] S220: Based on a preset number of attention heads and a preset dimensionality reduction dimension, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively reduced in dimension to obtain a matrix representation after dimensionality reduction.

[0079] The number of attention heads h and the dimension reduction d k are all preset, which can be set by the user based on the actual application situation. When performing dimensionality reduction processing, the first modal embedding layer parameter matrix E mod1 and the second modality embedding layer parameter matrix E mod2 The model is fed into the transformer block to start intra-modal encoding. Each transformer block has h attention heads. After encoding, we get the query matrix, key matrix, and value matrix. Then, for each attention head h, we select d k , performing dimensionality reduction operations on its queries, keys, and values.

[0080] In one embodiment, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively reduced in dimension based on a preset number of attention heads and a preset dimensionality reduction dimension to obtain a matrix representation after dimensionality reduction, including: obtaining a preset number of attention heads h and a preset dimensionality reduction dimension; sending the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to N and M transformer blocks respectively to start intra-modal encoding, each transformer block having h attention heads; for each attention head, selecting a dimensionality reduction dimension based on the preset dimensionality reduction dimension, performing dimensionality reduction operations on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix, and obtaining a matrix representation after dimensionality reduction.

[0081] Specifically, E mod1 、E mod2 Enter N and M transformer blocks respectively to start intra-modal encoding, and each transformer block has h attention heads. The original query matrix (Query) is The key matrix (Key) is And the value matrix (Value) is It is divided into h blocks by h attention heads, where the query matrix, key matrix and value matrix of one head are and n is the length of the modal sequence (such as the word length of a sentence, the number of pixels in an image, etc.), Dimensionality reduction d k <d. For each attention head h, select d k, perform dimensionality reduction operations on its query, key, and value to obtain the reduced dimensionality representation:

[0082] Q′ h = PCA(Q h , d k )

[0083] K′ h = PCA(K h ,d k )

[0084] V′ h = PCA(V h ,d k ).

[0085] S240: Perform attention calculation on the matrix representation of each attention head after dimensionality reduction, and accumulate the attention score output of each word unit of each attention head after dimensionality reduction to obtain the importance matrix of all attention heads.

[0086] Here, the performance evaluation and discarding decision of the attention head in the modality are performed. Specifically, for each attention head h, the attention calculation after dimensionality reduction is performed, and the attention score output of each word unit of each head after dimensionality reduction is accumulated. The score of each head can be obtained by summing or averaging certain dimensions of the score matrix. For simplicity, the absolute value sum is used to obtain the importance matrix A′ of the attention head h as follows:

[0087]

[0088] Furthermore, we can also use the importance matrix A′ h Execute the attention head discarding strategy. Specifically, we can use the importance matrix A′ of each head h , attention head abandonment rate p h And the attention head performance threshold hp , select the top-k heads that need to be retained from the h heads, and obtain the retained attention head set H′. Use the retained attention head set H′ for attention calculation and subsequent reasoning tasks.

[0089] H = top-k(A, h*(1-p h ),threshold hp ).

[0090] like Figure 4 As shown, in one embodiment, S400 includes:

[0091] S420: Based on the retained attention head set, obtain the corresponding retained first modal feature vector and second modal feature vector.

[0092] Based on the retained attention head set H′, the corresponding retained first modal feature vector and second modal feature vector are obtained to continue the subsequent attention calculation and reasoning tasks.

[0093] S440: Perform matrix transformation on the first modal eigenvector and the second modal eigenvector respectively.

[0094] When entering the inter-modal fusion stage, the modal 1 feature vector and the mode 2 eigenvector (d is the dimension of the feature vector) through their respective W Q , W k , W v Matrix transformation, get Q 1 , K 1 , V 1 and Q 2 , K 2 , V 2 The matrix is ​​used to calculate the attention score matrix Score later.

[0095] S460: Use attention cross calculation to perform inter-modal similar feature aggregation calculation.

[0096] When the multimodal model performs inter-modal fusion, the cross-attention calculation form is used to perform inter-modal similar feature aggregation calculation. Specifically, the query matrix after the transformation of the first modal feature vector matrix and the key matrix after the transformation of the second modal feature vector matrix can be multiplied to obtain an attention score matrix; based on the attention score matrix, inter-modal similar feature aggregation calculation is performed to classify vectors with similar features within the modality into one category. Furthermore, the query matrix of modality 1 and the key matrix of modality 2 can be multiplied to obtain a score matrix. Take modality 1 performing a query in modality 2 as an example, and vice versa. First, take out the query vector of modality 1. Take out the modal 2 key matrix The high-dimensional feature representations in mode 2 are aggregated. If two vectors are similar in the original data space, they are classified as the same category, that is, vectors with similar features in the mode are classified as the same category.

[0097] In addition, the aggregation process for each high-dimensional feature representation in modality 2 specifically includes the following sub-steps:

[0098] 1) Input the data set to be aggregated K = {k 1 , k 2 ,...k m},in Specify the number of categories to be aggregated N, the number of subspaces L, each subspace corresponds to a hyperplane, and the number of hash functions K, each subspace corresponds to a hash function.

[0099] 2) Select a hyperplane. For each subspace l = 1 to L, randomly generate a hyperplane:

[0100] h l (x) = sign(w l *k+b l )

[0101] where w l is the normal vector of the hyperplane, randomly generated from a Gaussian distribution with mean 0; b l The intercept of the hyperplane is randomly selected from the interval [0, R], where R is a sufficiently large value.

[0102] 3) Projection. For each data point k i And for each subspace l, calculate the projection result:

[0103] p il =w l ·k i .

[0104] 4) Mapping to binary hash code. For each subspace l, use K hash functions to map the projection result to a binary hash code:

[0105]

[0106] in, is the threshold of the hash function, chosen randomly.

[0107] 5) Build a hash table. Build a hash table for each subspace l and store the hash code As the key, the corresponding data point index i is stored in the hash table.

[0108] 6) Iterative clustering. Initialize N cluster centers. In each iteration, for each data point k i , perform the same projection and hashing operations as steps 3) and 4) to obtain the hash code For each cluster center c k , calculate its hash code

[0109] Calculate data point k i With each cluster center c k The Hamming weight between:

[0110]

[0111] in, A function that indicates whether two bits are equal.

[0112] 7) Assign data points. Assign each data point to the cluster center with the smallest Hamming weight. According to the current cluster division, update each cluster center to the average value of the data points belonging to the cluster. Then recalculate the Hamming distance between each data point and the updated cluster center, and reallocate the data point to the nearest cluster center.

[0113] 8) Repeat steps 6) to 7) until the cluster center no longer changes or the predetermined number of iterations is reached.

[0114] S480: Based on the inter-modal similar feature aggregation calculation results and the preset attention score matrix threshold, an inter-modal attention decision is made to obtain a mask matrix of the attention score matrix.

[0115] Based on the inter-modal similar feature aggregation calculation results obtained in S460 and the preset attention score matrix threshold score , we get the mask matrix score′ of the attention score matrix score. In the mask matrix score′, 1 is the target point for which the score needs to be calculated, and 0 is the target point for which the calculation can be ignored.

[0116] Specifically, the above processing process includes performing int4 fixed-point calculation on each category aggregated by the second modality using the corresponding category center and the query vector of the first modality to obtain a fixed-point calculation result; and obtaining a mask matrix of the attention score matrix based on the preset attention score matrix threshold and the fixed-point calculation result.

[0117] That is, for each category aggregated from modality 2, we can use the category center point in the corresponding hash bucket and the query vector q of modality 1 to perform low-precision int4 fixed-point calculations, according to the threshold score , we get the mask matrix score′ of the score matrix, where 1 is the target point for which the score needs to be calculated, and 0 is the target point for which the calculation can be ignored.

[0118] In subsequent processing, we can restore and reorganize the high-precision attention calculation according to score′, extract the eigenvectors in the corresponding modal 2-key matrix K, and then perform high-precision calculations of float32 floating-point numbers to complete subsequent attention calculations and reasoning tasks. Finally, we perform performance evaluation and fine-tuning. The evaluation and fine-tuning here specifically refer to the fact that according to the above process, the hyperparameters involved in the dimensionality reduction or aggregation of model features may need to be fine-tuned or recalculated to restore or maintain performance.

[0119] In specific application examples, such as Figure 5 As shown, the multimodal parallel optimization calculation method based on feature dimension reduction aggregation in this application includes the following steps:

[0120] (1) Input the weight parameters of the multimodal model (the model consists of two parts: intra-modal encoding and inter-modal fusion, and the modalities can be images, text, audio, etc.) and the embedding layer parameter matrix of modality 1 and modality 2 after data preprocessing.

[0121] (2) Set the number of attention heads h and the rate of attention head discard p h , attention head performance threshold hp , attention score mask threshold thresshold hs , dimension reduction d k ;

[0122] (3) Dimensionality reduction preparation and intra-modality dimensionality reduction. The embedding layer parameter matrices of modality 1 and modality 2 enter N and M transformer blocks respectively to start intra-modality encoding. Each transformer block has h attention heads. The original query matrix, key matrix, and value matrix are divided into h blocks by h attention heads. For each attention head h, select d k , perform dimensionality reduction operations on its query matrix, key matrix, and value to obtain the matrix representation after dimensionality reduction.

[0123] (4) Perform attention calculation on the matrix representation of each attention head h after dimensionality reduction, and accumulate the attention score output of each word unit of each head after dimensionality reduction. The score of each head can be obtained by summing or averaging certain dimensions of the score matrix, thereby obtaining the importance matrix of all attention heads.

[0124] (5) According to the importance matrix of each head, the attention head discard rate p h And the attention head performance threshold hp , select the top-k heads that need to be retained from the h heads, and obtain the retained attention head set H′. Use the retained attention head set H′ for attention calculation and subsequent reasoning tasks.

[0125] (6) Enter the inter-modal fusion stage. Input modality 1 and modality 2 are processed by encoder 1 and encoder 2 respectively, and then modality 1 feature vector and modality 2 feature vector are transformed by matrix to calculate the attention score matrix;

[0126] (7) Execute the inter-modal similar feature aggregation algorithm. When the multimodal model performs inter-modal fusion, it is necessary to adopt the attention cross calculation form, and multiply the query matrix of modality 1 and the key matrix of modality 2 to obtain a score matrix. For example, modality 1 is the query modality and modality 2 is the queried modality. First, take out the query vector of modality 1, and aggregate the high-dimensional feature representations in modality 2. If the two vectors are similar in the original data space and are of the same category, that is, the vectors with similar features in the modality are classified into the same category, which has greater similarity.

[0127] (8) Inter-modal attention decision. For each category aggregated from modality 2, a low-precision int4 fixed-point calculation is performed using the corresponding category center point and the query vector q of modality 1. score , get the mask matrix score′ of the score matrix, 1 is the target point that needs to calculate the score, and 0 is the target point that can be ignored;

[0128] (9) According to score′, restore and reorganize the high-precision attention calculation and extract the eigenvectors in the corresponding modal 2-key matrix K. Perform high-precision calculations of float32 floating-point numbers to complete subsequent attention calculations and reasoning tasks.

[0129] (10) Performance evaluation and fine-tuning. Based on the above process, the hyperparameters involved in the dimensionality reduction or aggregation of model features may need to be fine-tuned or recalculated to restore or maintain performance.

[0130] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a multimodal parallel optimization computing device based on feature dimensionality reduction aggregation for implementing the multimodal parallel optimization computing method based on feature dimensionality reduction aggregation involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more multimodal parallel optimization computing devices based on feature dimensionality reduction aggregation provided below can be referred to the limitations of the multimodal parallel optimization computing method based on feature dimensionality reduction aggregation above, and will not be repeated here.

[0132] In one embodiment, Figure 6 As shown, a multi-modal parallel optimization computing device based on feature dimension reduction aggregation is provided, comprising:

[0133] A parameter acquisition module 100, used to acquire a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix;

[0134] An attention calculation module 200 is used to perform dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads;

[0135] A screening module 300 is used to select a preset number of heads to be retained from all attention heads according to the importance matrix of each attention head, a preset attention head rejection rate and a preset attention head performance threshold, so as to obtain a set of retained attention heads;

[0136] A processing module 400 is used to perform inter-modal fusion, inter-modal similar feature aggregation calculation and inter-modal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix;

[0137] The floating point calculation module 500 is used to restore and reorganize the high-precision attention calculation according to the mask matrix, and extract the eigenvector in the corresponding second modal key matrix for floating point calculation.

[0138] In one embodiment, the attention calculation module 200 is also used to reduce the dimensions of the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, to obtain a matrix representation after dimensionality reduction; perform attention calculation on the matrix representation after dimensionality reduction of each attention head, and accumulate the attention score outputs of each word unit of each attention head after dimensionality reduction to obtain an importance matrix of all attention heads.

[0139] In one embodiment, the attention calculation module 200 is also used to obtain a preset number of attention heads h and a preset dimensionality reduction dimension; the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are respectively sent to N and M transformer blocks to start intra-modal encoding, and each transformer block has h attention heads; for each attention head, a dimensionality reduction dimension is selected based on the preset dimensionality reduction dimension, and a dimensionality reduction operation is performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain a matrix representation after dimensionality reduction.

[0140] In one embodiment, the processing module 400 is also used to obtain the corresponding retained first modal feature vector and second modal feature vector based on the retained attention head set; perform matrix transformation on the first modal feature vector and the second modal feature vector respectively; perform inter-modal similar feature aggregation calculation using the attention cross calculation form; perform inter-modal attention decisions based on the inter-modal similar feature aggregation calculation results and a preset attention score matrix threshold to obtain a mask matrix of the attention score matrix.

[0141] In one embodiment, the processing module 400 is also used to multiply the query matrix after the first modal feature vector matrix is ​​transformed by the key matrix after the second modal feature vector matrix is ​​transformed to obtain an attention score matrix; based on the attention score matrix, similar feature aggregation calculations are performed between modalities to classify vectors with similar features within the modality into one category.

[0142] In one embodiment, the processing module 400 is also used to perform int4 fixed-point calculation on each category aggregated by the second modality using the corresponding category center and the query vector of the first modality to obtain a fixed-point calculation result; based on a preset attention score matrix threshold and the fixed-point calculation result, a mask matrix of the attention score matrix is ​​obtained, in which 1 is the target point for which the attention score matrix needs to be calculated, and 0 is the target point for which the calculation can be ignored.

[0143] Each module in the multimodal parallel optimization computing device based on feature dimension reduction and aggregation can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0144] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-modal parallel optimization calculation method based on feature dimensionality reduction aggregation is implemented.

[0145] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0146] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the multimodal parallel optimization calculation method based on feature dimensionality reduction and aggregation is implemented.

[0147] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the multimodal parallel optimization calculation method based on feature dimensionality reduction and aggregation is implemented.

[0148] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the multimodal parallel optimization computing method based on feature dimensionality reduction and aggregation.

[0149] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-modal parallel optimization calculation method based on feature dimension reduction aggregation, characterized in that: The method comprises: Obtain a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix; Performing dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads; According to the importance matrix of each attention head, the preset attention head discard rate and the preset attention head performance threshold, a preset number of heads to be retained are selected from all attention heads to obtain a set of retained attention heads; Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix; The high-precision attention calculation is restored and reorganized according to the mask matrix, and the eigenvectors in the corresponding second modal key matrix are taken out for floating-point calculation.

2. The method according to claim 1, characterized in that The dimension reduction and attention calculation are performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain the importance matrix of all attention heads, including: Based on a preset number of attention heads and a preset dimensionality reduction dimension, respectively, the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix are reduced in dimension to obtain a matrix representation after dimensionality reduction; Attention calculation is performed on the matrix representation of each attention head after dimensionality reduction, and the attention score output of each word unit of each attention head after dimensionality reduction is accumulated to obtain the importance matrix of all attention heads.

3. The method according to claim 2, characterized in that The step of reducing the dimensions of the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix based on the preset number of attention heads and the preset dimension reduction dimension to obtain the matrix representation after dimension reduction includes: Get the preset number of attention heads h and the preset dimensionality reduction dimension; Sending the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to N and M transformer blocks respectively to start intra-modality encoding, each of the transformer blocks having h attention heads; For each of the attention heads, a dimension reduction dimension is selected based on the preset dimension reduction dimension, and a dimension reduction operation is performed on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain a matrix representation after dimension reduction.

4. The method according to claim 1, characterized in that Based on the reserved attention head set, inter-modality fusion, inter-modality similar feature aggregation calculation and inter-modality attention decision processing are performed to obtain a mask matrix of the attention score matrix including: Based on the reserved attention head set, obtaining a corresponding reserved first modality feature vector and a second modality feature vector; Performing matrix transformation on the first modal eigenvector and the second modal eigenvector respectively; Use attention cross calculation to aggregate similar features between modalities; Based on the aggregation calculation results of similar features between modalities and the preset attention score matrix threshold, inter-modal attention decisions are made to obtain the mask matrix of the attention score matrix.

5. The method according to claim 4, characterized in that The method of performing inter-modal similar feature aggregation calculation in the form of attention cross calculation includes: Multiplying the query matrix transformed by the first modal feature vector matrix and the key matrix transformed by the second modal feature vector matrix to obtain an attention score matrix; Based on the attention score matrix, similar feature aggregation calculation between modalities is performed to classify vectors with similar features within the modality into one category.

6. The method according to claim 5, characterized in that The inter-modal attention decision is performed based on the inter-modal similar feature aggregation calculation result and the preset attention score matrix threshold, and the mask matrix of the attention score matrix is ​​obtained, which includes: For each category aggregated from the second modality, use the corresponding category center and the query vector of the first modality to perform int4 fixed-point calculation to obtain the fixed-point calculation result; According to the preset attention score matrix threshold and the fixed-point calculation result, a mask matrix of the attention score matrix is ​​obtained, in which 1 is the target point for which the attention score matrix needs to be calculated, and 0 is the target point for which the calculation can be ignored.

7. A multi-modal parallel optimization computing device based on feature dimension reduction aggregation, characterized in that: The device comprises: A parameter acquisition module, used to acquire a first modality embedding layer parameter matrix and a second modality embedding layer parameter matrix; An attention calculation module, used to perform dimensionality reduction and attention calculation on the first modality embedding layer parameter matrix and the second modality embedding layer parameter matrix to obtain an importance matrix of all attention heads; A screening module is used to select a preset number of heads to be retained from all attention heads according to the importance matrix of each attention head, a preset attention head rejection rate, and a preset attention head performance threshold, to obtain a set of retained attention heads; A processing module, configured to perform inter-modal fusion, inter-modal similar feature aggregation calculation, and inter-modal attention decision processing based on the retained attention head set to obtain a mask matrix of the attention score matrix; The floating point calculation module is used to restore and reorganize the high-precision attention calculation according to the mask matrix, and take out the eigenvector in the corresponding second modal key matrix for floating point calculation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images