Image-text pair model training video memory optimization method
By combining and optimizing the tokens output from each layer of the model, the problem of large video memory usage during model training is solved, and the memory requirement is reduced and the model training efficiency is improved.
Patent Information
- Application Number
- CN202411669400.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-03-14
AI Technical Summary
In the prior art, the graphics and text model occupies a large amount of video memory during training, which limits its application in resource-constrained environments.
By merging the tokens output from each layer during the training process, the specific steps include calculating the similarity of the token, selecting the token pair with the highest similarity to merge, and inputting the merged token to the next layer.
With almost no loss of model performance, the video memory requirement during training is reduced, and the model training efficiency and video memory utilization are improved.
Smart Images
Figure CN119623554A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a method for optimizing video memory for image-text pair model training. Background Art
[0002] With the widespread application of deep learning in the field of image and text processing, Transformer-based image-text pair models (such as CLIP) have established a strong connection between images and text. However, these models are usually trained with image-text pair data and contrast loss functions, so a very large batch size is required during training to ensure the model effect. This requires a very large amount of video memory resources, limiting its application in resource-constrained environments. Summary of the invention
[0003] The present invention mainly solves the technical problems existing in the prior art such as large video memory occupancy and limited application during image-text pair model training, and provides a video memory optimization method for image-text pair model training which can effectively reduce video memory occupancy and improve training efficiency.
[0004] The present invention solves the above technical problems mainly through the following technical solutions: a method for optimizing video memory for image-text pair model training, merging the tokens (image tag embedding) output by each layer during the training process, and inputting the merged tokens into the next layer. The specific merging method includes the following steps: S1. Calculate the similarity of each token output by the previous layer; S2, select the n token pairs with the highest similarity; S3. Merge the selected token pairs.
[0005] Preferably, the step S1 is specifically as follows: S101, divide t tokens into two sets A and B evenly by allocating them in turns; t is the number of tokens output by the previous layer; S102: For each token in set A, calculate its similarity with each token in set B to obtain a similarity matrix.
[0006] Preferably, the similarity is calculated by the following formula: cosine_similarity = dot_product(A[i], B[j]) / (norm(A[i]) * norm(B[j])), Among them, cosine_similarity is the similarity, A[i] is the i-th token in set A, B[j] is the j-th token in set B, dot_product(A[i],B[j]) represents the dot product of vectors A[i] and B[j], norm(A[i]) and norm(B[j]) represent the norms of vectors A[i] and B[j] respectively.
[0007] Preferably, the step S2 is specifically as follows: Select the n most similar ones from the similarity matrix s values, record their index positions, and then select corresponding tokens from set A and set B according to these index positions to form token pairs; n s Determined by the following formula: n s =n s-1 ×(L s-1 / L s ), Where n s is the number of elements in the similarity matrix selected at the sth iteration, n s-1 is the number of elements in the similarity matrix selected at the s-1th iteration, L s is the loss value at the sth iteration, L s-1 is the loss value at the s-1th iteration, n1 at the first iteration is 5% of t, and n s The value range of is 5% of t to 30% of t. During training, each layer is counted as one iteration.
[0008] Preferably, the loss value L s It is obtained by calculating the loss function used in the training of the image-text pair model. The image-text pair model usually uses contrast loss when training, specifically the InfoNCE loss function.
[0009] Preferably, the step S3 is specifically as follows: For each selected token pair, the two tokens are merged in an average manner to obtain a new token; The fused new token is used to replace the token in the fused set A, and the fused token in the set B is deleted. Then the tokens in the replaced set A and the deleted set B are concatenated at their original positions to obtain a new input vector, which is the input of the image and text to the next layer of the model.
[0010] Preferably, the fusion of two tokens specifically refers to adding the values of each corresponding dimension of a pair of tokens and then taking the average value.
[0011] The substantial effect brought about by the present invention is that the video memory requirement during training can be reduced with almost no loss in model performance, thereby improving model training efficiency and video memory utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION
[0013] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.
[0014] Embodiment: This embodiment is a method for optimizing video memory for training a graph-text pair model, mainly for the Transformer structure model, by setting an n value, performing similarity calculation on the tokens output by each layer of the Transformer model, and selecting the n pairs of tokens with the highest similarity to merge, thereby reducing the input and calculation amount of the next layer. Figure 1 As shown, the specific steps are as follows: S1. Calculate the similarity of each token output last time; S2, select the n token pairs with the highest similarity; S3. Merge the selected token pairs.
[0015] The step S1 is specifically as follows: The Transformer-based image-text pair model usually consists of a Transformer-structured text encoder and a Transformer-structured visual encoder (such as Vision Transformer). This solution is mainly used for visual encoders to merge redundant feature tokens in image features. For each layer output of the Transformer model, it is represented as a vector of (t, bs, dim), where t represents the number of tokens, bs represents the batch size, and dim represents the feature dimension. In order to optimize the video memory, it is necessary to calculate the similarity between each token and other tokens.
[0016] First, we divide the t tokens into two sets A and B evenly by rotating them. For each token in set A, we calculate the similarity between it and all the tokens in set B. The similarity calculation can use cosine similarity as a metric, which measures the degree of directional similarity between two vectors. Specifically, for the i-th token in set A and the j-th token in set B, the cosine similarity between them can be calculated using the following formula: cosine_similarity = dot_product(A[i], B[j]) / (norm(A[i]) * norm(B[j])), Among them, dot_product(A[i],B[j]) represents the dot product of vectors A[i] and B[j], and norm(A[i]) and norm(B[j]) represent the norms of vectors A[i] and B[j], respectively. By calculating the similarity between each token in set A and all tokens in set B, a similarity matrix of size t / 2×t / 2 can be obtained, which represents the similarity between each token in set A and each token in set B. When calculating the similarity matrix, parallel computing technology (such as GPU acceleration) can be used to speed up the calculation to adapt to large-scale token numbers and batch sizes. In this way, the similarity between each token in one set and each token in another set can be obtained, providing a basis for subsequent merge token selection and merging operations. This solution divides the two groups of tokens in a binary way in order to improve efficiency when calculating similarity matching. For the sake of calculation speed, it does not pursue the extreme compression of merging all similar tokens, but rather achieves a good balance between speed and effect.
[0017] The step S2 is specifically as follows: In this scheme, based on a set value n, we select n groups of tokens with the highest similarity from set A and set B. Each group contains a token from set A and a token from set B. n is usually initialized to 5% of the number of tokens t. Then, in each iteration of model training, the value of n is dynamically increased or decreased based on the percentage increase or decrease of the loss value of the current iteration compared to the loss value of the previous iteration. The specific formula is, n s =n s-1 ×(L s-1 / L s ), where n s and n s-1 Represent the n values of the sth and s-1th iterations, and the values are rounded to the nearest integer.s and L s-1 They represent the loss values of the sth and s-1th iterations respectively, but the value range of n will not exceed 30% of the number of tokens t, and will not be less than 5% of t. During training, each layer is counted as one iteration.
[0018] Loss value L s It is obtained by calculating the loss function used in the training of the image-text pair model. The image-text pair model usually uses contrast loss when training, specifically the InfoNCE loss function.
[0019] First, select the n values with the highest similarity from the similarity matrix and record their index positions. Then, according to these index positions, select the corresponding tokens from set A and set B respectively. Through this process, the n groups of tokens with the highest similarity in set A and set B are selected, and they will be used as input for the next merge operation. This selection method can ensure that tokens with higher similarity are selected, so that important information can be retained more effectively during the merge process. The principle of this scheme is based on the merging of redundant tokens. The characteristics of the model will lead to the existence of redundant tokens with high similarity. The number of tokens output by a layer of the model will be large, and there will be no situation where the similarity is very low.
[0020] The step S3 is specifically as follows: For the selected n groups of tokens, their values will be merged in an average way to obtain a new token. Then, the fused token will be concatenated with other tokens in the original order to obtain a new input vector as the input of the next layer of the Transformer model.
[0021] The specific steps are as follows: For the selected n groups of tokens, their values are averaged to get a new token. Specifically, assuming that the dimension of each token is dim, then each token is actually an array with a dimension of dim. The values of each corresponding dimension of a group of token arrays are added together and the average value is taken. The dimension of the fused token is still dim.
[0022] The fused token is concatenated with other tokens in the original order to obtain a new input vector. Assume that the dimension of the original input vector is (t, bs, dim), where t is the number of original tokens, bs is the batch size, and dim is the feature dimension. Since n groups of tokens are selected for merging, the dimension of the new input vector is (tn, bs, dim). This new input vector will be used as the input of the next layer of the Transformer model to continue the subsequent calculation and training process.
[0023] It should be noted that after token merging is performed at each layer, the input dimension of the next layer will be reduced, thereby reducing the memory usage. For an L-layer Transformer model, this solution can reduce the input and computation of n*L tokens. For example, for a 24-layer ViT-L-14 model with an input image size of (224, 224), the number of tokens output by the Transformer layer is usually 257. If the proportion is 10%, then n=257*10%*24=600, and the present invention can reduce the computation and memory usage of 600 tokens in total. At the same time, since tokens with higher similarity are selected for merging, important information can be effectively retained, thereby reducing the memory requirements without losing model performance.
[0024] The above are the specific steps of the memory optimization method for model training of Transformer structure graphs in this solution. By calculating the similarity between each token and other tokens and selecting the n groups of tokens with the highest similarity for merging, the memory usage can be reduced and the training efficiency and memory utilization of deep learning models in resource-constrained environments can be improved.
[0025] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
[0026] Although similarity, fusion and other terms are often used in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.
Claims
1. A method for optimizing video memory for image-text model training, characterized in that: During the training process, the tokens output by each layer are merged and the merged tokens are input to the next layer. The specific merging method includes the following steps: S1. Calculate the similarity of each token output by the previous layer; S2, select the n token pairs with the highest similarity; S3. Merge the selected token pairs.
2. The method for optimizing video memory for image-text pair model training according to claim 1, characterized in that: The step S1 is specifically as follows: S101, divide t tokens into two sets A and B evenly by allocating them in turns; t is the number of tokens output by the previous layer; S102: For each token in set A, calculate its similarity with each token in set B to obtain a similarity matrix.
3. The method for optimizing video memory for image-text model training according to claim 2, characterized in that: The similarity is calculated by the following formula: cosine_similarity = dot_product(A[i], B[j]) / (norm(A[i]) * norm(B[j])), Among them, cosine_similarity is the similarity, A[i] is the i-th token in set A, B[j] is the j-th token in set B, dot_product(A[i], B[j]) represents the dot product of vectors A[i] and B[j], norm(A[i]) and norm(B[j]) represent the norms of vectors A[i] and B[j] respectively.
4. The method for optimizing video memory for image-text pair model training according to claim 2, characterized in that: The step S2 is specifically as follows: Select the n most similar ones from the similarity matrix s values, record their index positions, and then select corresponding tokens from set A and set B according to these index positions to form token pairs; n s Determined by the following formula: n s =n s-1 ×(L s-1 / L s ), Where n s is the number of elements in the similarity matrix selected at the sth iteration, n s-1 is the number of elements in the similarity matrix selected at the s-1th iteration, L s is the loss value at the sth iteration, L s-1 is the loss value at the s-1th iteration, n1 at the first iteration is 5% of t, and n s The value range of is 5% of t to 30% of t.
5. The method for optimizing video memory for image-text pair model training according to claim 4, characterized in that: The loss value L s It is obtained by calculating the loss function used when training the model through graphs and texts.
6. A method for optimizing video memory for image-text pair model training according to claim 1 or 4, characterized in that: The step S3 is specifically as follows: For each selected token pair, the two tokens are merged in an average manner to obtain a new token; The fused new token is used to replace the token in the fused set A, and the fused token in the set B is deleted. Then the tokens in the replaced set A and the deleted set B are concatenated at their original positions to obtain a new input vector, which is the input of the image and text to the next layer of the model.
7. The method for optimizing video memory for image-text pair model training according to claim 6, characterized in that: The fusion of two tokens specifically refers to adding the values of each corresponding dimension of a pair of tokens and taking the average value.