Fine adjustment method, device and equipment of image-text large model, medium and product
By performing distillation training and low-rank matrix adjustment on the large-scale graph-text model, the problems of high computational cost and deployment difficulty of the large-scale graph-text model are solved, and efficient fine-tuning is achieved in resource-constrained environments, maintaining the model's performance and adaptability.
Patent Information
- Application Number
- CN202510598581.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-02-10
AI Technical Summary
The large number of parameters in large graph and text models leads to high computational costs and deployment difficulties, making them difficult to apply on resource-constrained mobile devices or edge devices. Furthermore, existing fine-tuning methods cannot guarantee a balance between performance and lightweight design.
The large-scale graph-text model was trained by distillation using multiple data samples. The distillation loss function and cross-entropy loss function were calculated. The low-rank matrix of the large-scale graph-text model was adjusted, especially the connection weights between the attention layer and the fully connected layer. The low-rank matrix was used for fine-tuning.
While reducing model resource consumption, it maintains the performance and generalization ability of large graph-text models, enhances the ability to capture complex cross-modal associations, and adapts to different task requirements.
Smart Images
Figure CN121503577A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method, apparatus, device, medium, and product for fine-tuning large graphic models. Background Technology
[0002] With the rapid development of Large Vision-Language Model (LVLM), it has shown excellent performance in tasks such as image description generation, visual question answering, and intelligent search. However, the large number of parameters in LVLM leads to high computational costs and difficult deployment, making it difficult to apply on resource-constrained mobile devices or edge devices.
[0003] In related technologies, lightweight techniques are often used to fine-tune large image and text models. However, the fine-tuning methods in these technologies cannot guarantee a balance between the performance of large image and text models and lightweighting. Therefore, there is an urgent need for an efficient and low-cost fine-tuning method for large image and text models that can reduce model resource consumption while maintaining the performance of large image and text models. Summary of the Invention
[0004] This disclosure provides a method, apparatus, device, medium, and product for fine-tuning large graphic models.
[0005] According to a first aspect of this disclosure, a method for fine-tuning a large graphic model is provided, the method comprising:
[0006] Multiple data samples are acquired; wherein each data sample contains image data and text data;
[0007] The large-scale image-text model is trained by distillation using the multiple data samples to obtain a first output feature; wherein, the first output feature includes the output features of each network layer in the large-scale image-text model for the image data and the text data;
[0008] The distillation loss function and the cross-entropy loss function are calculated based on the first output feature; wherein the distillation loss function is used to indicate the degree of difference between the first output feature and the teacher model's processing results on the data sample;
[0009] Based on the distillation loss function and the cross-entropy loss function, the low-rank matrix of the large graph-text model is adjusted; wherein, the low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
[0010] Furthermore, the large graph-text model includes the following network layers: a visual backbone network, a multimodal mapping connector, and a language model.
[0011] Further, the step of calculating the distillation loss function based on the first output feature includes:
[0012] A first temporal feature is determined from the first output feature; wherein, the first temporal feature includes the image output feature of the visual backbone network in the large image-text model for the image data, and the text output feature of the language model in the large image-text model for the text data;
[0013] Determine the second temporal feature of the teacher network outputting the data samples in the large graph-text model;
[0014] The distillation loss function is calculated based on the first time-domain feature and the second time-domain feature.
[0015] Further, calculating the distillation loss function based on the first time-domain features and the second time-domain features includes:
[0016] The first time-domain feature is decomposed into a frequency domain by discrete wavelet transform to obtain a first frequency domain component; the first frequency domain component includes a first high-frequency component and a first low-frequency component.
[0017] The second time-domain feature is decomposed into a frequency domain by discrete wavelet transform to obtain a second frequency domain component; the second frequency domain component includes a second high-frequency component and a second low-frequency component.
[0018] The distillation loss function is calculated based on the first frequency domain component and the second frequency domain component.
[0019] Further, calculating the distillation loss function based on the first frequency domain component and the second frequency domain component includes:
[0020] The norm of the difference between the first low-frequency component and the second low-frequency component is calculated to obtain the low-frequency distillation loss function;
[0021] The norm of the difference between the first high-frequency component and the second high-frequency component is calculated to obtain the high-frequency distillation loss function;
[0022] The distillation loss function is obtained by weighting the low-frequency distillation loss function and the high-frequency distillation loss function.
[0023] Further, calculating the distillation loss function based on the first time-domain features and the second time-domain features includes:
[0024] The first sub-loss function is calculated based on the image output features in the first time-domain features and the image output features in the second time-domain features;
[0025] The second sub-loss function is calculated based on the text output features in the first time-domain features and the text output features in the second time-domain features;
[0026] The distillation loss function is determined based on the first sub-loss function and the second sub-loss function.
[0027] Further, the step of calculating the cross-entropy loss function based on the first output feature includes:
[0028] The target output feature of the language model of the large graph-text model is determined from the first output feature;
[0029] The cross-entropy loss function is determined based on the target output features and the true values of the data samples.
[0030] Furthermore, adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function includes:
[0031] The total training loss function is obtained by weighting the distillation loss function and the cross-entropy loss function.
[0032] The low-rank matrix of the large graph-text model is adjusted based on the total training loss function; wherein the low-rank matrix includes a first matrix and a second matrix, the first matrix is used to project the connection weights from the high-dimensional space to the low-rank subspace in a dimension reduction manner, and the second matrix is used to reconstruct the connection weights from the low-rank subspace in a dimension increase manner to the high-dimensional space.
[0033] Furthermore, after adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function, the method further includes:
[0034] The first parameter matrix and the adjusted low-rank matrix in the large graph-text model are merged to obtain the fine-tuned large graph-text model; wherein, the first parameter matrix is used to indicate the connection weights between the attention layer and the fully connected layer that do not need to be adjusted.
[0035] According to a second aspect of this disclosure, a fine-tuning device for a large graphic model is provided, the device comprising:
[0036] The acquisition module is used to acquire multiple data samples; wherein each data sample contains image data and text data;
[0037] The distillation module is used to perform distillation training on the large image-text model using the multiple data samples to obtain a first output feature; wherein, the first output feature includes the output features of each network layer in the large image-text model for the image data and the text data;
[0038] The calculation module is used to calculate the distillation loss function and the cross-entropy loss function based on the first output feature; wherein the distillation loss function is used to indicate the degree of difference between the first output feature and the processing results of the teacher model on the data sample;
[0039] The fine-tuning module is used to adjust the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function; wherein, the low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
[0040] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.
[0041] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.
[0042] According to a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, implements the methods described above in this disclosure.
[0043] This disclosure provides a method, apparatus, device, medium, and product for fine-tuning a large-scale image-text model. In this embodiment, firstly, multiple data samples are acquired, each containing image data and text data. Then, the large-scale image-text model is trained using distillation with these multiple data samples to obtain a first output feature. This first output feature includes the output features of each network layer in the large-scale image-text model for the image data and text data. Next, a distillation loss function and a cross-entropy loss function are calculated based on the first output feature. The distillation loss function indicates the degree of difference between the first output feature and the teacher model's processing results for the data samples. Finally, the low-rank matrix of the large-scale image-text model is adjusted based on the distillation loss function and the cross-entropy loss function. This low-rank matrix is the low-rank matrix of the connection weights between the attention layer and the fully connected layer in the large-scale image-text model that need to be adjusted.
[0044] As described above, the technical solution of this disclosure trains a large-scale image-text model by distilling multiple data samples containing image and text data. It calculates the distillation loss function and cross-entropy loss function based on the first output features of each network layer. Then, it adjusts the low-rank matrix between the attention layer and the fully connected layer in the large-scale image-text model using the distillation loss function and cross-entropy loss function. Compared to traditional techniques that adjust the entire weight matrix of the large-scale image-text model, this technical solution adjusts only the low-rank matrix of the connection weights between the attention layer and the fully connected layer. This significantly reduces the computational and storage overhead required for model training. Simultaneously, it uses distillation technology to decouple and extract features at multiple levels and resolutions, enabling the fine-tuned large-scale image-text model to effectively learn the global and local features of the teacher model. This enhances the model's ability to capture complex cross-modal associations, thus maintaining high performance and generalization ability while reducing the number of parameters. Attached Figure Description
[0045] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0046] Figure 1 A flowchart of a method for fine-tuning a large graphic model provided as an exemplary embodiment of this disclosure;
[0047] Figure 2 A schematic diagram of a large-scale graphic model structure provided for an exemplary embodiment of this disclosure;
[0048] Figure 3 A schematic diagram of a frequency distillation frame structure provided in an exemplary embodiment of this disclosure;
[0049] Figure 4 A flowchart of a method for fine-tuning a large graphic model provided as another exemplary embodiment of this disclosure;
[0050] Figure 5 A flowchart of a method for fine-tuning a large graphic model provided as another exemplary embodiment of this disclosure;
[0051] Figure 6 A schematic diagram of a low-rank matrix structure provided in an exemplary embodiment of this disclosure;
[0052] Figure 7 A flowchart of a method for fine-tuning a large graphic model provided as another exemplary embodiment of this disclosure;
[0053] Figure 8 A schematic block diagram of the functional modules of a fine-tuning device for a large graphic model provided as an exemplary embodiment of the present disclosure;
[0054] Figure 9 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure;
[0055] Figure 10 A structural block diagram of a computer system provided as an exemplary embodiment of this disclosure;
[0056] Figure 11 A structural block diagram of a computer program product provided for an exemplary embodiment of this disclosure. Detailed Implementation
[0057] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0058] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0059] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0060] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0061] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0062] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0063] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0064] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0065] In one embodiment, such as Figure 1 As shown, a method for fine-tuning a large graphic model is provided, including the following steps:
[0066] Step 101: Obtain multiple data samples.
[0067] Here, the executing entity can obtain multiple data samples for distillation training of the large image-text model, where each data sample contains image data and text data.
[0068] In one possible embodiment, the executing entity may load a multimodal instruction following dataset, wherein the multimodal instruction following dataset consists of pairs of image and text instructions. The executing entity obtains multiple data samples by loading the multimodal instruction following dataset, each data sample containing image data and corresponding text data, wherein the text data is the text instruction corresponding to the image data.
[0069] In one possible embodiment, the executing entity can perform image preprocessing and text preprocessing on the image data and text data contained in each data sample, respectively. Image preprocessing may involve adjusting the size of the image and normalizing it, while text processing may involve segmenting the text description into words and converting them into tokens, and then encoding them into the input format. It should be noted that other methods can also be used for image preprocessing and text preprocessing, and the specific preprocessing methods are not limited here.
[0070] In one possible embodiment, for example, for image preprocessing, the execution subject uniformly scales the image to a fixed resolution, such as 224×224 pixels, to ensure consistent input size and adapt to the input requirements of the visual backbone network of the large-scale image-text model. After adjusting the image size, the execution subject standardizes the image pixel values using predefined mean and standard deviation to eliminate illumination differences in the image. For text preprocessing, the execution subject uses a pre-trained language model's tokenizer, such as BERT or GPT series tokens, to segment the text instructions into sub-word units (Tokens), converts the Token sequence into numerical IDs, and uniformly fills them to a fixed length to form a regular input matrix. After forming the input matrix, the execution subject aligns the preprocessed image and text according to samples and encapsulates them into key-value pairs for distillation training of the large-scale image-text model.
[0071] Step 102: Distill the large image-text model using multiple data samples to obtain the first output feature.
[0072] Here, after acquiring multiple data samples, the executing entity can perform distillation training on the large image-text model using multiple data samples to obtain the first output feature. The first output feature includes the output features of each network layer in the large image-text model for image data and text data.
[0073] In one possible embodiment, the large graph-text model includes the following network layers: a visual backbone network, a multimodal mapping connector, and a language model.
[0074] Specifically, such as Figure 2 As shown, Figure 2 An exemplary schematic diagram of a large-scale image-text model is shown. This large-scale image-text model includes the following network layers: a visual backbone network, a multimodal mapping connector, and a language model. The visual backbone can be a lightweight CLIP-ViT model, which is a pre-trained image feature extractor with a small number of parameters and low computational complexity. The multimodal mapping connector can be a simple multilayer perceptron (MLP) containing two fully connected layers. The language model can be a pre-trained lightweight Qwen2-1.5B model, which has been trained on a large amount of text data and possesses a certain semantic understanding capability. It should be noted that this embodiment only illustrates one structure of a large-scale image-text model; other structures can be constructed according to actual needs. The specific structure of the large-scale image-text model is not limited here.
[0075] In one possible embodiment, following the previous example, the execution entity constructs a frequency distillation framework and performs distillation training on the large-scale graph-text model in the above embodiment using multiple data samples to obtain the first output feature, such as... Figure 3 As shown, Figure 3 An exemplary schematic diagram of the frequency distillation framework is shown. This frequency distillation framework includes a large image-text model serving as a student model and a large image-text model serving as a teacher model. The executing entity inputs data samples containing image data and text data into the large image-text model serving as the student model to obtain a first output feature. The first output feature includes the output features of each network layer in the large image-text model for image data and text data. Specifically, the first output feature includes the intermediate features of the large image-text model serving as the student model. These intermediate features are the image output features of the visual backbone network in the student model for image data and the text output features of the language model in the student model for text data. In the frequency distillation framework, "image feature output" represents the image output features of the visual backbone network in the student model for image data, and "text embedding output" represents the text output features of the language model in the student model for text data.
[0076] In the above embodiments, the teacher model can be Qwen2-7B. Qwen2-7B has high language generation and multimodal processing capabilities and can provide a reference standard for the student model. It should be noted that, in addition to Qwen2-7B, the teacher model can also be flexibly selected according to the actual situation. The specific model of the teacher model is not limited here.
[0077] Step 103: Calculate the distillation loss function and the cross-entropy loss function based on the first output feature.
[0078] Here, after the executing entity performs distillation training on the large image-text model using multiple data samples to obtain the first output feature, it can calculate the distillation loss function and the cross-entropy loss function based on the first output feature. The distillation loss function is used to indicate the degree of difference between the first output feature and the teacher model's processing results on the data samples.
[0079] In one possible embodiment, such as Figure 4 As shown, the distillation loss function is calculated based on the first output feature, including the following steps:
[0080] Step 1031: Determine the first time-domain feature from the first output feature.
[0081] Here, after the execution entity performs distillation training on the large image-text model using multiple data samples to obtain the first output feature, it can determine the first temporal feature from the first output feature. The first temporal feature includes the image output feature of the visual backbone network in the large image-text model for image data, and the text output feature of the language model in the large image-text model for text data.
[0082] In one possible embodiment, the executing entity determines a first temporal feature from the first output features. The first temporal feature includes image output features of the visual backbone network in the large graph-text model for image data, and text output features of the language model in the large graph-text model for text data. Specifically, such as... Figure 3 As shown, the first time-domain features are the "image feature output" and "text embedding output" of the large image-text model, which serves as the student model in the frequency distillation framework.
[0083] Step 1032: Determine the second temporal feature of the teacher network output of the data sample in the large graph-text model.
[0084] Here, the executing entity can determine the second temporal features of the teacher network outputting data samples in the large image-text model. The second temporal features include the image output features of the visual backbone network in the teacher model for image data, and the text output features of the language model in the teacher model for text data.
[0085] In one possible embodiment, the executing entity determines a second temporal feature of the teacher network's output to the data samples in the large graph-text model. This second temporal feature includes image output features of the visual backbone network in the teacher model for image data, and text output features of the language model in the teacher model for text data. Specifically, such as... Figure 3 As shown, the second time-domain feature is the "image feature output" and "text embedding output" of the large image-text model serving as the teacher model in the frequency distillation framework.
[0086] Step 1033: Calculate the distillation loss function based on the first time-domain characteristics and the second time-domain characteristics.
[0087] Here, after the executing entity determines the first time-domain feature from the first output feature and determines the second time-domain feature of the teacher network outputting the data sample in the large graph-text model, it can calculate the distillation loss function based on the first and second time-domain features.
[0088] In one possible embodiment, calculating the distillation loss function based on the first time-domain features and the second time-domain features includes the following steps:
[0089] The first time-domain feature is decomposed into frequency domain components by discrete wavelet transform;
[0090] The second time-domain feature is decomposed into its second frequency-domain component by discrete wavelet transform.
[0091] The distillation loss function is calculated based on the first and second frequency domain components.
[0092] Specifically, such as Figure 3 As shown, the executing entity determines the first time-domain feature from the first output feature, and after determining the second time-domain feature of the teacher network outputting data samples in the large graph-text model, it performs frequency domain decomposition on the first time-domain feature using discrete wavelet transform to obtain the first frequency domain component, and then performs frequency domain decomposition on the second time-domain feature using discrete wavelet transform to obtain the second frequency domain component. The calculation formula is as follows:
[0093] L T H T =DWT(X T );
[0094] L s H S =DWT(X S ).
[0095] Among them, L T H T The first frequency domain component includes the first high-frequency component H. T and the first low-frequency component L T X T The first time-domain feature; L s H S This is the second frequency domain component, which includes the second high-frequency component H. S Second low frequency component L s X S This is the second time-domain feature.
[0096] After obtaining the first frequency domain component and the second frequency domain component, the executing entity can calculate the distillation loss function based on the first frequency domain component and the second frequency domain component.
[0097] In one possible embodiment, the distillation loss function is calculated based on the first frequency domain component and the second frequency domain component, including the following steps:
[0098] The norm of the difference between the first low-frequency component and the second low-frequency component is calculated to obtain the low-frequency distillation loss function;
[0099] The norm of the difference between the first high-frequency component and the second high-frequency component is calculated to obtain the high-frequency distillation loss function;
[0100] The distillation loss function is obtained by weighting the low-frequency distillation loss function and the high-frequency distillation loss function.
[0101] Specifically, after obtaining the first frequency domain component and the second frequency domain component, the executing entity processes the first low-frequency component L. T With the second low-frequency component L s The norm of the difference between them is calculated to obtain the low-frequency distillation loss function. And for the first high-frequency component H T With the second high-frequency component H S The norm of the difference between them is calculated to obtain the high-frequency distillation loss function. The calculation formula is:
[0102]
[0103]
[0104] The executing entity obtains the low-frequency distillation loss function. and high-frequency distillation loss function Then, the low-frequency distillation loss function and the high-frequency distillation loss function are weighted and calculated to obtain the distillation loss function.
[0105] In one possible embodiment, the executing entity calculates the distillation loss function using a first formula, which is:
[0106]
[0107] in, Let α be the distillation loss function, β be the first weight, and β be the second weight.
[0108] In one possible embodiment, the cross-entropy loss function is calculated based on the first output feature, including the following steps:
[0109] The target output features of the language model of the large graph-text model are determined from the first output features;
[0110] The cross-entropy loss function is determined based on the target output features and the true values of the data samples.
[0111] Specifically, after training the large-scale graph-text model using distillation through multiple data samples to obtain the first output feature, the executing entity can determine the target output feature of the language model of the large-scale graph-text model from the first output feature. This target output feature is the predicted output of the large-scale graph-text model. After determining the target output feature of the language model of the large-scale graph-text model from the first output feature, the executing entity determines the cross-entropy loss function L based on the target output feature and the true values of the data samples. ce .
[0112] Step 104: Adjust the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function.
[0113] Here, after the execution entity calculates the distillation loss function and the cross-entropy loss function based on the first output feature, it can adjust the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function. The low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
[0114] In one possible embodiment, such as Figure 5 As shown, adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function includes the following steps:
[0115] Step 1041: Perform a weighted calculation on the distillation loss function and the cross-entropy loss function to obtain the total training loss function.
[0116] Here, the executing entity calculates the distillation loss function based on the first output feature. and cross-entropy loss function L ce Then, the distillation loss function can be evaluated. and cross-entropy loss function L ce We perform weighted calculations to obtain the total training loss function L. total .
[0117] In one possible embodiment, the execution entity calculates the total training loss function L based on the second formula and the third weight γ. total The second formula is:
[0118]
[0119] Step 1042: Adjust the low-rank matrix of the large graph-text model based on the total training loss function.
[0120] Here, after the execution entity performs weighted calculations on the distillation loss function and the cross-entropy loss function to obtain the total training loss function, it can adjust the low-rank matrix of the large graph-text model based on the total training loss function. The low-rank matrix includes a first matrix and a second matrix. The first matrix is used to project the connection weights from the high-dimensional space to the low-rank subspace, and the second matrix is used to reconstruct the connection weights from the low-rank subspace to the high-dimensional space.
[0121] In one possible embodiment, the execution entity reduces the number of parameters that need to be trained during fine-tuning by adding a low-rank matrix between the attention layer and the fully connected layer of the large image-text model for parameter update and adjustment, thereby reducing computational and storage overhead. The parameters of the large image-text model exist in the form of a weight matrix W′. In traditional fine-tuning methods, the entire weight matrix W′ is updated, which is resource-intensive and difficult to converge. In this embodiment, the execution entity decomposes the weight matrix W′ as follows:
[0122] W′=W+ΔW.
[0123] Where W is the first parameter matrix, used to indicate the connection weights between the attention layer and the fully connected layer of the large image-text model that do not need adjustment, and ΔW is the matrix of connection weights between the attention layer and the fully connected layer of the large image-text model that need adjustment, i.e., the changing part of the weight matrix W′. The executing entity decomposes the changing part ΔW of the weight matrix W′ into two low-rank matrices A and B. ΔW can be expressed as ΔW=A×B, as follows. Figure 6 As shown, Figure 6 An exemplary diagram of a low-rank matrix structure is shown, where A is the first matrix, A∈R. d×r B is the second matrix, B∈R r×k The first matrix A is used to project the connection weights from the high-dimensional space to the low-rank subspace, that is, to project the input dimension d onto the subspace of rank r. The second matrix B is used to reconstruct the connection weights from the low-rank subspace to the high-dimensional space, that is, to reproject the low-rank subspace onto the output dimension k, where r is a low-rank value that is much smaller than d and k.
[0124] In one possible implementation, the executing agent uses the total training loss function L total Perform backpropagation to update the first matrix A and the second matrix B in the low-rank matrix of the large graph-text model. The connection weights between the attention layer and the fully connected layer of the large graph-text model that do not need to be adjusted remain unchanged, that is, the first parameter matrix W remains unchanged. Continue to iterate the above steps until the low-rank matrix satisfies the convergence condition.
[0125] In the above embodiments, the executing entity introduces low-rank matrix decomposition, which greatly reduces the number of parameters required in the low-rank matrix update process, and directly updates the weight matrix W′∈R. d×k The previous method required d×k parameters, but in the above embodiment, only the first matrix A and the second matrix B need to be updated, and the total number of parameters is d×r+r×k. Since r<<d,k, the above embodiment significantly reduces the number of parameters that need to be updated during the fine-tuning of the large graphic model.
[0126] This disclosure provides a method, apparatus, device, medium, and product for fine-tuning a large-scale image-text model. In this embodiment, firstly, multiple data samples are acquired, each containing image data and text data. Then, the large-scale image-text model is trained using distillation with these multiple data samples to obtain a first output feature. This first output feature includes the output features of each network layer in the large-scale image-text model for the image data and text data. Next, a distillation loss function and a cross-entropy loss function are calculated based on the first output feature. The distillation loss function indicates the degree of difference between the first output feature and the teacher model's processing results for the data samples. Finally, the low-rank matrix of the large-scale image-text model is adjusted based on the distillation loss function and the cross-entropy loss function. This low-rank matrix is the low-rank matrix of the connection weights between the attention layer and the fully connected layer in the large-scale image-text model that need to be adjusted.
[0127] As described above, the technical solution of this disclosure trains a large-scale image-text model by distilling multiple data samples containing image and text data. It calculates the distillation loss function and cross-entropy loss function based on the first output features of each network layer. Then, it adjusts the low-rank matrix between the attention layer and the fully connected layer in the large-scale image-text model using the distillation loss function and cross-entropy loss function. Compared to traditional techniques that adjust the entire weight matrix of the large-scale image-text model, this technical solution adjusts only the low-rank matrix of the connection weights between the attention layer and the fully connected layer. This significantly reduces the computational and storage overhead required for model training. Simultaneously, it uses distillation technology to decouple and extract features at multiple levels and resolutions, enabling the fine-tuned large-scale image-text model to effectively learn the global and local features of the teacher model. This enhances the model's ability to capture complex cross-modal associations, thus maintaining high performance and generalization ability while reducing the number of parameters.
[0128] In one embodiment, such as Figure 7 As shown, the calculation of the distillation loss function based on the first time-domain features and the second time-domain features also includes the following steps:
[0129] Step 701: Calculate the first sub-loss function based on the image output features in the first time domain features and the image output features in the second time domain features.
[0130] Here, after the executing entity determines the first temporal feature from the first output feature and determines the second temporal feature of the teacher network outputting data samples in the large image-text model, the executing entity can calculate the first sub-loss function based on the image output feature in the first temporal feature and the image output feature in the second temporal feature.
[0131] In one possible embodiment, the image output features are in the first temporal domain features. The image output features in the second time domain are defined as [0.8, 1.2, -0.3]. Given a value of [0.6, 1.0, -0.1], the executing entity outputs image features based on the first temporal domain features. Image output features in the second time domain The first sub-loss function is calculated using the following formula:
[0132]
[0133] Where, ζ img For the first sub-loss function, by and Substituting the data, the first sub-loss function ζ can be calculated. img =0.04.
[0134] Step 702: Calculate the second sub-loss function based on the text output features in the first time-domain features and the text output features in the second time-domain features.
[0135] In one possible embodiment, the text output features are among the first temporal features. The second time-domain feature is [0.3, -0.2, 0.8], representing the text output feature. Given a value of [0.4, -0.5, 1.0], the executing entity outputs features based on the text features in the first time domain. Second time-domain features of text output features The second sub-loss function is calculated using the following formula:
[0136]
[0137] Where, ζ text For the first sub-loss function, by and Substituting the data, the second sub-loss function ζ can be calculated. text =0.0467.
[0138] Step 703: Determine the distillation loss function based on the first sub-loss function and the second sub-loss function.
[0139] Here, after calculating the first sub-loss function and the second sub-loss function, the executing entity can determine the distillation loss function based on the first sub-loss function and the second sub-loss function.
[0140] In one possible embodiment, assuming the weights of the first sub-loss function and the second sub-loss function are A and B respectively, the weighted sum of the first and second sub-loss functions is calculated to obtain the distillation loss function ζ. distill The calculation formula is:
[0141] ζ distill =Aζ img +Bζ text .
[0142] In this embodiment, firstly, the executing entity calculates a first sub-loss function based on the image output features in the first time-domain features and the image output features in the second time-domain features; then, the executing entity calculates a second sub-loss function based on the text output features in the first time-domain features and the text output features in the second time-domain features; finally, the executing entity determines the distillation loss function based on the first and second sub-loss functions.
[0143] As described above, this embodiment achieves precise decoupling of multimodal features by processing image and text output features separately to calculate sub-loss functions, allowing the model to focus on key information in both images and text. This embodiment uses two sub-loss functions to determine the distillation loss function, optimizing the model's learning process. By reasonably adjusting the weights of both in the total loss, the model's emphasis on learning global and detailed information from images and text can be flexibly balanced to adapt to the needs of different tasks. Ultimately, this enables the model to learn efficiently even in low-resource environments, effectively improving the performance and generalization ability of the lightweight image-text model, and enhancing its adaptability and accuracy in practical applications.
[0144] In one embodiment, after adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function, the execution entity further includes the following steps:
[0145] The first parameter matrix and the adjusted low-rank matrix in the large graphic model are merged to obtain the fine-tuned large graphic model.
[0146] Here, after adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function, the execution entity can merge the first parameter matrix of the large graph-text model with the adjusted low-rank matrix to obtain the fine-tuned large graph-text model. The first parameter matrix is used to indicate the connection weights between the attention layer and the fully connected layer that do not need to be adjusted.
[0147] In one possible embodiment, after adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function, the executing entity merges the first parameter matrix W and the adjusted low-rank matrix W1 in the large graph-text model to obtain the target parameter matrix T, T = W + W1. Here, the first parameter matrix W is used to indicate the connection weights between the attention layer and the fully connected layer that do not need to be adjusted, and the adjusted low-rank matrix W1 is used to indicate the connection weights between the attention layer and the fully connected layer that have been adjusted. The executing entity constructs the fine-tuned large graph-text model through the target parameter matrix T.
[0148] As described above, in this embodiment, the executing entity obtains the target parameter matrix by merging the first parameter matrix that does not need adjustment with the adjusted low-rank matrix, thereby constructing a fine-tuned large-scale graph-text model. This not only preserves the stability of the core structure of the model and its original semantic understanding capabilities, but also optimizes for specific tasks through targeted adjustments to the low-rank matrix. This significantly reduces the consumption of computing resources while effectively improving the task adaptability and inference efficiency of the large-scale graph-text model.
[0149] As described above, the technical solution disclosed herein uses low-order matrix fine-tuning technology to locally fine-tune a pre-trained large-scale image-text model without requiring significant modifications to the model architecture. Users can flexibly fine-tune at different model layers to adapt to the needs of different tasks. This modular design improves the versatility and portability of the technical solution disclosed herein, enabling seamless integration with existing pre-trained large-scale models. Furthermore, the technical solution disclosed herein introduces frequency domain distillation technology, utilizing wavelet transform to decompose and decouple the intermediate feature layers of the teacher model in the frequency domain, thereby providing more granular guidance to the student model, enhancing the student model's ability to learn knowledge from the teacher model, and improving the distillation effect. This technical solution significantly reduces the computational and storage requirements of large-scale image and text models by introducing low-order matrix fine-tuning and frequency domain distillation techniques. It enables efficient training and deployment in low-resource environments. This lightweight solution reduces reliance on high-performance hardware, lowers computational costs, and improves model development and iteration speed. Furthermore, this technical solution allows large-scale image and text models to be easily deployed on edge devices and mobile applications, expanding application scenarios in fields such as smart homes, autonomous driving, and augmented reality. In addition, this technical solution also has broad application prospects in industries such as medical imaging, intelligent search and recommendation, and automated content generation, contributing to the innovative development of multimodal artificial intelligence technology.
[0150] By dividing each functional module according to its corresponding function, this disclosure provides a fine-tuning device for a large graphic model. This fine-tuning device can be a server or a chip applied to a server. Figure 8 A schematic block diagram of the functional modules of a fine-tuning device for a large graphic model provided as an exemplary embodiment of this disclosure. (See diagram for reference.) Figure 8 As shown, the fine-tuning device for this large graphic model includes:
[0151] The acquisition module 801 is used to acquire multiple data samples; wherein each data sample includes image data and text data;
[0152] Distillation module 802 is used to perform distillation training on the large image-text model using the multiple data samples to obtain a first output feature; wherein, the first output feature includes the output features of each network layer in the large image-text model for the image data and the text data; the large image-text model includes the following network layers: visual backbone network, multimodal mapping connector and language model;
[0153] The calculation module 803 is used to calculate the distillation loss function and the cross-entropy loss function based on the first output feature; wherein the distillation loss function is used to indicate the degree of difference between the first output feature and the processing result of the teacher model on the data sample;
[0154] The fine-tuning module 804 is used to adjust the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function; wherein, the low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
[0155] In one embodiment, the computing module 803 includes:
[0156] A first determining unit is configured to determine a first temporal feature from the first output feature; wherein the first temporal feature includes the image output feature of the visual backbone network in the large image-text model for the image data, and the text output feature of the language model in the large image-text model for the text data;
[0157] The second determining unit is used to determine the second temporal features of the teacher network outputting the data samples in the large graph-text model;
[0158] The first calculation unit is used to calculate the distillation loss function based on the first time-domain feature and the second time-domain feature.
[0159] In one embodiment, the computing module 803 includes:
[0160] The first decomposition unit is used to perform frequency domain decomposition on the first time domain feature through discrete wavelet transform to obtain a first frequency domain component; the first frequency domain component includes: a first high-frequency component and a first low-frequency component.
[0161] The second decomposition unit is used to perform frequency domain decomposition on the second time domain feature through discrete wavelet transform to obtain the second frequency domain component; the second frequency domain component includes: a second high-frequency component and a second low-frequency component.
[0162] The second calculation unit is used to calculate the distillation loss function based on the first frequency domain component and the second frequency domain component.
[0163] In one embodiment, the computing module 803 includes:
[0164] The third calculation unit is used to perform norm calculation on the difference between the first low-frequency component and the second low-frequency component to obtain the low-frequency distillation loss function.
[0165] The fourth calculation unit is used to perform norm calculation on the difference between the first high-frequency component and the second high-frequency component to obtain the high-frequency distillation loss function;
[0166] The fifth calculation unit is used to perform weighted calculations on the low-frequency distillation loss function and the high-frequency distillation loss function to obtain the distillation loss function.
[0167] In one embodiment, the computing module 803 includes:
[0168] The sixth calculation unit is used to calculate the first sub-loss function based on the image output features in the first time-domain features and the image output features in the second time-domain features;
[0169] The seventh calculation unit is used to calculate the second sub-loss function based on the text output features in the first time-domain features and the text output features in the second time-domain features;
[0170] The third determining unit is used to determine the distillation loss function based on the first sub-loss function and the second sub-loss function.
[0171] In one embodiment, the computing module 803 includes:
[0172] The fourth determining unit is used to determine the target output features of the language model of the large graph-text model from the first output features;
[0173] The fifth determining unit determines the cross-entropy loss function based on the target output features and the true values of the data samples.
[0174] In one embodiment, the fine-tuning module 804 includes:
[0175] The eighth calculation unit is used to perform weighted calculations on the distillation loss function and the cross-entropy loss function to obtain the total training loss function;
[0176] An adjustment unit is used to adjust the low-rank matrix of the large image-text model based on the total training loss function; wherein the low-rank matrix includes a first matrix and a second matrix, the first matrix is used to project the connection weights from the high-dimensional space to the low-rank subspace in a dimension reduction manner, and the second matrix is used to reconstruct the connection weights from the low-rank subspace in a dimension increase manner to the high-dimensional space.
[0177] In one embodiment, the apparatus further includes:
[0178] The merging module is used to merge the first parameter matrix and the adjusted low-rank matrix in the large image-text model to obtain the fine-tuned large image-text model; wherein, the first parameter matrix is used to indicate the connection weights between the attention layer and the fully connected layer that do not need to be adjusted.
[0179] Figure 9 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 9 As shown, the electronic device 900 includes at least one processor 901 and a memory 902 coupled to the processor 901. The processor 901 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.
[0180] The processor 901 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 901 or by software instructions. The processor 901 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 902, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 901 reads information from the memory 902 and, in conjunction with its hardware, completes the steps of the method described above.
[0181] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 10 The computer system 1000 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 10 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.
[0182] Computer system 1000 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0183] like Figure 10 As shown, the computer system 1000 includes a computing unit 1001, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the computer system 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0184] Multiple components in the computer system 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device capable of inputting information into the computer system 1000. The input unit 1006 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1007 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 1008 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1009 allows the computer system 1000 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ device, WiFi device, WiMax device, cellular communication device, and / or the like.
[0185] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 1002 and / or communication unit 1009. In some embodiments, the computing unit 1001 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).
[0186] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.
[0187] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0188] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0189] Figure 11 A computer program product 1100 is provided as an exemplary embodiment of the present disclosure. The computer program product 1100 includes a computer program 1101, wherein the computer program 1101, when executed by a processor, implements the methods disclosed in the embodiments of the present disclosure.
[0190] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof. These programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.
[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0192] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.
[0193] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0194] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0195] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.
Claims
1. A method for fine-tuning a large graphic model, characterized in that, include: Multiple data samples are acquired; wherein each data sample contains image data and text data; The large-scale image-text model is trained by distillation using the multiple data samples to obtain a first output feature; wherein, the first output feature includes the output features of each network layer in the large-scale image-text model for the image data and the text data; The distillation loss function and the cross-entropy loss function are calculated based on the first output feature; wherein the distillation loss function is used to indicate the degree of difference between the first output feature and the processing results of the teacher model on the data sample; Based on the distillation loss function and the cross-entropy loss function, the low-rank matrix of the large graph-text model is adjusted; wherein, the low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
2. The method according to claim 1, characterized in that, The large graph-text model includes the following network layers: visual backbone network, multimodal mapping connector and language model.
3. The method according to claim 1, characterized in that, The calculation of the distillation loss function based on the first output feature includes: A first temporal feature is determined from the first output feature; wherein, the first temporal feature includes the image output feature of the visual backbone network in the large image-text model for the image data, and the text output feature of the language model in the large image-text model for the text data; Determine the second temporal feature of the teacher network outputting the data samples in the large graph-text model; The distillation loss function is calculated based on the first time-domain feature and the second time-domain feature.
4. The method according to claim 3, characterized in that, The step of calculating the distillation loss function based on the first time-domain features and the second time-domain features includes: The first time-domain feature is decomposed into a frequency domain by discrete wavelet transform to obtain a first frequency domain component; the first frequency domain component includes a first high-frequency component and a first low-frequency component. The second time-domain feature is decomposed into a frequency domain by discrete wavelet transform to obtain a second frequency domain component; the second frequency domain component includes a second high-frequency component and a second low-frequency component. The distillation loss function is calculated based on the first frequency domain component and the second frequency domain component.
5. The method according to claim 4, characterized in that, The calculation of the distillation loss function based on the first frequency domain component and the second frequency domain component includes: The norm of the difference between the first low-frequency component and the second low-frequency component is calculated to obtain the low-frequency distillation loss function; The norm of the difference between the first high-frequency component and the second high-frequency component is calculated to obtain the high-frequency distillation loss function; The distillation loss function is obtained by weighting the low-frequency distillation loss function and the high-frequency distillation loss function.
6. The method according to claim 3, characterized in that, The step of calculating the distillation loss function based on the first time-domain features and the second time-domain features includes: The first sub-loss function is calculated based on the image output features in the first time-domain features and the image output features in the second time-domain features; The second sub-loss function is calculated based on the text output features in the first time-domain features and the text output features in the second time-domain features; The distillation loss function is determined based on the first sub-loss function and the second sub-loss function.
7. The method according to claim 1, characterized in that, The calculation of the cross-entropy loss function based on the first output feature includes: The target output feature of the language model of the large graph-text model is determined from the first output feature; The cross-entropy loss function is determined based on the target output features and the true values of the data samples.
8. The method according to claim 1, characterized in that, The adjustment of the low-rank matrix of the large-scale graph-text model based on the distillation loss function and the cross-entropy loss function includes: The total training loss function is obtained by weighting the distillation loss function and the cross-entropy loss function. The low-rank matrix of the large graph-text model is adjusted based on the total training loss function; wherein the low-rank matrix includes a first matrix and a second matrix, the first matrix is used to project the connection weights from the high-dimensional space to the low-rank subspace in a dimension reduction manner, and the second matrix is used to reconstruct the connection weights from the low-rank subspace in a dimension increase manner to the high-dimensional space.
9. The method according to claim 1, characterized in that, After adjusting the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function, the method further includes: The first parameter matrix and the adjusted low-rank matrix in the large graph-text model are merged to obtain the fine-tuned large graph-text model; wherein, the first parameter matrix is used to indicate the connection weights between the attention layer and the fully connected layer that do not need to be adjusted.
10. A fine-tuning device for a large graphic model, characterized in that, include: The acquisition module is used to acquire multiple data samples; wherein each data sample contains image data and text data; The distillation module is used to perform distillation training on the large image-text model using the multiple data samples to obtain a first output feature; wherein, the first output feature includes the output features of each network layer in the large image-text model for the image data and the text data; The calculation module is used to calculate the distillation loss function and the cross-entropy loss function based on the first output feature; wherein the distillation loss function is used to indicate the degree of difference between the first output feature and the processing results of the teacher model on the data sample; The fine-tuning module is used to adjust the low-rank matrix of the large graph-text model based on the distillation loss function and the cross-entropy loss function; wherein, the low-rank matrix is the low-rank matrix of the connection weights that need to be adjusted between the attention layer and the fully connected layer in the large graph-text model.
11. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-9.
13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-9.