Model fine tuning method and device based on fine-grained knowledge perception and medium

Abstract

The invention discloses a model fine tuning method and device based on fine-grained knowledge perception and a medium, and belongs to the technical field of artificial intelligence. Obtaining an initial visual data sample library and an initial text data sample library corresponding to the target field and the related field; processing the initial visual data sample library and the initial text data sample library according to a cross-modal alignment processing module; training according to a preset training task; decomposing the processed vision-text data sample library, and outputting a plurality of granularity vision-text feature vectors; performing multi-task transfer learning on the initial knowledge perception model according to the plurality of granularity vision-text feature vectors; and carrying out distillation learning on the first knowledge perception model. The technical problems of poor adaptability and poor learning effect of cross-modal data in a target domain task caused by relatively high language deviation and visual understanding difficulty existing in a pre-training model in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model fine-tuning method, device and medium based on fine-grained knowledge perception. Background Art

[0002] With the development of artificial intelligence technology, deep learning and natural language processing (NLP) have gradually become core technologies in various fields. Especially in multimodal learning, how to combine data of different modalities such as images, texts, and sounds for effective analysis has become a research focus. Multimodal learning has made significant progress in image annotation, visual question answering, autonomous driving and other fields. In order to improve the performance of multimodal models, researchers have explored technologies such as transfer learning and knowledge distillation to transfer the knowledge of pre-trained models to specific tasks and improve the generalization ability and computational efficiency of the model.

[0003] Although existing cross-modal learning methods have achieved good results in some tasks, they still have problems when dealing with fine-grained features and multi-task learning. Traditional multimodal models usually rely on coarse-grained feature sharing, which leads to unsatisfactory performance of the model in specific tasks, especially when the details of visual and text data differ greatly. In addition, existing knowledge transfer and distillation learning methods fail to effectively solve the contradiction between model accuracy and computational efficiency when dealing with complex tasks and fine-grained learning. Therefore, how to solve the language bias and difficulty of visual understanding in pre-trained models in existing technologies has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] This application aims to solve the problems of language bias and difficulty in visual understanding in pre-trained models in the prior art by providing a model fine-tuning method, device and medium based on fine-grained knowledge perception, which leads to technical problems such as poor adaptability of cross-modal data in target domain tasks and poor learning effect.

[0005] In a first aspect, the present application provides a model fine-tuning method based on fine-grained knowledge perception, the method comprising analyzing a target domain and determining related domains based on the target domain; obtaining an initial visual data sample library and an initial text data sample library corresponding to the target domain and the related domains; processing the initial visual data sample library and the initial text data sample library according to a cross-modal alignment processing module to obtain a processed visual-text data sample library; obtaining an initial knowledge perception model according to a preset training task; decomposing the processed visual-text data sample library according to a fine-grained perception module, and outputting multiple granular visual-text feature vectors based on the preset training task; performing multi-task transfer learning on the initial knowledge perception model according to the multiple granular visual-text feature vectors to obtain a first knowledge perception model; performing distillation learning on the first knowledge perception model to output a second knowledge perception model.

[0006] In a second aspect, the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is used to execute the model fine-tuning method based on fine-grained knowledge perception provided by the present application.

[0007] In a third aspect, the present application provides a computer-readable storage medium storing a computer program, which is used to execute the model fine-tuning method based on fine-grained knowledge perception provided by the present application.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] By adopting a model fine-tuning method based on fine-grained knowledge perception, including technical solutions such as relevant field analysis of the target field, cross-modal alignment processing, fine-grained feature decomposition and multi-task transfer learning, the technical problems of language bias and difficulty in visual understanding in pre-trained models in the existing technology, which lead to poor adaptability of cross-modal data in target field tasks and poor learning effect, are solved. The technical effect of enhancing the model's adaptability to target field tasks, improving the semantic understanding ability of multimodal data, and significantly improving the model's prediction accuracy and generalization performance, is achieved.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a model fine-tuning method based on fine-grained knowledge perception is provided for this application.

[0012] Figure 2 A schematic diagram of the structure of an electronic device provided in this application.

[0013] Description of the reference numerals: processor 21 , memory 22 , input device 23 , output device 24 . DETAILED DESCRIPTION

[0014] The overall idea of ​​the technical solution provided by this application is as follows:

[0015] The embodiments of the present application provide a model fine-tuning method, device and medium based on fine-grained knowledge perception. First, the target domain is analyzed and relevant domain data is introduced. The visual and text data are fused through the cross-modal alignment module. The multi-granularity features are extracted using the fine-grained perception module. The adaptability of the model is optimized through transfer learning, and finally the goal of improving the pre-trained model's understanding and adaptability to the target domain tasks is achieved.

[0016] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.

[0017] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a model fine-tuning method based on fine-grained knowledge perception, the method comprising:

[0018] Step S100: Analyze the target field and determine related fields based on the target field.

[0019] Specifically, the target domain in this application refers to the specific task or application scenario that the model hopes to be applied and optimized. For example, if the task is image description generation, the target domain may be "automatic image description generation". Related fields refer to other fields that have similarities or connections with the target domain, usually with intersections or similarities in knowledge, data, methods, etc. The selection of related fields helps to introduce diverse knowledge and features to the task.

[0020] First, the core tasks and key features of the target domain are identified. Next, based on the characteristics and task requirements of the target domain, data samples of possible related domains are collected and their relevance to the target domain is evaluated through feature analysis methods. Finally, relevant domains are screened out based on the analysis results and sorted by priority.

[0021] By analyzing the target domain and related domains, we can more effectively integrate knowledge resources from multiple domains and optimize the distribution of training data, thereby improving the training efficiency and task adaptability of the model. The characteristics of the target domain can be more fully expressed and optimized with the assistance of cross-domain knowledge.

[0022] Step S200: Acquire an initial visual data sample library and an initial text data sample library corresponding to the target field and the related field.

[0023] Specifically, the visual data sample library contains a collection of visual (image or video) data in the target field or related fields, which is used to train or fine-tune the model. The text data sample library contains a collection of text data in the target field or related fields, which is used for natural language understanding or text information analysis in multimodal tasks. The initial sample library refers to the original data set that has not been preprocessed or feature extracted, which serves as the input for subsequent model training.

[0024] First, identify possible data sources for the target domain and identified related domains. Use data crawling tools (such as Scrapy, Beautiful Soup) to collect data from the Internet or open databases; access public dataset resources in the field, such as visual data, text data, or Kaggle datasets. Download relevant data from clear data sources. For visual data, this can include pictures and videos; for text data, this can include documents, books, or technical reports.

[0025] Clean and filter the initially collected data, including removing invalid or duplicate data, to ensure high data quality and domain relevance. Use Pandas and NumPy for data cleaning and filtering. Label the visual and text data for subsequent model training. Integrate the cleaned visual and text data into the sample library based on the data relevance between the target domain and related domains. Specifically, classify and store the data according to domain or task characteristics, such as by category, chronological order, or task label.

[0026] By building an initial visual and text data sample library, sufficient multimodal data support can be provided for model training. This process ensures the diversity and relevance of data sources, laying the foundation for subsequent feature extraction and model fine-tuning.

[0027] Step S300: Processing the initial visual data sample library and the initial text data sample library according to the cross-modal alignment processing module to obtain a processed visual-text data sample library.

[0028] Specifically, the cross-modal alignment processing module is a technical module for feature extraction and fusion of visual data and text data, aiming to eliminate the differences between different modalities and map them into a unified shared feature space. The visual-text data sample library contains a data set that has been cross-modally processed and fused with visual and text features for subsequent model training or fine-tuning.

[0029] First, feature extraction is performed on the initial visual data sample library (such as pictures, videos) and text data sample library (such as descriptive text, tags). Specifically, a deep convolutional neural network (such as ResNet, VGG) is used to extract the visual feature vector of the picture. Natural language processing technology (such as BERT, GPT) is used to extract text embedding representation.

[0030] Then, a cross-modal alignment network is used to map the visual and text features into a unified shared space, so that the two have a consistent representation. Specifically, a cross-modal model is used to map the visual and text feature vectors into a unified space through a shared embedding layer. Contrastive learning is applied to optimize the similarity between visual and text feature pairs.

[0031] Finally, the aligned visual and text features are combined to generate a fused visual-text data sample library. The sample library is constructed using database tools and stored according to modality pairs or task labels.

[0032] Through the cross-modal alignment processing module, a high-quality visual-text data sample library is constructed, which effectively improves the adaptability of the multimodal model to different data types, enabling the model to demonstrate higher understanding capabilities in complex tasks involving images and texts.

[0033] Step S400: Obtain an initial knowledge perception model according to a preset training task.

[0034] Specifically, the preset training tasks refer to the task objectives and scenarios defined before model training, which are designed to design the model training method according to specific needs. For example, image classification tasks, text generation tasks, or cross-modal visual-text matching tasks. The initial knowledge perception model refers to the first version of the model obtained through training, which can perceive and process visual, textual and other modal information and their interactive relationships. This model provides the basis for subsequent fine-tuning.

[0035] First, define the training objectives of the model according to the needs of the target domain. For example: Visual tasks: image classification, object detection, scene segmentation. Text tasks: text classification, question answering, summary generation. Cross-modal tasks: image-text matching, visual question answering (VQA), image description generation. Choose the appropriate dataset and evaluation metrics based on the task.

[0036] Use the visual-text data sample library generated in the previous stage to construct training samples according to task requirements. For example: Image-text matching: Visual samples and corresponding text descriptions form positive sample pairs, and randomly generate negative sample pairs. Text generation task: Provide multiple text descriptions for each image.

[0037] Select a suitable neural network architecture. For unimodal tasks, select standard architectures such as ResNet and BERT. For cross-modal tasks, select specialized cross-modal models such as CLIP and ViLBERT. The optimizer uses AdamW or SGD to optimize model parameters. The classification task uses cross entropy loss. The cross-modal alignment task uses contrastive learning loss. During the training process, use a framework (such as PyTorch, TensorFlow) to iteratively optimize the model. After the training is completed, the model performance is evaluated through the validation set, and the best model is selected as the initial knowledge-aware model. Specifically, the validation set is used to calculate performance indicators such as accuracy, recall, F1 value, etc.

[0038] Obtaining an initial knowledge perception model through preset training tasks can provide a powerful initial model for subsequent tasks. The initial model already has the ability to understand visual, textual, and cross-modal information, which can effectively reduce the training time and data requirements for subsequent fine-tuning.

[0039] Step S500: Decomposing the processed visual-text data sample library according to the fine-grained perception module, and outputting a plurality of granular visual-text feature vectors based on the preset training task.

[0040] Specifically, the fine-grained perception module is a functional module for extracting fine features from visual and text data. By decomposing and analyzing local information in the data, it can capture detailed features in the visual and text modalities, thereby improving the model's ability to understand complex tasks. Granular visual-text feature vectors refer to more specific and detailed visual and text features decomposed from the sample library. These features can provide a more refined description of certain local information and serve specific task objectives.

[0041] First, a visual-text data sample library is used as input. Each data item in the sample library contains a joint modal feature vector z, which is a representation of the visual data and text data in a shared space. Then, z is decomposed using a fine-grained perception module to split it into multiple granular feature vectors, each of which represents a local feature or dimension in the data. For example: For an image containing a traffic light, the fine-grained perception module can extract the visual features of the "red light area" and decompose "red light" and "stop" from the text "red light means stop" as phrase-level text features.

[0042] According to the preset training tasks (such as classification, retrieval, etc.), the granular feature decomposition method is adjusted to make the generated feature vector more suitable for the task requirements. Specifically, a multi-head attention mechanism is applied to capture the multi-dimensional interaction between visual and textual modalities. Contrastive learning is used to optimize the similarity and distinction between feature vectors. The decomposed granular visual-textual feature vectors are organized into a new sample set for subsequent model training or inference.

[0043] Through the decomposition of the fine-grained perception module, the visual-text data sample library is converted into multiple more detailed feature vector sets, which can significantly improve the model's understanding and processing capabilities for complex tasks. This module ensures the model's high-precision perception and alignment of multimodal data by capturing local details of visual and textual modalities.

[0044] Step S600: performing multi-task transfer learning on the initial knowledge perception model according to the multiple granular visual-text feature vectors to obtain a first knowledge perception model.

[0045] Specifically, multi-task transfer learning is a method of transferring knowledge from multiple related tasks to the same model. By optimizing multiple tasks simultaneously, the generalization ability of the model on new tasks is improved. The first knowledge-aware model refers to the model obtained by optimizing the initial knowledge-aware model through multi-task transfer learning, which can perform well in multiple tasks.

[0046] Using the granular visual-text feature vector generated by the fine-grained perception module as input, the granular visual-text feature vector is assigned to different tasks according to the training objectives, and each task corresponds to a sub-model. Then, a transfer learning model is trained for each task. Each model starts from the initial knowledge perception model and uses granular features for fine-tuning to gradually adapt to the task requirements. Furthermore, the parameters of multiple transfer learning models are integrated through the sharing module to extract common features between tasks. Finally, through repeated iterative optimization and validation set evaluation, the final first knowledge perception model is obtained.

[0047] Through multi-task transfer learning, the initial knowledge perception model is optimized to the first knowledge perception model that adapts to multiple tasks. This method makes full use of the detailed information of granular visual-text feature vectors, improving the model's ability to process multimodal data and its adaptability to complex tasks.

[0048] Step S700: Perform distillation learning on the first knowledge-aware model and output a second knowledge-aware model.

[0049] Specifically, distillation learning is a model compression technology that reduces model complexity by training a lightweight student model to imitate a high-performance but complex teacher model while keeping performance as close to the teacher model as possible. The second knowledge-aware model refers to the student model trained by distillation learning, and its goal is to achieve performance close to the first knowledge-aware model with fewer computing resources. Soft labels refer to the probability distribution generated by the teacher model, which contains fine-grained information about the data samples and can help the student model learn more knowledge than hard labels (true category labels).

[0050] Use the trained first knowledge-aware model as the teacher model to generate soft labels. Design a smaller neural network as the student model to reduce computing resource consumption. Then, train the student model. Specifically, input samples to the teacher model to generate soft labels. Input the same samples to the student model to generate prediction probabilities. Calculate the distillation loss and update the student model parameters through the optimizer. Use the validation set to evaluate the performance of the student model to ensure that its performance on key tasks is close to that of the teacher model. Save the trained student model as the second knowledge-aware model.

[0051] Through distillation learning, the second knowledge-aware model retains the core knowledge of the teacher model while significantly reducing the model size and inference time. The introduction of soft labels enables the student model to not only learn hard label information, but also obtain richer feature knowledge by imitating the fine-grained output of the teacher model.

[0052] Furthermore, the initial visual data sample library and the initial text data sample library are processed according to a cross-modal alignment processing module to obtain a processed visual-text data sample library. The method includes: performing feature extraction on the initial visual data sample library and the initial text data sample library respectively according to the cross-modal alignment processing module, and outputting a visual feature vector and a text feature vector; wherein the cross-modal alignment processing module includes a cross-modal alignment network; and mapping the visual feature vector and the text feature vector to a shared space according to the cross-modal alignment network to obtain a processed visual-text data sample library.

[0053] Specifically, a visual feature vector refers to a numerical representation extracted from visual data (such as images or videos) to describe the key information of an image. A text feature vector refers to a numerical representation extracted from a natural language processing model to describe the semantic information of the text content. The text feature vector captures key information in the text, such as grammatical structure, theme, etc. The cross-modal alignment network is a neural network architecture that maps visual and text features into a shared space, allowing them to be compared, aligned, and learned in a unified space. This network optimizes learning to maximize the semantic similarity between different modalities. The shared space is an abstract feature space in which all data from different modalities are mapped to a common representation space. In this way, data from different modalities can be analyzed and compared in a unified manner.

[0054] First, extract features from the initial visual data sample library and text data sample library respectively. Use deep convolutional neural networks such as ResNet, VGG, or EfficientNet to extract image features. Through these networks, the main information in the image (such as edges, colors, shapes, etc.) will be converted into a vector representation. Use natural language processing models to extract feature vectors of text, which can capture the semantic information of text.

[0055] Next, a cross-modal alignment network is used to map the visual feature vector and the text feature vector to a shared space. A cross-modal alignment network is used to project visual and text features into the same space. The network learns the mapping relationship so that the visual features and text features are semantically aligned in a shared space. For example, the CLIP model optimizes the goal to make the image and text as close as possible in a high-dimensional space, so that it can process image and text data at the same time. After alignment, the generated visual feature vector and text feature vector are merged and stored as a visual-text data sample library. This sample library contains the mapping data of the visual and text modalities.

[0056] Through the cross-modal alignment processing module, visual data and text data are effectively mapped to a shared space, eliminating the differences between different modalities. In this way, the model can understand and process visual and text information at the same time, thereby improving the effect of multimodal tasks (such as classification, retrieval or generation tasks combining images and text).

[0057] Furthermore, the expression for mapping the visual feature vector and the text feature vector to the shared space according to the cross-modal alignment network is as follows: z=concat(f vision (x vision ),f text (x text )); where x vision is the initial visual data sample input, x text is the initial text data sample input, f vision (x vision ) is the extracted visual feature vector, f text (x text ) is the extracted text feature vector, z is the joint modality feature vector of vision-text, concat is the feature mapping operation, and the processed visual-text data sample library is D = {z1,z2,...,z N}, z1 is the i-th visual-text data sample after processing, and N is the total number of samples.

[0058] Specifically, concat is used to combine the feature vectors of the two modalities into a joint feature vector. That is, the visual feature vector and the text feature vector are concatenated using the concat operation to form a joint modality feature vector z. For example, if the dimension of the visual feature vector is 1024 and the dimension of the text feature vector is 768, the dimension of z after concatenation is 1792.

[0059] By mapping visual features and text features to a shared space through a cross-modal alignment network and performing concatenation operations, a feature vector sample library D that can represent joint modal information is constructed. This process significantly enhances the model's adaptability in multimodal tasks and its understanding of cross-modal relationships.

[0060] Furthermore, multi-task transfer learning is performed on the initial knowledge perception model according to the multiple granular visual-text feature vectors. The method includes: performing transfer learning on the initial knowledge perception model respectively according to the multiple granular visual-text feature vectors, and outputting multiple transfer learning models, wherein each transfer learning model includes corresponding model parameters; performing common feature extraction on the model parameters of the multiple transfer learning models through a sharing module, and iteratively optimizing the model parameters of the multiple transfer learning models according to the extracted common model parameters until a first knowledge perception model is obtained in a convergence state.

[0061] Specifically, the shared module refers to integrating the model parameters of multiple transfer learning models, extracting common features between tasks, and optimizing to generate a general model.

[0062] First, the initial knowledge perception model is transferred using multiple granular visual-text feature vectors. By adjusting the model parameters, it adapts to the task requirements corresponding to different granular features, thereby outputting multiple transfer learning models. Each transfer learning model focuses on specific granular features and optimizes related task objectives, such as classification or matching tasks. The training of the transfer learning model uses a deep learning framework, which freezes the low-level parameters of the initial knowledge perception model to retain its basic features and only updates the high-level parameters related to the task, thereby achieving rapid adaptation.

[0063] Next, the parameters of these transfer learning models are integrated through the shared module to extract common features between tasks. The shared module usually adopts a multi-task learning framework and uses a multi-head attention mechanism or a weight sharing mechanism to capture the association between tasks. Subsequently, the model parameters are iteratively optimized using the common features, and the parameters are adjusted through the optimizer (such as AdamW or SGD) so that the model gradually achieves optimal performance on all tasks.

[0064] During the optimization process, a joint loss function (such as a combination of classification loss and contrast loss) is introduced to ensure the balanced performance of the model among multiple tasks. Through repeated training and validation set evaluation, the optimization is stopped when the model performance tends to be stable, and the first knowledge perception model is finally generated. The model has the ability to adapt to multiple tasks and can complete tasks such as "traffic light detection", "lane classification" and "obstacle recognition" at the same time. It also improves the generalization ability and the ability to handle complex scenes through multi-task collaboration.

[0065] Through multi-task transfer learning, the model learns fine-grained task characteristics from multiple granular visual-text feature vectors, and integrates common features between tasks through shared modules, significantly improving the model's performance on multimodal tasks. The first knowledge perception model can not only complete multiple tasks at the same time, but also has higher generalization capabilities.

[0066] Furthermore, according to the extracted common model parameters, a joint loss function is introduced to iteratively optimize the model parameters of the multiple transfer learning models. The joint loss function includes classification loss and feature alignment loss. The expression of the joint loss function is as follows:

[0067] L total =λ1L multi-class +λ2L KL ;

[0068] Among them, L total is the joint loss function of multi-task transfer learning, λ1 is the hyperparameter for balancing classification loss, λ2 is the hyperparameter for balancing feature alignment loss, and L multi-class is the weighted sum of all task losses in multi-task transfer learning, L KL Symmetric KL loss based on feature distribution alignment for different tasks.

[0069] Specifically, the symmetric KL divergence is used to measure the similarity between the feature distributions of two different tasks in multi-task transfer learning, thereby achieving feature alignment. The formula is:

[0070]

[0071] Among them, P refers to the probability distribution of the first task based on feature alignment, and Q refers to the probability distribution of the second task based on feature alignment. i Refers to the probability value of the first task based on the probability distribution P of feature alignment on the i-th feature, Q i Refers to the probability value of the second task based on the probability distribution Q of feature alignment on the i-th feature,

[0072] By calculating the divergence of P and Q at the same time and summing the results, the symmetry is guaranteed, that is, L KL (P, Q) = L KL(Q, P) can more comprehensively reflect the difference between the two distributions, accumulate all categories i, and measure the overall distribution similarity. KL , which can make the feature distribution of different tasks in transfer learning more consistent and reduce the feature differences between tasks.

[0073] The feature embedding of the transfer learning model is extracted into the shared space, and the distribution of the feature embedding is used as the input for the symmetric KL divergence calculation. In each iteration, the task data is input, and the classification loss and feature alignment loss are calculated respectively; then they are weighted according to the weight parameters λ1 and λ2 to obtain the value of the joint loss function. The parameters are updated through back propagation so that L total Gradually converges during the training process. After multiple rounds of training and verification, when the joint loss function converges, the optimized first knowledge perception model is finally obtained. This model not only has high classification accuracy in each task, but also enhances the collaboration effect between tasks through optimized feature alignment capabilities.

[0074] By introducing the joint loss function, the multi-task transfer learning model achieves the integrated optimization of classification accuracy and feature alignment, significantly improving the generalization ability of the model and the efficiency of task collaboration. The classification loss ensures the adaptability of the model to specific tasks, while the feature alignment loss strengthens the feature sharing ability between tasks and avoids task conflicts.

[0075] Furthermore, the expression for calculating the classification loss is as follows: Among them, L multi-class is the weighted sum of all task losses in multi-task transfer learning, k is the number of tasks in multi-task transfer learning, and α j is the weight corresponding to the jth task, is the classification loss of the jth task; Among them, N j is the number of samples of the granular visual-text feature vector input in the jth task, is the true label of the i-th sample number in the j-th task, z i is the input feature vector of the i-th sample number in the j-th task, f(z i ) is the common model parameter extracted by the shared module, C j (f(z i )) is the predicted output of the j-th task for the general model parameters.

[0076] Specifically, the classification loss of each task is calculated by weighting α j A weighted sum is performed to ensure that the losses of important tasks (such as high priority tasks) contribute more to the total loss. It is used to quantify the classification error of each task. Optimizing this loss can improve the classification performance of the task. multi-class It is part of the joint loss. As an optimization objective, it combines the classification performance of multiple tasks to guide the iterative optimization of model parameters.

[0077] By weighted summation of classification losses, the model achieves comprehensive optimization of classification objectives of different tasks in multi-task transfer learning. The introduction of task weights effectively balances the importance of tasks, allowing key tasks to receive higher optimization priority. The iterative optimization of classification losses significantly improves the classification performance and task adaptability of the model.

[0078] Furthermore, the first knowledge-aware model is subjected to distillation learning to output a second knowledge-aware model, and the method includes: obtaining a candidate student model, taking the first knowledge-aware model as a teacher model, and establishing a transmission relationship between the teacher model and the candidate student model layer by layer through a transmission matrix, wherein the number of layers of the candidate student model does not match the number of layers of the first knowledge-aware model; calculating a cost matrix based on the teacher model and the candidate student model; optimizing the transmission matrix with the cost matrix through a dual EM distillation strategy to obtain an optimized transmission matrix; and according to the optimized transmission matrix, allocating the weights of different layers in the first knowledge-aware model to each layer of the candidate student model, and outputting a second knowledge-aware model.

[0079] Specifically, the candidate student model is a lightweight model with fewer layers and a simpler structure, which is used to receive knowledge from the first knowledge perception model (teaching model). It is designed to achieve efficient reasoning in scenarios with limited resources. The teacher model (teacher model) refers to the first knowledge perception model, which is a model with excellent performance but large scale, and is used to guide the training and optimization of the student model. The transfer matrix is ​​used to represent the mapping relationship between different layers of the teacher model and different layers of the candidate student model. The problem of mismatch between the number of layers of the teacher model and the student model is solved. The cost matrix is ​​used to quantify the transmission cost between the teacher model layer and the student model layer. The cost of each transmission relationship depends on the similarity or difference of the features between the layers. The dual EM distillation strategy is an optimization strategy that finds the optimal inter-layer mapping relationship by alternately optimizing the transfer matrix and the cost matrix, so as to more effectively transfer the knowledge of the teacher model to the student model.

[0080] Design a lightweight candidate student model according to the scenario requirements. Its number of layers and structure do not match the teacher model, but it needs to have enough capacity to learn the knowledge of the teacher model. Specifically, use a deep learning framework (such as PyTorch, TensorFlow) to define the structure of the student model. Then, define the transfer matrix to represent the mapping relationship between each layer of the teacher model and each layer of the student model. Due to the mismatch in the number of layers, it is necessary to introduce sparsity constraints in the transfer matrix. Initialize a random transfer matrix to represent the initial mapping relationship between each teacher model layer and the student model layer. Use sparse matrix tools to build an efficient matrix structure. Finally, by comparing the feature distribution of the teacher model layer and the student model layer, calculate the transfer cost and generate a cost matrix. Specifically, extract the output features of each layer of the teacher model and the student model respectively. Use feature similarity indicators (such as Euclidean distance, cosine similarity, or KL divergence) to calculate the cost.

[0081] Using the dual EM strategy, the optimal mapping relationship between the teacher model layer and the student model layer is found by iteratively optimizing the transfer matrix and the cost matrix. In the E step, the transfer matrix is ​​fixed and the cost matrix is ​​minimized. In the M step, the cost matrix is ​​fixed and the transfer matrix is ​​optimized to improve the effectiveness of the mapping. The E step and the M step are performed alternately until the transfer matrix converges. According to the optimized transfer matrix, the weights of different layers in the teacher model are mapped to the corresponding layers of the student model.

[0082] By distilling the first knowledge-aware model, the inter-layer mapping relationship between the teacher model and the student model is optimized using the transfer matrix and the dual EM distillation strategy. The generated second knowledge-aware model significantly reduces the computational complexity while maintaining high task performance.

[0083] Furthermore, the target domain is analyzed to determine related domains based on the target domain. The method also includes: obtaining multiple related domain samples; outputting correlation coefficient weights according to the domain correlation between the target domain and the multiple related domain samples; and configuring the data ratio of the initial visual data sample library and the initial text data sample library according to the correlation coefficient weights.

[0084] Specifically, the domain correlation coefficient weight is used to quantify the coefficient of the correlation between the target domain and each related domain, indicating the importance of each related domain to the target domain model training.

[0085] According to the task characteristics of the target domain, collect data samples from multiple related fields, including visual data and text data. Collect samples of related fields through public data sets, enterprise data resources or web crawling. Analyze the feature similarity between the target domain and samples of related fields, and calculate the correlation coefficient weight of each related field. Specifically, extract the features of visual and text data and calculate the feature similarity. Use cosine similarity or KL divergence to measure the closeness of the data distribution of the target domain and related fields. Normalize the calculation results to obtain the correlation coefficient weight of each related field.

[0086] Adjust the proportion of target domain and related domain data in the initial visual data sample library and the initial text data sample library according to the correlation coefficient weight. Specifically, redistribute data samples according to the correlation coefficient weight, increase the data in the field with high correlation, and reduce the data in the field with low correlation. Balance the sample library through sampling techniques (such as oversampling and undersampling). Use data processing tools (such as Pandas, PyTorch DataLoader) to adjust the data proportion.

[0087] By performing correlation analysis and data ratio configuration on the target field and related field data, we can better utilize the auxiliary role of related field data and improve the model performance of the target field. The scientific adjustment of data ratio can not only avoid the interference of excessive proportion of related field data on the target field, but also make full use of cross-domain knowledge for model optimization.

[0088] In summary, the model fine-tuning method based on fine-grained knowledge perception provided by the embodiment of the present application has the following technical effects:

[0089] 1. By building a model fine-tuning method based on fine-grained knowledge perception, the language bias problem of the pre-trained model can be effectively alleviated and the complexity of visual understanding can be reduced. Combined with fine-grained feature extraction and cross-modal knowledge alignment, the model's generalization ability in processing complex multimodal tasks is significantly improved, while achieving efficient adaptation to specific tasks in the target domain, thereby improving the execution effect of the task and the prediction accuracy of the model.

[0090] 2. Through the cross-modal alignment processing module, the features of the visual and text data sample libraries are extracted and mapped to a shared space, ensuring the deep integration of visual and textual modalities. The implementation of feature alignment strengthens the semantic association between modalities, enabling the model to more accurately understand the relationship between visual and textual data, thereby achieving higher performance and stronger robustness in multimodal tasks.

[0091] 3. By distilling the first knowledge perception model and optimizing it using the dual EM strategy, the problem of mismatch between the number of layers of the teacher model and the student model is solved. The performance of the second knowledge perception model after distillation is close to that of the teacher model, while significantly reducing the computational complexity and resource consumption of the model, achieving more efficient multi-task processing and model deployment.

[0092] Embodiment 2: Figure 2 The schematic diagram of the structure of the electronic device provided in the third embodiment of the present invention shows a block diagram of an exemplary electronic device suitable for implementing the implementation mode of the present invention. Figure 2 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 2 As shown, the electronic device includes a processor 21, a memory 22, an input device 23 and an output device 24; the number of processors 21 in the electronic device can be one or more. Figure 2 Taking a processor 21 as an example, the processor 21, memory 22, input device 23 and output device 24 in the electronic device can be connected through a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0093] The memory 22, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the model fine-tuning method based on fine-grained knowledge perception in the embodiment of the present application. The processor 21 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 22, that is, realizing the above-mentioned model fine-tuning method based on fine-grained knowledge perception.

[0094] Any step of the method described above can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor to implement any method in the embodiments of the present application, without any unnecessary restrictions.

[0095] Furthermore, the first or second mentioned above may not only represent an order relationship, but may also represent a specific concept, and / or refer to multiple elements that can be selected individually or in full. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. Model fine-tuning method based on fine-grained knowledge perception, characterized by: The method comprises: Analyze the target area and identify related areas based on the target area; Acquire an initial visual data sample library and an initial text data sample library corresponding to the target field and the related field; Processing the initial visual data sample library and the initial text data sample library according to the cross-modal alignment processing module to obtain a processed visual-text data sample library; According to the preset training tasks, an initial knowledge perception model is obtained; Decomposing the processed visual-text data sample library according to the fine-grained perception module, and outputting a plurality of granular visual-text feature vectors based on the preset training task; Performing multi-task transfer learning on the initial knowledge perception model according to the multiple granular visual-text feature vectors to obtain a first knowledge perception model; Perform distillation learning on the first knowledge-aware model and output a second knowledge-aware model.

2. The method according to claim 1, characterized in that The initial visual data sample library and the initial text data sample library are processed according to the cross-modal alignment processing module to obtain a processed visual-text data sample library, the method comprising: According to the cross-modal alignment processing module, feature extraction is performed on the initial visual data sample library and the initial text data sample library respectively, and a visual feature vector and a text feature vector are output; Wherein, the cross-modal alignment processing module includes a cross-modal alignment network; The visual feature vector and the text feature vector are mapped to a shared space according to the cross-modal alignment network to obtain a processed visual-text data sample library.

3. The method according to claim 2, characterized in that The expression for mapping the visual feature vector and the text feature vector to the shared space according to the cross-modal alignment network is as follows: z=concat(f vision (x vision ),f text (x text )); Among them, x vision is the initial visual data sample input, x text is the initial text data sample input, f vision (x vision ) is the extracted visual feature vector, f text (x text ) is the extracted text feature vector, z is the joint modality feature vector of vision-text, concat is the feature mapping operation, and the processed visual-text data sample library is D = {z1,z2,...,z N }, z1 is the i-th visual-text data sample after processing, and N is the total number of samples.

4. The method according to claim 1, characterized in that The initial knowledge perception model is subjected to multi-task transfer learning according to the multiple granular visual-text feature vectors, the method comprising: Performing transfer learning on the initial knowledge perception model according to the multiple granular visual-text feature vectors, and outputting multiple transfer learning models, wherein each transfer learning model includes corresponding model parameters; The common features of the model parameters of the multiple transfer learning models are extracted through a shared module, and the model parameters of the multiple transfer learning models are iteratively optimized according to the extracted common model parameters until a first knowledge perception model is obtained in a convergence state.

5. The method according to claim 4, characterized in that According to the extracted common model parameters, a joint loss function is introduced to iteratively optimize the model parameters of the multiple transfer learning models. The joint loss function includes classification loss and feature alignment loss. The expression of the joint loss function is as follows: L total =λ1L multi-class +λ2L KL ; Among them, L total is the joint loss function of multi-task transfer learning, λ1 is the hyperparameter for balancing classification loss, λ2 is the hyperparameter for balancing feature alignment loss, and L multi-class is the weighted sum of all task losses in multi-task transfer learning, L KL Symmetric KL loss based on feature distribution alignment for different tasks.

6. The method according to claim 5, characterized in that The expression for calculating the classification loss is as follows: Among them, L multi-class is the weighted sum of all task losses in multi-task transfer learning, k is the number of tasks in multi-task transfer learning, and α j is the weight corresponding to the jth task, is the classification loss of the jth task; Among them, N j is the number of samples of the granular visual-text feature vector input in the jth task, is the true label of the i-th sample number in the j-th task, z i is the input feature vector of the i-th sample number in the j-th task, f(z i ) is the common model parameter extracted by the shared module, C j (f(z i )) is the predicted output of the j-th task for the general model parameters.

7. The method according to claim 1, characterized in that Performing distillation learning on the first knowledge perception model to output a second knowledge perception model, the method comprising: Acquire a candidate student model, use the first knowledge perception model as a teacher model, and establish a transmission relationship between the teacher model and the candidate student model layer by layer through a transmission matrix, wherein the number of layers of the candidate student model does not match the number of layers of the first knowledge perception model; Calculating a cost matrix based on the teacher model and the candidate student model; The transmission matrix is ​​optimized by using the cost matrix through a dual EM distillation strategy to obtain an optimized transmission matrix; According to the optimized transfer matrix, the weights of different layers in the first knowledge-aware model are distributed to each layer of the candidate student model, and the second knowledge-aware model is output.

8. The method according to claim 1, characterized in that Analyzing the target field and determining related fields based on the target field, the method further includes: Obtain multiple samples of related fields; Outputting a correlation coefficient weight according to the domain correlation between the target domain and the plurality of related domain samples; The data ratio of the initial visual data sample library and the initial text data sample library is configured according to the correlation coefficient weight.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; Wherein, the processor is used to execute the model fine-tuning method based on fine-grained knowledge perception as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, so that the computer program is used to execute the model fine-tuning method based on fine-grained knowledge perception as described in any one of claims 1 to 8.

Images