Model fine tuning method and related product

Abstract

The invention discloses a model fine tuning method and related products, and the method comprises the steps: obtaining a network structure and parameters of a first model, and constructing a second model based on the network structure and parameters of the first model; determining a dimension and a core tensor of a calculation layer in the second model; performing decomposition processing on the calculation layer in the second model based on the dimension and the core tensor of the calculation layer to obtain a decomposed second model; performing training adjustment on parameters of a plurality of decomposed calculation layers in the decomposed second model based on the fine tuning training data set to obtain a trained second model; and obtaining an identification result based on the first model, the trained second model and the to-be-identified text information. According to the method, through the core tensor, the spatial local characteristics of convolution are effectively reserved. And the second model can learn the training data set corresponding to the first model in the training process, so that the forgetting performance of the second model in the training process is further reduced.

Description

Technical Field

[0001] The present application relates to the field of model fine-tuning technology, and in particular to a model fine-tuning method and related products. Background Art

[0002] With the development of deep learning and neural networks, the application areas of deep learning and neural networks are constantly increasing. However, precisely because of the continuous development of deep learning and neural networks, general models may not be able to meet the needs of certain specific scenarios. For example, in the field of Optical Character Recognition (OCR), OCR models have made significant progress and can accurately recognize text information from various types of images. However, in actual applications, due to factors such as font changes, typesetting differences, and image quality, existing general OCR models may not meet the needs of certain specific scenarios. To address the above problems, researchers have proposed fine-tuning pre-trained OCR models through transfer learning to adapt to new data sets or task requirements.

[0003] Model parameter fine-tuning reduces the model training time by reducing the model training parameters. Reparameterization technology is usually used to fine-tune some parameters of the trained model. In large language models, the parameters that usually need to be adjusted are matrices. The reparameterization technology is to add an incremental matrix to the matrix to be adjusted. When fine-tuning the model, the matrix to be adjusted is not adjusted, but only the added incremental matrix is adjusted, thereby achieving the purpose of efficient fine-tuning. Several low-rank adaptations (LoRA) are used to add incremental matrices to the matrix to be adjusted. However, in practice, it is found that the incremental matrix determined based on LoRA is completely independent of the matrix to be adjusted, and this method mainly processes two-dimensional data by converting it into natural language processing. That is, when determining the incremental matrix, the knowledge contained in the matrix to be adjusted in the multidimensional space is not considered, resulting in serious model forgetting during fine-tuning training. Summary of the Invention

[0004] Based on the above problems, this application provides a model fine-tuning method and related products, the purpose of which is to reduce the forgetfulness of the model during training.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, the present application provides a method for fine-tuning a model, comprising: The network structure and parameters of the first model are obtained, and the second model is constructed based on the network structure and parameters of the first model.

[0006] Determine the dimensions and core tensors of computational layers in the second model; the computational layers include convolutional layers and fully connected layers.

[0007] The computing layer in the second model is decomposed based on the dimension and core tensor of the computing layer to obtain a decomposed second model; the decomposed second model includes multiple decomposed computing layers.

[0008] Based on the fine-tuning training data set, the parameters of multiple decomposed computing layers in the second model after the decomposition processing are trained and adjusted to obtain the trained second model; the fine-tuning training data set includes a first training data set and a second training data set; the first training data set is constructed based on the training data set corresponding to the first model; the second training data set is constructed based on the training data set corresponding to the scenario to be learned.

[0009] A recognition result is obtained based on the first model, the trained second model and the text information to be recognized.

[0010] Optionally, decomposing the computation layer in the second model based on the dimension and core tensor of the computation layer to obtain the decomposed second model specifically includes: If the computing layer is a convolutional layer, the dimension representing the computing layer is four-dimensional, and the convolutional layer in the second model is decomposed based on the dimension of the convolutional layer and the core tensor to obtain a decomposed convolutional layer.

[0011] If the computing layer is a fully connected layer, the dimension representing the computing layer is two-dimensional, and the fully connected layer in the second model is decomposed based on the dimension of the fully connected layer and the core tensor to obtain a decomposed fully connected layer.

[0012] Based on the decomposed convolutional layer and the decomposed fully connected layer, a second model after the decomposition processing is obtained.

[0013] Optionally, decomposing the convolutional layer in the second model based on the dimension and core tensor of the convolutional layer to obtain a decomposed convolutional layer specifically includes: ; Among them, △W1 represents the four-dimensional weight matrix corresponding to the decomposed convolutional layer; Represents the core tensor of the convolutional layer of the second model; A1 represents the first one-dimensional weight matrix corresponding to the decomposed convolutional layer; B1 represents the second one-dimensional weight matrix corresponding to the decomposed convolutional layer; C1 represents the third one-dimensional weight matrix corresponding to the decomposed convolutional layer; D1 represents the fourth one-dimensional weight matrix corresponding to the decomposed convolutional layer.

[0014] Optionally, decomposing the fully connected layer in the second model based on the dimension and core tensor of the fully connected layer to obtain a decomposed fully connected layer specifically includes: ; Among them, △W2 represents the two-dimensional weight matrix corresponding to the decomposed fully connected layer; Represents the core tensor of the fully connected layer of the second model; A2 represents the first one-dimensional weight matrix corresponding to the decomposed fully connected layer; B2 represents the second one-dimensional weight matrix corresponding to the decomposed fully connected layer.

[0015] Optionally, the method further includes: Get the training dataset corresponding to the first model.

[0016] Redundant data filtering is performed on the training data set corresponding to the first model to obtain a third training data set.

[0017] Noise filtering is performed on the third training data set to obtain the first training data set.

[0018] A second training data set is constructed based on the training data set corresponding to the scenario to be learned.

[0019] A fine-tuning training dataset is constructed based on the first training dataset and the second training dataset.

[0020] Optionally, performing redundant data filtering on the training data set corresponding to the first model to obtain a third training data set specifically includes: The training data set corresponding to the first model is sampled to obtain a fourth training data set.

[0021] The third model is constructed based on the neural network.

[0022] The third model is trained based on the fourth training data set to obtain a trained third model.

[0023] The trained third model is used to predict each sample in the training data set corresponding to the first model, and a prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model is obtained.

[0024] The confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model are determined based on the prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model.

[0025] The training data set corresponding to the first model is processed by filtering redundant data based on the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model to obtain a third training data set.

[0026] Optionally, performing noise filtering on the third training data set to obtain the first training data set specifically includes: The first model is used to predict each sample in the third training data set to obtain a prediction result of the first model corresponding to each sample in the third training data set.

[0027] The confidence and word accuracy corresponding to each sample in the third training data set are determined based on the prediction result of the first model corresponding to each sample in the third training data set.

[0028] The third training dataset is subjected to noise filtering based on the confidence and word accuracy corresponding to each sample in the third training dataset to obtain the first training dataset.

[0029] Optionally, obtaining a recognition result based on the first model, the trained second model, and the text information to be recognized specifically includes: The parameters of the computational layer in the trained second model are added to the first model to obtain the first model after parameter adjustment.

[0030] The text information to be recognized is recognized based on the first model after the parameter adjustment to obtain a recognition result.

[0031] A second aspect of the present application provides a model fine-tuning device comprising: The model acquisition module is used to acquire the network structure and parameters of the first model and construct the second model based on the network structure and parameters of the first model.

[0032] A core tensor determination module is used to determine the dimension and core tensor of the computing layer in the second model; the computing layer includes a convolutional layer and a fully connected layer.

[0033] A decomposition module is used to decompose the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain a decomposed second model; the decomposed second model includes multiple decomposed computing layers.

[0034] A training module is used to train and adjust the parameters of multiple decomposed computing layers in the second model after the decomposition processing based on a fine-tuning training data set to obtain a trained second model; the fine-tuning training data set includes a first training data set and a second training data set; the first training data set is constructed based on the training data set corresponding to the first model; the second training data set is constructed based on the training data set corresponding to the scenario to be learned.

[0035] The recognition module is used to obtain a recognition result based on the first model, the trained second model and the text information to be recognized.

[0036] The third aspect of the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model fine-tuning method provided in the first aspect.

[0037] Compared with the existing technology, this application has the following beneficial effects: This application obtains the network structure and parameters of the first model, and constructs a second model based on the network structure and parameters of the first model; determines the dimension and core tensor of the computing layer in the second model; the computing layer includes a convolutional layer and a fully connected layer; based on the dimension and core tensor of the computing layer, the computing layer in the second model is decomposed to obtain a decomposed second model; the decomposed second model includes multiple decomposed computing layers; based on a fine-tuning training data set, the parameters of the multiple decomposed computing layers in the decomposed second model are trained and adjusted to obtain a trained second model; the fine-tuning training data set includes a first training data set and a second training data set; the first training data set is constructed based on the training data set corresponding to the first model; the second training data set is constructed based on the training data set corresponding to the scenario to be learned; based on the first model, the trained second model and the text information to be recognized, a recognition result is obtained.

[0038] The present application determines the core tensor and dimension of the computing layer, and decomposes the computing layer based on the core tensor and dimension, thereby reconstructing the computing layer. Unlike the method based on low-rank matrix decomposition, the present application does not need to change the structure of the computing layer. On the contrary, through the core tensor, the spatial local characteristics of the convolution are effectively retained, reducing the forgetfulness of the second model during the training process. And the fine-tuning training set of the present application includes the training data set of the first model, so that the second model can learn the training data set corresponding to the first model during the training process, further reducing the forgetfulness of the second model during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of a model fine-tuning method provided in an embodiment of the present application; Figure 2 A schematic diagram of the decomposition of the convolutional layer provided in the embodiment of the present application; Figure 3 A schematic diagram of the decomposition of the fully connected layer provided in an embodiment of the present application; Figure 4 A flowchart of constructing a fine-tuning training dataset provided in an embodiment of the present application; Figure 5 A structural diagram of a model fine-tuning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] As described above, current model parameter fine-tuning is mainly based on LoRA to determine the incremental matrix, and then train the incremental matrix to achieve the purpose of efficient fine-tuning. However, this method does not consider the knowledge contained in the matrix to be adjusted in multidimensional space when determining the incremental matrix, resulting in serious model forgetting during fine-tuning training.

[0042] In view of the above problems, the present application provides a method for model fine-tuning and related products, the method including: obtaining the network structure and parameters of the first model, and constructing a second model based on the network structure and parameters of the first model; determining the dimension and core tensor of the computing layer in the second model; decomposing the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain a decomposed second model; training and adjusting the parameters of multiple decomposed computing layers in the decomposed second model based on the fine-tuning training data set to obtain a trained second model; obtaining a recognition result based on the first model, the trained second model and the text information to be recognized.

[0043] This application identifies the core tensors and dimensions of computational layers (including convolutional layers and fully connected layers). This process provides accurate basic data for subsequent decomposition operations, ensuring that the decomposition is targeted and efficient. Based on the determined core tensors and dimensions, the computational layer is decomposed and reconstructed. Unlike traditional low-rank matrix decomposition methods, this application does not require changing the original structure of the computational layer. By using core tensors, the spatial locality of the convolutional layer is effectively preserved, allowing the decomposed model to still capture important spatial information, thereby improving the model's expressiveness and accuracy.

[0044] During the decomposition and reconstruction process, the original structure of the computational layer is maintained, reducing interference with existing knowledge and significantly reducing the forgetfulness of the second model during training. The fine-tuning training set used in this application not only contains newly collected data, but also includes the training data set of the first model. This allows the second model to learn and inherit the knowledge of the first model during training. In this way, the second model can further adapt to new task requirements without losing the original knowledge, further reducing the forgetfulness during training.

[0045] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0046] Figure 1 A flow chart of a model fine-tuning method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the methods for model fine-tuning include: S101: Acquire the network structure and parameters of the first model, and construct a second model based on the network structure and parameters of the first model.

[0047] This application does not limit the first model. For example, the first model includes an OCR model, which is used to accurately identify text information from various types of images.

[0048] As an optional embodiment, the first model is an OCR model, and the network structure and parameters of the OCR model are obtained. For example, the network structure of the OCR model includes an input layer, a convolutional layer, a fully connected layer, and an output layer; then the network structure of the second model constructed based on the OCR model also includes an input layer, a convolutional layer, a fully connected layer, and an output layer.

[0049] The purpose of obtaining the network structure and parameters of the first model is to build a similar second model based on the first model's structure, preparing for subsequent parameter adjustments based on the second model. Reusing the design and knowledge of the first model provides an efficient starting point for the second model, avoiding starting training from scratch and saving time and resources.

[0050] S102: Determine the dimension and core tensor of the computing layer in the second model.

[0051] Clarify the key properties of the computational layer (such as dimensions and core tensors) to lay the foundation for subsequent decomposition operations and ensure that the decomposition process is targeted and controllable.

[0052] This application does not limit the computing layer. For example, the computing layer includes a convolutional layer and a fully connected layer.

[0053] S103: Decompose the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain a decomposed second model.

[0054] The decomposed second model includes multiple decomposed calculation layers.

[0055] This application does not limit the decomposition method. As an optional embodiment: Computational layers include convolutional layers and fully connected layers. The dimensions and core tensors of the convolutional and fully connected layers are determined separately. The weight matrix of the convolutional layer is decomposed based on the dimensions of the convolutional layer to obtain multiple decomposed convolutional layers. The convolutional layer is obtained by multiplying the multiple decomposed convolutional layers with the core tensor of the convolutional layer. The weight matrix of the fully connected layer is decomposed based on the dimensions of the fully connected layer to obtain multiple decomposed fully connected layers. The fully connected layer is obtained by multiplying the multiple decomposed fully connected layers with the core tensor of the fully connected layer.

[0056] By decomposing computational layers (such as convolutional and fully connected layers), complex computational layers are split into multiple smaller and simpler sub-layers. This decomposition significantly reduces the computational complexity and storage requirements of the model while retaining the core functionality of the original model.

[0057] S104: Based on the fine-tuning training data set, the parameters of the multiple decomposed computing layers in the decomposed second model are trained and adjusted to obtain a trained second model.

[0058] This application does not limit the fine-tuning training dataset. As an optional embodiment, the fine-tuning training set includes a first training dataset and a second training dataset; the first training dataset is constructed based on the training dataset corresponding to the first model; the second training dataset is constructed based on the training dataset corresponding to the scenario to be learned. Constructing the fine-tuning training dataset based on the training dataset corresponding to the first model ensures that the second model can learn the training dataset corresponding to the first model when trained based on the fine-tuning training set, thereby reducing the second model's forgetfulness during training.

[0059] As an optional embodiment, in the process of training the decomposed second model using a fine-tuning training data set, except for the calculation layer, the other layers in the decomposed second model are in a frozen state, and only the parameters of the multiple decomposed calculation layers in the decomposed second model are adjusted, thereby further reducing the forgetting tendency of the second model during training.

[0060] S105: Obtain a recognition result based on the first model, the trained second model and the text information to be recognized.

[0061] This application does not limit the method for determining the recognition result. As an optional embodiment, the parameters of the calculation layer in the trained second model are added to the first model to obtain the first model with adjusted parameters; the text information to be recognized is recognized based on the first model with adjusted parameters to obtain the recognition result.

[0062] This application directly adds the parameters of the computing layer in the trained second model to the first model, which not only ensures that the parameters in the first model include parameters that have learned the task characteristics of the scene to be learned, but also ensures that the network structure of the first model will not change.

[0063] Combining the knowledge gained from the first model with the optimization capabilities of the trained second model improves the ability to understand and analyze the text information being recognized. Through this collaborative mechanism, the resulting recognition results are both efficient and accurate, meeting the needs of practical applications.

[0064] The above is the main technical solution of this application. Now we will introduce the expansion and implementation of the main technical solution of this application. The details are as follows: With respect to S103 of decomposing the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain the decomposed second model, the present application provides an optional embodiment: There are two problems with full parameter fine-tuning: the first is the model forgetting problem. If the full amount of data + the data of the scene to be learned are not used, the model's effect in the original scene will be greatly reduced, that is, the model will forget the original knowledge. The second is the efficiency problem. Full parameter fine-tuning consumes the most computational energy. Based on the above problems, this application adopts a fine-tuning solution, using a small amount of original data + the data of the scene to be learned to perform fine-tuning in the scene to be learned, maintaining the effect of the original scene.

[0065] If the computing layer is a convolutional layer, the dimension representing the computing layer is four-dimensional, and the convolutional layer in the second model is decomposed based on the dimension of the convolutional layer and the core tensor to obtain a decomposed convolutional layer.

[0066] If the computing layer is a fully connected layer, the dimension representing the computing layer is two-dimensional, and the fully connected layer in the second model is decomposed based on the dimension of the fully connected layer and the core tensor to obtain a decomposed fully connected layer.

[0067] Based on the decomposed convolutional layer and the decomposed fully connected layer, a second model after the decomposition processing is obtained.

[0068] Taking the OCR model as an example, the OCR model weight matrix is formulated in the N-dimensional parameter space. Specifically, for a pre-trained weight matrix , the model updates the parameter △W, and the formula is as follows: ; in, ,and , J N Much smaller than I N This application will core tensor is regarded as a low-rank core space with the same topological structure as the original parameter space, where A (N)Represents the Nth one-dimensional weight matrix; Represents an I1×I2× ×I N dimensional real number space; Represents an I N ×J N dimensional real number space; N represents the number of dimensions; I N Indicates the size of the pre-training weight matrix in N dimensions; J N Indicates A (N) The size in dimension N. Scale △W by a constant s.

[0069] For example, both the first model and the second model are OCR models. Figure 2 The convolutional layer decomposition diagram provided in the embodiment of the present application is as follows: Figure 2 As shown, the weight matrix of the convolutional layer in the first model is a 4-dimensional parameter space. The weight matrix of the convolutional layer of the first model can be expressed as , Indicates a d in ×d out ×k×k dimensional real number space; based on W c Obtain the updated weight matrix of the convolutional layer of the second model. In order to maintain the spatial locality of the convolution, decompose the updated weight matrix of the convolutional layer of the second model according to the number of spatial dimensions to obtain the four-dimensional weight matrix △W1 corresponding to the decomposed convolutional layer. The formula is as follows: ; Among them, d in represents the input channel of the first model; d out Represents the output channel of the first model; k represents the convolution kernel size of the first model; A1 represents the first one-dimensional weight matrix corresponding to the decomposed convolution layer; B1 represents the second one-dimensional weight matrix corresponding to the decomposed convolution layer; C1 represents the third one-dimensional weight matrix corresponding to the decomposed convolution layer; D1 represents the fourth one-dimensional weight matrix corresponding to the decomposed convolution layer; , , and ; Represents a r1×r2×r3×r3 dimensional real number space; Indicates a d in ×r1-dimensional real number space; Indicates a d out ×r2-dimensional real number space; r1 and r2 represent sizes much smaller than {d in , d out} rank; r3 is a value much smaller than rank; r3 is usually set to {1, 2}; represents the core tensor of the convolutional layer of the second model, can be viewed as a compressed convolution parameter space. Essentially, As the core tensor of convolution, it means that there is a core tensor in any convolution layer. The goal of this application is to determine this core tensor for each convolution space and configure the corresponding weight matrices A1, B1, C1, and D1 to reconstruct the space. Unlike methods based on low-rank matrix decomposition, this application does not require changing the structure of the convolution. Instead, by learning the core tensor, the spatial locality of the convolution is effectively preserved.

[0070] Furthermore, the number of trainable parameters is significantly reduced while maintaining or even potentially enhancing the representation power of the convolution process. Assuming the rank of both the input and output dimensions is uniform (r1=r2=r), the parameter requirements of this application are calculated as ,Therefore, as the number of cores increases, it shows better parameter efficiency, and O represents the upper bound of the time complexity or space complexity of the algorithm.

[0071] Figure 3 The fully connected layer decomposition diagram provided in the embodiment of this application is as follows Figure 3 As shown in , the linear layer (fully connected layer) in the neural network multiplies the input data by the weight matrix through matrix multiplication to obtain the output. Assume that the input vector is , the weight matrix is , then the bias is , then the output y of the linear layer is: y=Wx+b; in, represents an n-dimensional real number space; Represents an m×n dimensional real number space; represents an m-dimensional real space; r represents a constant; the weight matrix of the fully connected layer based on the first model , we get the updated weight matrix of the fully connected layer corresponding to the second model, and decompose the updated weight matrix of the fully connected layer of the second model according to the number of spatial dimensions to get the two-dimensional weight matrix △W2 corresponding to the decomposed fully connected layer, which is expressed as: ; in, represents a d1×d2 dimensional real number space; d1 and d2 are constants; , , Similar to the convolutional layer, the core tensor of the fully connected layer of the second model is can be viewed as the core space of a two-dimensional parameter space, where Represents an r1×r2-dimensional real number space; Represents a d1×r1 dimensional real number space; Represents a d1×r2-dimensional real space; A2 represents the first one-dimensional weight matrix corresponding to the decomposed fully connected layer; B2 represents the second one-dimensional weight matrix corresponding to the decomposed fully connected layer; A2 and B2 are the corresponding weights used to reconstruct changes in linear space.

[0072] This application decomposes and reconstructs the computational layer based on the determined core tensors and dimensions. Unlike traditional low-rank matrix decomposition methods, this application does not require changing the original structure of the computational layer. By using core tensors, the spatial locality of the convolutional layer is effectively preserved, allowing the decomposed model to still capture important spatial information, thereby improving the model's expressiveness and accuracy.

[0073] Currently, there are data cleansing issues with fast model customization. Taking the OCR model as an example, the scale of OCR task training data often reaches tens of millions. Due to the scarcity of real data and the cost of annotation, OCR task training data is primarily based on synthetic data. However, the sources of synthetic data corpora are highly overlapping and are influenced by human preferences. The generated data has a long-tail distribution in terms of font, word distribution, and background diversity. This extreme distribution leads to two major problems with the training data. The first is data redundancy. The frequency of common characters is as high as hundreds of thousands, while the frequency of some rare characters is only dozens to hundreds. Redundant data has no benefit to model learning and instead exacerbates data category imbalance, resulting in insufficient model learning or even forgetting of difficult examples. The second is noisy data. The annotated data is affected by uncontrollable factors, such as homographs and different codes caused by incomplete splicing and ambiguous annotations, which makes it difficult for the model to converge on this type of data.

[0074] The main purpose of data cleaning is to reduce redundant and noisy data, thereby reducing training scale and time. Existing image data cleaning methods primarily include statistical methods, manual detection, and keyword aggregation. These methods directly delete image data from datasets that have too few categories, are ambiguous, duplicated, or contain a lot of noise. Manual detection is the most intuitive method for image screening. Its advantages lie in its accuracy and resistance to accidental deletion. Its disadvantages lie in its high labor costs and slow speed. When faced with massive amounts of image data, manual image cleaning is inefficient. Alternatively, algorithms can be used to automatically filter low-quality image data. For example, these methods calculate image similarity, select images with high similarity, and filter out duplicate data. Alternatively, trained models can be used to perform classification predictions on original images, directly deleting images with low classification accuracy to achieve data cleaning. However, the algorithm-based cleaning method does not take into account the long-tail distribution of fonts, word distribution, and background diversity in OCR tasks. When the stability or robustness of the algorithm or model is not high enough, it is easy to predict normally related categories as wrong categories, or the category prediction is correct but the prediction score is low, which will directly delete the original data, resulting in the accidental deletion of valid data, or invalid data will be retained due to model misclassification, ultimately leading to unsatisfactory data cleaning results.

[0075] To address the above issues, this application also provides an optional embodiment for obtaining a fine-tuning training set: Get the training dataset corresponding to the first model.

[0076] Redundant data filtering is performed on the training data set corresponding to the first model to obtain a third training data set.

[0077] This application does not limit the method for processing redundant data. As an optional embodiment: The training data set corresponding to the first model is sampled to obtain a fourth training data set.

[0078] By sampling the first training dataset, the training time and computing resource requirements can be reduced while retaining sufficient diversity to ensure effective training of the third model.

[0079] The third model is constructed based on the neural network.

[0080] The third model is trained based on the fourth training data set to obtain a trained third model.

[0081] Using the sampled fourth training dataset to train the third model ensures both training efficiency and the model's expressiveness on key samples. This approach facilitates rapid iteration and model optimization.

[0082] The trained third model is used to predict each sample in the training data set corresponding to the first model, and a prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model is obtained.

[0083] By letting the trained third model predict all samples of the original training dataset, detailed prediction results for each sample can be obtained, providing a basis for further evaluating the quality of the samples.

[0084] The confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model are determined based on the prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model.

[0085] By analyzing the prediction results, each sample can be assigned a confidence and word accuracy score, which can quantify the quality of the sample and help identify high-quality and low-quality samples.

[0086] The training data set corresponding to the first model is processed by filtering redundant data based on the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model to obtain a third training data set.

[0087] By removing samples with low confidence and word accuracy, we can significantly reduce noise and redundant information in the training dataset. This filtering process improves the quality of the dataset, making the trained model more accurate and stable, reducing the risk of overfitting, and improving overall performance.

[0088] Noise filtering is performed on the third training data set to obtain the first training data set.

[0089] This application does not limit the method for filtering noise. As an optional embodiment: The first model is used to predict each sample in the third training data set to obtain a prediction result of the first model corresponding to each sample in the third training data set.

[0090] The confidence and word accuracy corresponding to each sample in the third training data set are determined based on the prediction result of the first model corresponding to each sample in the third training data set.

[0091] The third training dataset is subjected to noise filtering based on the confidence and word accuracy corresponding to each sample in the third training dataset to obtain the first training dataset.

[0092] A second training data set is constructed based on the training data set corresponding to the scenario to be learned.

[0093] A fine-tuning training dataset is constructed based on the first training dataset and the second training dataset.

[0094] This application also provides a specific embodiment: Figure 4 The flowchart of constructing a fine-tuning training data set provided in the embodiment of the present application is as follows: Figure 4 As shown, the old training set (training data A) is randomly sampled to form a small-scale training data, and the sampling multiple in this application is 1 / 20.

[0095] Randomly initialize the OCR model and use the sampled training data to train the initialized OCR model. Set several epochs for training to obtain a low-precision model, for example, set the epoch to 3.

[0096] Use the low-precision model to predict each sample in the training data A, obtain the confidence and word accuracy, and then filter the redundant data through the scoring module to obtain the training data B.

[0097] The OCR model with the highest word accuracy trained using training data A is selected as the high-precision model.

[0098] Use the high-precision model to predict each sample in the training data B, obtain the confidence and word accuracy, and then filter the noise data through the scoring module to obtain the final training data.

[0099] This application also provides an optional embodiment of the scoring module: The scoring module is determined by three parameters: confidence, word accuracy and word frequency. This application only counts the word frequency of Chinese samples, and only scores English samples based on the other two parameters. For Chinese samples, the i-th sample d' i Frequency of occurrence F i , the calculation formula is as follows: F i =n' i / N'; Among them, N' represents the total number of Chinese words, n' i Indicates the number of times the i-th sample appears.

[0100] F i Perform regularization processing to obtain the frequency F of the i-th sample after regularization processing i norm : F i norm =log(F i +￠) / max(log(F j +￠)); Among them, F j represents the frequency of occurrence of the jth sample; ￠ represents a small constant (such as 1e-5) to prevent log(0).

[0101] The confidence a of the i-th sample in the training data A i , the formula is as follows: ; Among them, M represents the total number of samples in the i-th order, A i M Represents the confidence of the i-th sample in the M-th calculation.

[0102] The word accuracy c of the i-th sample i The calculation formula is as follows: c i =H / N'; in, Represents the number of correct words predicted in the i-th sample.

[0103] The scoring formula for Chinese scenes is as follows: ; in, , and Respectively represent the confidence weight, word accuracy weight and word frequency weight. , and Set to 0.3, 0.3, 0.4. S1 i Indicates the Chinese scene score value of the i-th sample, S1 i When the value is greater than 0.8, the i-th sample is considered to be a redundant sample, and when it is less than 0.1, the i-th sample is considered to be a noise sample and is removed from the training data A.

[0104] The scoring formula for English scenes is as follows: ; Among them, S2 i Indicates the English scene score value of the i-th sample; and Set to 0.5 and 0.5.

[0105] By decomposing the computational layer, this application significantly reduces the computational complexity and storage requirements of the model, improving operational efficiency. The lightweight design makes the model more suitable for deployment in resource-constrained scenarios. The decomposed model can be fine-tuned according to specific task requirements, is highly adaptable, and can be optimized for different application scenarios. Key features are retained during the decomposition process, and the model's performance is further restored and optimized during the fine-tuning phase, ensuring high accuracy of the recognition results.

[0106] Figure 5 A structural diagram of a model fine-tuning device provided in an embodiment of the present application, such as Figure 5 As shown, based on a model fine-tuning method provided in the previous embodiment, this application also provides a model fine-tuning device including: The model acquisition module is used to acquire the network structure and parameters of the first model and construct the second model based on the network structure and parameters of the first model.

[0107] A core tensor determination module is used to determine the dimension and core tensor of the computing layer in the second model; the computing layer includes a convolutional layer and a fully connected layer.

[0108] A decomposition module is used to decompose the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain a decomposed second model; the decomposed second model includes multiple decomposed computing layers.

[0109] A training module is configured to train and adjust parameters of multiple decomposed computational layers in the decomposed second model based on a fine-tuning training dataset to obtain a trained second model. The fine-tuning training dataset includes a first training dataset and a second training dataset; the first training dataset is constructed based on a training dataset corresponding to the first model; and the second training dataset is constructed based on a training dataset corresponding to the scenario to be learned.

[0110] The recognition module is used to obtain a recognition result based on the first model, the trained second model and the text information to be recognized.

[0111] As an optional embodiment, the decomposition module includes: A convolutional layer decomposition unit is used to decompose the convolutional layer in the second model based on the dimension of the convolutional layer and the core tensor to obtain a decomposed convolutional layer if the computing layer is a convolutional layer.

[0112] A fully connected layer decomposition unit is used to decompose the fully connected layer in the second model based on the dimension of the fully connected layer and the core tensor if the computing layer is a fully connected layer, so as to obtain a decomposed fully connected layer.

[0113] The second model determination unit after decomposition processing is used to obtain the second model after decomposition processing based on the decomposed convolutional layer and the decomposed fully connected layer.

[0114] As an optional embodiment, the convolutional layer decomposition unit is specifically configured to: ; Among them, △W1 represents the four-dimensional weight matrix corresponding to the decomposed convolutional layer; Represents the core tensor of the convolutional layer of the second model; A1 represents the first one-dimensional weight matrix corresponding to the decomposed convolutional layer; B1 represents the second one-dimensional weight matrix corresponding to the decomposed convolutional layer; C1 represents the third one-dimensional weight matrix corresponding to the decomposed convolutional layer; D1 represents the fourth one-dimensional weight matrix corresponding to the decomposed convolutional layer.

[0115] As an optional embodiment, the fully connected layer decomposition unit is specifically configured to: ; Among them, △W2 represents the two-dimensional weight matrix corresponding to the decomposed fully connected layer; Represents the core tensor of the fully connected layer of the second model; A2 represents the first one-dimensional weight matrix corresponding to the decomposed fully connected layer; B2 represents the second one-dimensional weight matrix corresponding to the decomposed fully connected layer.

[0116] As an optional embodiment, the device further includes: The first acquisition unit is used to acquire a training data set corresponding to the first model.

[0117] The redundancy processing unit is used to perform redundant data filtering processing on the training data set corresponding to the first model to obtain a third training data set.

[0118] The noise processing unit is configured to perform noise filtering on the third training data set to obtain the first training data set.

[0119] The second training data set determining unit is configured to construct a second training data set based on the training data set corresponding to the scenario to be learned.

[0120] A fine-tuning training data set determining unit is configured to construct a fine-tuning training data set based on the first training data set and the second training data set.

[0121] As an optional embodiment, the redundant processing unit includes: The sampling processing subunit is used to perform sampling processing on the training data set corresponding to the first model to obtain a fourth training data set.

[0122] The third module constructs a subunit for constructing a third model based on a neural network.

[0123] A training subunit is used to train the third model based on the fourth training data set to obtain a trained third model.

[0124] The first prediction subunit is used to use the trained third model to predict each sample in the training data set corresponding to the first model, and obtain the prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model.

[0125] The confidence and word accuracy first calculation subunit is used to determine the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model based on the prediction results of the trained third model corresponding to each sample in the training data set corresponding to the first model.

[0126] The redundancy processing subunit is used to filter out redundant data from the training data set corresponding to the first model based on the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model to obtain a third training data set.

[0127] As an optional embodiment, the noise processing unit includes: The second prediction subunit is used to use the first model to predict each sample in the third training data set, and obtain a prediction result of the first model corresponding to each sample in the third training data set.

[0128] The confidence and word accuracy second calculation subunit is used to determine the confidence and word accuracy corresponding to each sample in the third training data set based on the prediction result of the first model corresponding to each sample in the third training data set.

[0129] The noise processing subunit is configured to perform noise filtering processing on the third training data set based on the confidence and word accuracy corresponding to each sample in the third training data set to obtain the first training data set.

[0130] As an optional embodiment, the identification module includes: An adding unit is used to add the parameters of the calculation layer in the trained second model to the first model to obtain the first model with adjusted parameters.

[0131] The recognition unit is used to perform recognition processing on the text information to be recognized based on the first model after the parameters are adjusted to obtain a recognition result.

[0132] An embodiment of the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a model fine-tuning method.

[0133] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements a method for model fine-tuning when the computer program is executed by a processor.

[0134] An embodiment of the present application provides a computer program product, including a computer program, which implements a method for model fine-tuning when executed by a processor.

[0135] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and apparatus embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0136] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for fine-tuning a model, characterized in that: The model fine-tuning method includes: Obtaining the network structure and parameters of the first model, and constructing the second model based on the network structure and parameters of the first model; Determining the dimensions and core tensors of the computational layers in the second model; the computational layers include convolutional layers and fully connected layers; Decomposing the computing layer in the second model based on the dimension and core tensor of the computing layer to obtain a decomposed second model; the decomposed second model includes a plurality of decomposed computing layers; Training and adjusting parameters of multiple decomposed computing layers in the decomposed second model based on a fine-tuning training dataset to obtain a trained second model; the fine-tuning training dataset includes a first training dataset and a second training dataset; the first training dataset is constructed based on a training dataset corresponding to the first model; and the second training dataset is constructed based on a training dataset corresponding to the scenario to be learned; A recognition result is obtained based on the first model, the trained second model and the text information to be recognized.

2. The model fine-tuning method according to claim 1, characterized in that: Decomposing the computing layer in the second model based on the dimension of the computing layer and the core tensor to obtain the decomposed second model specifically includes: If the computing layer is a convolutional layer, the dimension representing the computing layer is four-dimensional, and the convolutional layer in the second model is decomposed based on the dimension of the convolutional layer and the core tensor to obtain a decomposed convolutional layer; If the computing layer is a fully connected layer, the dimension representing the computing layer is two-dimensional, and the fully connected layer in the second model is decomposed based on the dimension of the fully connected layer and the core tensor to obtain a decomposed fully connected layer; Based on the decomposed convolutional layer and the decomposed fully connected layer, a second model after the decomposition processing is obtained.

3. The model fine-tuning method according to claim 2, characterized in that: Decomposing the convolutional layer in the second model based on the dimension and core tensor of the convolutional layer to obtain a decomposed convolutional layer specifically includes: ； Among them, △W1 represents the four-dimensional weight matrix corresponding to the decomposed convolutional layer; Represents the core tensor of the convolutional layer of the second model; A1 represents the first one-dimensional weight matrix corresponding to the decomposed convolutional layer; B1 represents the second one-dimensional weight matrix corresponding to the decomposed convolutional layer; C1 represents the third one-dimensional weight matrix corresponding to the decomposed convolutional layer; D1 represents the fourth one-dimensional weight matrix corresponding to the decomposed convolutional layer.

4. The model fine-tuning method according to claim 2, characterized in that: Decomposing the fully connected layer in the second model based on the dimension and core tensor of the fully connected layer to obtain a decomposed fully connected layer specifically includes: ； Among them, △W2 represents the two-dimensional weight matrix corresponding to the decomposed fully connected layer; Represents the core tensor of the fully connected layer of the second model; A2 represents the first one-dimensional weight matrix corresponding to the decomposed fully connected layer; B2 represents the second one-dimensional weight matrix corresponding to the decomposed fully connected layer.

5. The model fine-tuning method according to claim 1, characterized in that: The method further comprises: Obtaining a training data set corresponding to the first model; Performing redundant data filtering on the training data set corresponding to the first model to obtain a third training data set; Performing noise filtering on the third training data set to obtain the first training data set; Constructing a second training data set based on the training data set corresponding to the scenario to be learned; A fine-tuning training dataset is constructed based on the first training dataset and the second training dataset.

6. The model fine-tuning method according to claim 5, characterized in that: The performing redundant data filtering on the training data set corresponding to the first model to obtain the third training data set specifically includes: Sampling the training data set corresponding to the first model to obtain a fourth training data set; Building a third model based on neural network; Training the third model based on the fourth training data set to obtain a trained third model; Using the trained third model to predict each sample in the training data set corresponding to the first model, obtaining a prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model; Determining the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model based on the prediction result of the trained third model corresponding to each sample in the training data set corresponding to the first model; The training data set corresponding to the first model is processed by filtering redundant data based on the confidence and word accuracy corresponding to each sample in the training data set corresponding to the first model to obtain a third training data set.

7. The model fine-tuning method according to claim 5, characterized in that: The performing noise filtering on the third training data set to obtain the first training data set specifically includes: Using the first model to predict each sample in the third training data set, obtaining a prediction result of the first model corresponding to each sample in the third training data set; Determining the confidence and word accuracy corresponding to each sample in the third training dataset based on the prediction result of the first model corresponding to each sample in the third training dataset; The third training dataset is subjected to noise filtering based on the confidence and word accuracy corresponding to each sample in the third training dataset to obtain the first training dataset.

8. The model fine-tuning method according to claim 1, characterized in that: Obtaining a recognition result based on the first model, the trained second model, and the text information to be recognized specifically includes: Adding the parameters of the computational layer in the trained second model to the first model to obtain the first model after parameter adjustment; The text information to be recognized is recognized based on the first model after the parameter adjustment to obtain a recognition result.

9. A device for fine-tuning a model, characterized in that: The model fine-tuning device includes: A model acquisition module, configured to acquire the network structure and parameters of the first model and construct a second model based on the network structure and parameters of the first model; A core tensor determination module, configured to determine the dimensions and core tensors of the computational layers in the second model; the computational layers include convolutional layers and fully connected layers; a decomposition module, configured to decompose the computational layers in the second model based on the dimensions and core tensors of the computational layers to obtain a decomposed second model; the decomposed second model comprising a plurality of decomposed computational layers; A training module is configured to train and adjust parameters of multiple decomposed computing layers in the decomposed second model based on a fine-tuning training dataset to obtain a trained second model; the fine-tuning training dataset includes a first training dataset and a second training dataset; the first training dataset is constructed based on a training dataset corresponding to the first model; and the second training dataset is constructed based on a training dataset corresponding to the scenario to be learned; The recognition module is used to obtain a recognition result based on the first model, the trained second model and the text information to be recognized.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model fine-tuning method according to any one of claims 1 to 8.

Images