Multi-modal model training fine tuning optimization method based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a multi-modal model training fine-tuning optimization method based on artificial intelligence, which comprises the following specific steps: preparing and preprocessing data, collecting a large-scale multi-modal data set, constructing an image text data set containing one million high-quality images and text description accurately corresponding to the images, and performing fine-tuning optimization on the image text data set. According to the method, data are subjected to comprehensive and meticulous cleaning operation, noise data and mistakenly labeled data in the data are strictly removed, so that the accuracy and reliability of the data are ensured, and the effective utilization rate of the data can be increased by at least 20% through data cleaning; wherein the image data is screened, blurred, low-resolution and unclear-content images are removed, it is ensured that each image can clearly convey specific visual information, and text description, grammar error checking, spelling error and semantic ambiguity parts are corrected or deleted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based multimodal model training fine-tuning optimization method. Background Art

[0002] With the continuous development of artificial intelligence technology, multimodal models have shown great potential in various fields. Multimodal models can process multiple different types of data at the same time, such as images, text, audio, etc., so as to better understand and analyze complex real-world information. However, training and fine-tuning multimodal models still face many challenges.

[0003] On the one hand, the collection and preprocessing of large-scale multimodal datasets is a difficult task. The quality and diversity of the data directly affect the performance of the model, while noise and mislabeling in the data may cause the model to learn the wrong patterns. In addition, the selection and application of data augmentation techniques also need to be carefully considered to avoid excessive enhancement that may cause data distortion or loss of original features.

[0004] On the other hand, the architecture design and training strategy of the multimodal model are also crucial. Advanced architectures such as the multimodal variant of the Transformer architecture have powerful representation capabilities, but they also bring huge computational complexity and parameter scale. How to improve training efficiency without losing performance and how to effectively integrate information from different modalities are still urgent problems to be solved. At the same time, the adjustment of hyperparameters and the evaluation and monitoring of the model are also key links in the training process, which require continuous experiments and optimization. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the shortcomings of the prior art, the present invention provides an artificial intelligence-based multimodal model training fine-tuning optimization method, which has the advantages of improving data quality and diversity, and solves the problem that the quality and diversity of data directly affect the performance of the model.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solution: a multimodal model training fine-tuning optimization method based on artificial intelligence, comprising the following specific steps:

[0009] S1 Data preparation and preprocessing: collect large-scale multimodal data sets, build an image-text data set containing 1 million high-quality images and accurately corresponding text descriptions, perform comprehensive and detailed data cleaning operations, strictly remove noise data and incorrectly labeled data, so as to ensure the accuracy and reliability of the data. Through data cleaning, the effective utilization rate of data can be increased by at least 20%;

[0010] The image data is screened to remove blurry, low-resolution, and unclear images to ensure that each image can clearly convey specific visual information. For text descriptions, grammatical errors, spelling errors, and unclear semantics are checked and corrected or deleted.

[0011] Among them, a data quality assessment indicator system was established to quantitatively evaluate data from multiple dimensions, including image clarity, text accuracy, and the relevance of images and texts, to ensure that only high-quality data is used for model training;

[0012] Among them, data enhancement operations are performed, such as random cropping, rotation, and flipping of images, as well as random replacement of some words, deletion of words, and insertion of words in text, which effectively expands the data set to 1.5 times or even more of the original size, thereby significantly improving the generalization ability of the model;

[0013] The S2 model architecture is optimized and adopts advanced multimodal fusion architecture, such as the multimodal variant of the Transformer architecture. It can efficiently process information in both image and text modes and has extremely powerful representation capabilities. The model parameter scale can reach billions of levels. Such a large parameter space ensures that the model can learn rich multimodal features.

[0014] The multimodal variant of the Transformer architecture usually consists of multiple encoder and decoder layers, where images and text are encoded by different encoders, and then the features of the two modalities are fused through the fusion layer. The fused features are then decoded by the decoder to output the final prediction result;

[0015] To improve the performance of the model, the Transformer architecture can be improved by increasing the number of heads in the multi-head attention mechanism, increasing the number of layers of the encoder and decoder, and using a larger hidden layer dimension. These improvements can improve the representation ability of the model, but will also increase the computational complexity and training time of the model.

[0016] S3 training strategy optimization adopts a phased training strategy. First, pre-training is performed on a large-scale dataset to fully learn the general multimodal features. Then, fine-tuning is performed on a small-scale dataset for a specific task to improve the performance of the model for a specific task. The number of training steps in the pre-training phase is 1 million or more, and the number of training steps in the fine-tuning phase is 100,000 steps.

[0017] In the pre-training stage, unsupervised learning or self-supervised learning methods, contrastive learning, and masked language models can be used. These methods can exploit the inherent structure and correlation in large-scale data to learn common multimodal features, laying a solid foundation for the subsequent fine-tuning stage;

[0018] In the fine-tuning stage, the model is supervised by using the labeled data of the specific task. The structure and parameters of the model can be adjusted according to the characteristics of the task to improve the adaptability and performance of the model to the specific task. In the image classification task, the last layer output of the model can be adjusted to the number of classification categories, and the cross entropy loss function can be used for training.

[0019] S4 hyperparameter adjustment, adjust the learning rate, adopt a learning rate decay strategy, such as cosine annealing decay, set the initial learning rate to 0.001, and gradually reduce the learning rate as the training progresses to improve the convergence speed and stability of the model;

[0020] The learning rate is an important hyperparameter in model training. It determines the step size of model parameter update. If the initial learning rate is set too high, the model may not converge. If it is set too low, the training speed may be too slow. By adopting the learning rate decay strategy, the learning rate can be gradually reduced during the training process, so that the model can maintain a good convergence speed and stability at different training stages.

[0021] Cosine annealing decay is a commonly used learning rate decay strategy, which decays the learning rate in the form of a cosine function. At the beginning of training, the learning rate is large, and as the training progresses, the learning rate gradually decreases and eventually approaches zero. This decay strategy can make the model converge quickly in the early stages of training and perform fine-tuning more stably in the later stages of training.

[0022] S5 Evaluation and monitoring: During the training process, the model is evaluated regularly, and the accuracy, recall, and F1 value indicators are used to measure the performance of the model. An evaluation is performed every 1,000 training steps, and the performance of the model on the validation set is recorded.

[0023] Accuracy refers to the ratio of the number of samples predicted correctly by the model to the total number of samples, recall refers to the ratio of the number of positive samples predicted correctly by the model to the number of actual positive samples, and F1 value is the harmonic mean of accuracy and recall. These indicators can measure the performance of the model from different perspectives. In multimodal tasks, it is usually necessary to comprehensively consider multiple indicators to evaluate the performance of the model.

[0024] During the training process, regular model evaluation can help you understand the model's training progress and performance in a timely manner so that you can adjust the training strategy and hyperparameters. The dataset can be divided into training set, validation set, and test set. The validation set can be used for evaluation during the training process, and the test set can be used for the final performance evaluation after the training is completed.

[0025] Preferably, in S1, the image data enhancement may be set to different cropping ratios, rotation angle ranges and flipping modes to increase data diversity, the random cropping ratio may vary between 10% and 30% of the image side length, the rotation angle may be randomly selected between -30 degrees and 30 degrees, and the flipping mode may include horizontal flipping and vertical flipping;

[0026] Text data enhancement can use synonym replacement, random word deletion, and random word insertion methods. Randomly select a word in the text and replace it with its synonym, or randomly delete or insert a word with a certain probability. This can increase the diversity of the text and enable the model to learn richer language expressions.

[0027] Preferably, in S2, an attention mechanism is introduced to enable the model to focus on important information between different modalities more accurately. In the image-text fusion layer, the use of the self-attention mechanism allows the model to automatically learn the correspondence between the image and the text, greatly improving the fusion effect. The computational complexity of the attention mechanism is about 30% of the total computational complexity of the model, but the performance improvement it brings is significant;

[0028] The self-attention mechanism can calculate the correlation between different positions in the image and text, so as to determine which positions are more important for the current task. In the image-text matching task, the self-attention mechanism can help the model find the area in the image that is most relevant to the text description, and the words in the text that are most relevant to the image.

[0029] In addition to the self-attention mechanism, a cross-modal attention mechanism can also be introduced to further enhance the interaction between images and texts. The cross-modal attention mechanism can calculate the bidirectional attention between images and texts, that is, the attention of images to texts and the attention of texts to images, so as to better integrate the information of the two modalities.

[0030] Preferably, in S3, mixed precision training is used to increase the model calculation speed by about 2 times without losing the model accuracy. At the same time, gradient accumulation technology is used. When hardware resources are limited, large batch training can be effectively performed by accumulating gradients of multiple small batches, thereby further improving training efficiency.

[0031] Mixed precision training means using both single-precision floating-point numbers (float32) and half-precision floating-point numbers (float16) for calculations during model training. Half-precision floating-point numbers are faster than single-precision floating-point numbers, but have lower accuracy. By properly arranging the calculation order and data type conversion, the calculation speed of the model can be improved without losing the model accuracy.

[0032] Preferably, in S4, the batch size is adjusted, and a suitable batch size is selected according to the hardware resources and the size of the data set. Generally speaking, a larger batch size can improve the training efficiency, but it may also cause the model to overfit. After experiments, it is determined that when the batch size is 64, a better balance is achieved between performance and efficiency.

[0033] Batch size refers to the number of samples input into the model each time it is trained. A larger batch size can make full use of hardware resources and improve computing efficiency, but it may also cause the model to overfit because the model sees fewer samples in each batch and is prone to falling into a local optimal solution. A smaller batch size can increase the randomness of the model and improve the generalization ability of the model, but it will also reduce training efficiency.

[0034] When choosing a batch size, you need to consider hardware resources, data set size, and model complexity. You can experiment with different batch sizes, observe the performance and training time of the model, and choose a batch size that strikes a good balance between performance and efficiency.

[0035] Preferably, in S5, the changing trends of the training loss and validation loss of the model are monitored. If the training loss continues to decrease while the validation loss no longer decreases or even increases, it indicates that the model may be overfitting. In this case, early stopping or regularization measures can be adopted to prevent overfitting.

[0036] Preferably, the optimization method can significantly improve the performance and training efficiency of AI-based multimodal models, and provide stronger support for the application of multimodal tasks.

[0037] Preferably, in S5, the training loss refers to the loss function value of the model on the training set, and the validation loss refers to the loss function value of the model on the validation set. The decrease in the training loss indicates that the performance of the model on the training set is constantly improving, but this does not necessarily mean that the performance of the model in practical applications will also improve. If the validation loss no longer decreases or even increases, it means that the model may be overfitted, that is, the model overfits the noise and specific patterns of the data on the training set, resulting in a decrease in performance on the validation set.

[0038] Preferably, in S5, the early stopping method is a commonly used method to prevent overfitting, which stops training when the verification loss no longer decreases to prevent the model from continuing to overfit the training data. Regularization is another method to prevent overfitting, which limits the complexity of the model by adding a regularization term to the loss function, thereby improving the generalization ability of the model. Common regularization methods include L1 regularization, L2 regularization and Dropout.

[0039] Preferably, in S4, the gradient accumulation technology refers to accumulating the gradients of multiple small batches and then performing a parameter update when hardware resources are limited, so that large batch training can be effectively performed to improve the convergence speed and performance of the model. When the hardware can only support a batch size of 32, the gradients of 4 batches can be accumulated and then a parameter update can be performed, which is equivalent to training with a batch size of 128.

[0040] (III) Beneficial effects

[0041] Compared with the prior art, the present invention provides a multimodal model training fine-tuning optimization method based on artificial intelligence, which has the following beneficial effects:

[0042] 1. This AI-based multimodal model training fine-tuning optimization method improves data quality and diversity: through careful data collection, cleaning and enhancement operations, it ensures data accuracy and reliability, while increasing data diversity, which enables the model to learn richer multimodal features, improves the generalization ability of the model, and can better adapt to different tasks and scenarios. At the same time, the solution optimizes the model architecture: using advanced Transformer architecture multimodal variants, combined with attention mechanisms, improves the model's representation ability and attention to different modal information, which enables the model to more accurately capture the correspondence between images and texts, thereby improving the effect of multimodal fusion and providing stronger support for more complex multimodal tasks.

[0043] 2. This AI-based multimodal model training fine-tuning optimization method improves training efficiency: the application of phased training strategy, mixed precision training and gradient accumulation technology significantly improves the model training speed without losing model accuracy. This makes the training of large-scale multimodal models more efficient, reduces training time and cost, and provides the possibility for rapid iteration and optimization of models.

[0044] 3. This AI-based multimodal model training fine-tuning optimization method effectively prevents overfitting: by reasonably adjusting hyperparameters, adopting early stopping and regularization measures, it effectively prevents model overfitting, which makes the performance of the model on the training set and validation set more stable, improves the generalization ability of the model, and can achieve better results in practical applications; at the same time, the solution can achieve regular evaluation and monitoring: regular model evaluation and monitoring during the training process to timely understand the performance and training progress of the model, which makes it possible to adjust the training strategy and hyperparameters according to the evaluation results to ensure that the model always develops in the optimal direction, improving the controllability and reliability of model training. DETAILED DESCRIPTION

[0045] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0046] The present invention provides a technical solution, specifically, a multimodal model training fine-tuning optimization method based on artificial intelligence, comprising the following specific steps:

[0047] S1 Data preparation and preprocessing: collect large-scale multimodal data sets, build an image-text data set containing 1 million high-quality images and accurately corresponding text descriptions, perform comprehensive and detailed data cleaning operations, strictly remove noise data and incorrectly labeled data, so as to ensure the accuracy and reliability of the data. Through data cleaning, the effective utilization rate of data can be increased by at least 20%;

[0048] The image data is screened to remove blurry, low-resolution, and unclear images to ensure that each image can clearly convey specific visual information. For text descriptions, grammatical errors, spelling errors, and unclear semantics are checked and corrected or deleted.

[0049] Among them, a data quality assessment indicator system was established to quantitatively evaluate data from multiple dimensions, including image clarity, text accuracy, and the relevance of images and texts, to ensure that only high-quality data is used for model training;

[0050] Among them, data enhancement operations are performed, such as random cropping, rotation, and flipping of images, as well as random replacement of some words, deletion of words, and insertion of words in text, which effectively expands the data set to 1.5 times or even more of the original size, thereby significantly improving the generalization ability of the model;

[0051] Image data enhancement: You can set different cropping ratios, rotation angle ranges, and flipping methods to increase data diversity. The random cropping ratio can vary from 10% to 30% of the image side length, the rotation angle can be randomly selected from -30 degrees to 30 degrees, and the flipping methods can include horizontal flipping and vertical flipping.

[0052] Text data enhancement can use synonym replacement, random word deletion, and random word insertion methods to randomly select a word in the text and replace it with its synonym, or randomly delete or insert a word with a certain probability. This can increase the diversity of the text and enable the model to learn richer language expressions.

[0053] The S2 model architecture is optimized and adopts advanced multimodal fusion architecture, such as the multimodal variant of the Transformer architecture. It can efficiently process information in both image and text modes and has extremely powerful representation capabilities. The model parameter scale can reach billions of levels. Such a large parameter space ensures that the model can learn rich multimodal features.

[0054] The multimodal variant of the Transformer architecture usually consists of multiple encoder and decoder layers, where images and text are encoded by different encoders, and then the features of the two modalities are fused through the fusion layer. The fused features are then decoded by the decoder to output the final prediction result;

[0055] To improve the performance of the model, the Transformer architecture can be improved by increasing the number of heads in the multi-head attention mechanism, increasing the number of layers of the encoder and decoder, and using a larger hidden layer dimension. These improvements can improve the representation ability of the model, but will also increase the computational complexity and training time of the model.

[0056] The introduction of the attention mechanism enables the model to focus more accurately on important information between different modalities. In the image-text fusion layer, the use of the self-attention mechanism allows the model to automatically learn the correspondence between images and texts, greatly improving the fusion effect. The computational complexity of the attention mechanism is about 30% of the total computational complexity of the model, but the performance improvement it brings is significant.

[0057] The self-attention mechanism can calculate the correlation between different positions in the image and text, so as to determine which positions are more important for the current task. In the image-text matching task, the self-attention mechanism can help the model find the area in the image that is most relevant to the text description, and the words in the text that are most relevant to the image.

[0058] In addition to the self-attention mechanism, a cross-modal attention mechanism can also be introduced to further enhance the interaction between images and texts. The cross-modal attention mechanism can calculate the bidirectional attention between images and texts, that is, the attention of images to texts and the attention of texts to images, so as to better integrate the information of the two modalities.

[0059] S3 training strategy optimization adopts a phased training strategy. First, pre-training is performed on a large-scale dataset to fully learn the general multimodal features. Then, fine-tuning is performed on a small-scale dataset for a specific task to improve the performance of the model for a specific task. The number of training steps in the pre-training phase is 1 million or more, and the number of training steps in the fine-tuning phase is 100,000 steps.

[0060] In the pre-training stage, unsupervised learning or self-supervised learning methods, contrastive learning, and masked language models can be used. These methods can exploit the inherent structure and correlation in large-scale data to learn common multimodal features, laying a solid foundation for the subsequent fine-tuning stage;

[0061] In the fine-tuning stage, the model is supervised by using the labeled data of the specific task. The structure and parameters of the model can be adjusted according to the characteristics of the task to improve the adaptability and performance of the model to the specific task. In the image classification task, the last layer output of the model can be adjusted to the number of classification categories, and the cross entropy loss function can be used for training.

[0062] Among them, the use of mixed precision training can increase the calculation speed of the model by about 2 times without losing the model accuracy. At the same time, the use of gradient accumulation technology can effectively perform large batch training by accumulating the gradients of multiple small batches when hardware resources are limited, further improving the training efficiency;

[0063] Mixed precision training means using both single-precision floating point numbers (float32) and half-precision floating point numbers (float16) for calculations during model training. Half-precision floating point numbers are faster than single-precision floating point numbers, but have lower precision. By properly arranging the calculation order and data type conversion, the calculation speed of the model can be improved without losing the model accuracy.

[0064] Gradient accumulation technology means accumulating the gradients of multiple small batches and then updating the parameters when hardware resources are limited. This can effectively perform large batch training and improve the convergence speed and performance of the model. When the hardware can only support a batch size of 32, the gradients of 4 batches can be accumulated and then a parameter update can be performed, which is equivalent to training with a batch size of 128.

[0065] S4 hyperparameter adjustment, adjust the learning rate, adopt a learning rate decay strategy, such as cosine annealing decay, set the initial learning rate to 0.001, and gradually reduce the learning rate as the training progresses to improve the convergence speed and stability of the model;

[0066] The learning rate is an important hyperparameter in model training. It determines the step size of model parameter update. If the initial learning rate is set too high, the model may not converge. If it is set too low, the training speed may be too slow. By adopting the learning rate decay strategy, the learning rate can be gradually reduced during the training process, so that the model can maintain a good convergence speed and stability at different training stages.

[0067] Cosine annealing decay is a commonly used learning rate decay strategy, which decays the learning rate in the form of a cosine function. At the beginning of training, the learning rate is large, and as the training progresses, the learning rate gradually decreases and eventually approaches zero. This decay strategy can make the model converge quickly in the early stages of training and perform fine-tuning more stably in the later stages of training.

[0068] Among them, adjust the batch size. According to the hardware resources and data set size, choose an appropriate batch size. Generally speaking, a larger batch size can improve training efficiency, but it may also cause model overfitting. After experiments, it is determined that a batch size of 64 achieves a good balance between performance and efficiency.

[0069] Batch size refers to the number of samples input into the model each time it is trained. A larger batch size can make full use of hardware resources and improve computing efficiency, but it may also cause the model to overfit because the model sees fewer samples in each batch and is prone to falling into a local optimal solution. A smaller batch size can increase the randomness of the model and improve the generalization ability of the model, but it will also reduce training efficiency.

[0070] When choosing a batch size, you need to consider hardware resources, data set size, and model complexity. You can experiment with different batch sizes, observe the performance and training time of the model, and choose a batch size that strikes a good balance between performance and efficiency.

[0071] S5 Evaluation and monitoring: During the training process, the model is evaluated regularly, and the accuracy, recall, and F1 value indicators are used to measure the performance of the model. An evaluation is performed every 1,000 training steps, and the performance of the model on the validation set is recorded.

[0072] Accuracy refers to the ratio of the number of samples predicted correctly by the model to the total number of samples, recall refers to the ratio of the number of positive samples predicted correctly by the model to the number of actual positive samples, and F1 value is the harmonic mean of accuracy and recall. These indicators can measure the performance of the model from different perspectives. In multimodal tasks, it is usually necessary to comprehensively consider multiple indicators to evaluate the performance of the model.

[0073] During the training process, regular model evaluation can help you understand the model's training progress and performance in a timely manner so that you can adjust the training strategy and hyperparameters. The data set can be divided into training set, validation set, and test set. The validation set is used for evaluation during the training process, and the test set is used for the final performance evaluation after the training is completed.

[0074] Monitor the changing trends of the model's training loss and validation loss. If the training loss continues to decrease while the validation loss stops decreasing or even increases, it means that the model may be overfitting. In this case, early stopping or regularization measures can be used to prevent overfitting.

[0075] The training loss refers to the loss function value of the model on the training set, and the validation loss refers to the loss function value of the model on the validation set. The decrease in the training loss indicates that the performance of the model on the training set is constantly improving, but this does not necessarily mean that the performance of the model in actual applications will also improve. If the validation loss no longer decreases or even increases, it means that the model may be overfitting, that is, the model overfits the noise and specific patterns of the data on the training set, resulting in a decrease in performance on the validation set.

[0076] Early stopping is a commonly used method to prevent overfitting. It stops training when the verification loss no longer decreases to prevent the model from overfitting the training data. Regularization is another method to prevent overfitting. It limits the complexity of the model by adding regularization terms to the loss function, thereby improving the generalization ability of the model. Common regularization methods include L1 regularization, L2 regularization, and Dropout.

[0077] Through the above optimization methods, the performance and training efficiency of AI-based multimodal models can be significantly improved, providing stronger support for the application of multimodal tasks;

[0078] Furthermore, this solution improves data quality and diversity: through careful data collection, cleaning and enhancement operations, the accuracy and reliability of the data are ensured, while the diversity of the data is increased, which enables the model to learn richer multimodal features, improves the generalization ability of the model, and can better adapt to different tasks and scenarios. At the same time, this solution optimizes the model architecture: using advanced Transformer architecture multimodal variants, combined with attention mechanisms, improves the model's representation ability and attention to different modal information, which enables the model to more accurately capture the correspondence between images and texts, thereby improving the effect of multimodal fusion and providing stronger support for more complex multimodal tasks;

[0079] Furthermore, the solution improves training efficiency: the application of phased training strategy, mixed precision training and gradient accumulation technology significantly improves the training speed of the model without losing model accuracy, which makes the training of large-scale multimodal models more efficient, reduces training time and cost, and provides the possibility for rapid iteration and optimization of models.

[0080] Furthermore, the solution effectively prevents overfitting: by reasonably adjusting hyperparameters, adopting early stopping method and regularization measures, it effectively prevents model overfitting, which makes the performance of the model on the training set and validation set more stable, improves the generalization ability of the model, and can achieve better results in practical applications; at the same time, the solution can achieve regular evaluation and monitoring: regular model evaluation and monitoring are performed during the training process to timely understand the performance and training progress of the model, which makes it possible to adjust the training strategy and hyperparameters according to the evaluation results to ensure that the model always develops in the optimal direction, thereby improving the controllability and reliability of model training.

[0081] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal model training fine-tuning optimization method based on artificial intelligence, characterized by: The specific steps include: S1 Data preparation and preprocessing: collect large-scale multimodal data sets, build an image-text data set containing 1 million high-quality images and accurately corresponding text descriptions, perform comprehensive and detailed data cleaning operations, strictly remove noise data and incorrectly labeled data, so as to ensure the accuracy and reliability of the data. Through data cleaning, the effective utilization rate of data can be increased by at least 20%; The image data is screened to remove blurry, low-resolution, and unclear images to ensure that each image can clearly convey specific visual information. For text descriptions, grammatical errors, spelling errors, and unclear semantics are checked and corrected or deleted. Among them, a data quality assessment indicator system was established to quantitatively evaluate data from multiple dimensions, including image clarity, text accuracy, and the relevance of images and texts, to ensure that only high-quality data is used for model training; Among them, data enhancement operations are performed, such as random cropping, rotation, and flipping of images, as well as random replacement of some words, deletion of words, and insertion of words in text, which effectively expands the data set to 1.5 times or even more of the original size, thereby significantly improving the generalization ability of the model; The S2 model architecture is optimized and adopts advanced multimodal fusion architecture, such as the multimodal variant of the Transformer architecture. It can efficiently process information in both image and text modes and has extremely powerful representation capabilities. The model parameter scale can reach billions of levels. Such a large parameter space ensures that the model can learn rich multimodal features. The multimodal variant of the Transformer architecture usually consists of multiple encoder and decoder layers, where images and text are encoded by different encoders, and then the features of the two modalities are fused through the fusion layer. The fused features are then decoded by the decoder to output the final prediction result; To improve the performance of the model, the Transformer architecture can be improved by increasing the number of heads in the multi-head attention mechanism, increasing the number of layers of the encoder and decoder, and using a larger hidden layer dimension. These improvements can improve the representation ability of the model, but will also increase the computational complexity and training time of the model. S3 training strategy optimization adopts a phased training strategy. First, pre-training is performed on a large-scale dataset to fully learn the general multimodal features. Then, fine-tuning is performed on a small-scale dataset for a specific task to improve the performance of the model for a specific task. The number of training steps in the pre-training phase is 1 million or more, and the number of training steps in the fine-tuning phase is 100,000 steps. In the pre-training stage, unsupervised learning or self-supervised learning methods, contrastive learning, and masked language models can be used. These methods can exploit the inherent structure and correlation in large-scale data to learn common multimodal features, laying a solid foundation for the subsequent fine-tuning stage; In the fine-tuning stage, the model is supervised by using the labeled data of the specific task. The structure and parameters of the model can be adjusted according to the characteristics of the task to improve the adaptability and performance of the model to the specific task. In the image classification task, the last layer output of the model can be adjusted to the number of classification categories, and the cross entropy loss function can be used for training. S4 hyperparameter adjustment, adjust the learning rate, adopt a learning rate decay strategy, such as cosine annealing decay, set the initial learning rate to 0.001, and gradually reduce the learning rate as the training progresses to improve the convergence speed and stability of the model; The learning rate is an important hyperparameter in model training. It determines the step size of model parameter update. If the initial learning rate is set too high, the model may not converge. If it is set too low, the training speed may be too slow. By adopting the learning rate decay strategy, the learning rate can be gradually reduced during the training process, so that the model can maintain a good convergence speed and stability at different training stages. Cosine annealing decay is a commonly used learning rate decay strategy, which decays the learning rate in the form of a cosine function. At the beginning of training, the learning rate is large, and as the training progresses, the learning rate gradually decreases and eventually approaches zero. This decay strategy can make the model converge quickly in the early stages of training and perform fine-tuning more stably in the later stages of training. S5 Evaluation and monitoring: During the training process, the model is evaluated regularly, and the accuracy, recall, and F1 value indicators are used to measure the performance of the model. An evaluation is performed every 1,000 training steps, and the performance of the model on the validation set is recorded. Accuracy refers to the ratio of the number of samples predicted correctly by the model to the total number of samples, recall refers to the ratio of the number of positive samples predicted correctly by the model to the number of actual positive samples, and F1 value is the harmonic mean of accuracy and recall. These indicators can measure the performance of the model from different perspectives. In multimodal tasks, it is usually necessary to comprehensively consider multiple indicators to evaluate the performance of the model. During the training process, regular model evaluation can help you understand the model's training progress and performance in a timely manner so that you can adjust the training strategy and hyperparameters. The dataset can be divided into training set, validation set, and test set. The validation set can be used for evaluation during the training process, and the test set can be used for the final performance evaluation after the training is completed.

2. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S1, the image data enhancement may be performed by setting different cropping ratios, rotation angle ranges, and flipping modes to increase the diversity of the data. The random cropping ratio may vary between 10% and 30% of the image side length, the rotation angle may be randomly selected between -30 degrees and 30 degrees, and the flipping mode may include horizontal flipping and vertical flipping. Text data enhancement can use synonym replacement, random word deletion, and random word insertion methods. Randomly select a word in the text and replace it with its synonym, or randomly delete or insert a word with a certain probability. This can increase the diversity of the text and enable the model to learn richer language expressions.

3. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S2, the attention mechanism is introduced to enable the model to focus on important information between different modalities more accurately. In the image-text fusion layer, the self-attention mechanism is used to allow the model to automatically learn the correspondence between images and texts, greatly improving the fusion effect. The computational complexity of the attention mechanism is about 30% of the total computational complexity of the model, but the performance improvement it brings is significant. The self-attention mechanism can calculate the correlation between different positions in the image and text, so as to determine which positions are more important for the current task. In the image-text matching task, the self-attention mechanism can help the model find the area in the image that is most relevant to the text description, and the words in the text that are most relevant to the image. In addition to the self-attention mechanism, a cross-modal attention mechanism can also be introduced to further enhance the interaction between images and texts. The cross-modal attention mechanism can calculate the bidirectional attention between images and texts, that is, the attention of images to texts and the attention of texts to images, so as to better integrate the information of the two modalities.

4. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S3, mixed precision training is used to increase the model's calculation speed by about 2 times without losing model accuracy. At the same time, gradient accumulation technology is used. When hardware resources are limited, large batch training can be effectively performed by accumulating gradients of multiple small batches, further improving training efficiency. Mixed precision training means using both single-precision floating-point numbers (float32) and half-precision floating-point numbers (float16) for calculations during model training. Half-precision floating-point numbers are faster than single-precision floating-point numbers, but have lower accuracy. By properly arranging the calculation order and data type conversion, the calculation speed of the model can be improved without losing the model accuracy.

5. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S4, the batch size is adjusted. An appropriate batch size is selected according to the hardware resources and the size of the data set. Generally speaking, a larger batch size can improve the training efficiency, but it may also cause the model to overfit. After experiments, it is determined that a batch size of 64 achieves a good balance between performance and efficiency. Batch size refers to the number of samples input into the model each time it is trained. A larger batch size can make full use of hardware resources and improve computing efficiency, but it may also cause the model to overfit because the model sees fewer samples in each batch and is prone to falling into a local optimal solution. A smaller batch size can increase the randomness of the model and improve the generalization ability of the model, but it will also reduce training efficiency. When choosing a batch size, you need to consider hardware resources, data set size, and model complexity. You can experiment with different batch sizes, observe the performance and training time of the model, and choose a batch size that strikes a good balance between performance and efficiency.

6. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S5, the changing trends of the training loss and validation loss of the model are monitored. If the training loss continues to decrease while the validation loss no longer decreases or even increases, it means that the model may be overfitting. At this time, early stopping or regularization measures can be taken to prevent overfitting.

7. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: The optimization method can significantly improve the performance and training efficiency of AI-based multimodal models, providing stronger support for the application of multimodal tasks.

8. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S5, the training loss refers to the loss function value of the model on the training set, and the validation loss refers to the loss function value of the model on the validation set. The decrease in the training loss indicates that the performance of the model on the training set is constantly improving, but this does not necessarily mean that the performance of the model in practical applications will also improve. If the validation loss no longer decreases or even increases, it means that the model may be overfitted, that is, the model overfits the noise and specific patterns of the data on the training set, resulting in a decrease in performance on the validation set.

9. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S5, the early stopping method is a commonly used method to prevent overfitting. It stops training when the verification loss no longer decreases to prevent the model from continuing to overfit the training data. Regularization is another method to prevent overfitting. It limits the complexity of the model by adding a regularization term to the loss function, thereby improving the generalization ability of the model. Common regularization methods include L1 regularization, L2 regularization and Dropout.

10. The method for fine-tuning and optimizing multimodal model training based on artificial intelligence according to claim 1, characterized in that: In S4, the gradient accumulation technology refers to accumulating the gradients of multiple small batches and then performing a parameter update when hardware resources are limited. This can effectively perform large batch training and improve the convergence speed and performance of the model. When the hardware can only support a batch size of 32, the gradients of 4 batches can be accumulated and then a parameter update can be performed, which is equivalent to training with a batch size of 128.