Dynamic data enhancement method for multi-modal large model pre-training
By dynamically adjusting the multimodal data enhancement method, the problem that traditional static data enhancement methods cannot adapt to the training process is solved, and efficient training and generalization improvement of multimodal large models are achieved, ensuring that the model adapts to changes in real time during the pre-training process .
Patent Information
- Application Number
- CN202411010790.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-19
AI Technical Summary
The data augmentation method used in the pre-training process of traditional multimodal large models is static and cannot be adjusted dynamically, and lacks the comprehensiveness of multimodal data augmentation, resulting in poor model training effect and insufficient adaptability.
Using dynamic data enhancement method, through the similarity calculation and threshold judgment of the verification set, the training data is dynamically adjusted, including rotation, cropping, scaling of the image, reorganizing and rewriting of text, and adjusting the speed and tone of speech to ensure that the model is Adapt to changes in real time during training.
It significantly improves the training effect and performance of the model, improves the generalization and adaptability of the model, reduces the complexity of manual intervention, and avoids overfitting problems.
Smart Images

Figure CN118981640A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer science and artificial intelligence, and in particular to a dynamic data enhancement method for pre-training of a multimodal large model. Background Art
[0002] With the rapid development of artificial intelligence technology, multimodal large models have demonstrated powerful capabilities in image recognition, natural language processing, and speech recognition. In the current pre-training process of multimodal large models, data enhancement technology is one of the important means to improve the performance of multimodal large models. However, most traditional data enhancement methods are static and cannot be dynamically adjusted according to the actual situation during the training process. At the same time, traditional data enhancement methods are often limited to single-modal data processing and lack a comprehensive method for multimodal data enhancement, which makes it impossible to fully utilize the characteristics and advantages of data in different modalities. Summary of the invention
[0003] The present invention mainly solves the technical problem that the training data in the prior art is static data and has insufficient adaptability, and provides a dynamic data enhancement method for pre-training of multimodal large models that can improve the model training effect.
[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a dynamic data enhancement method for multimodal large model pre-training, comprising the following steps:
[0005] S1. Validation set application and similarity calculation: During the pre-training process, after a certain number of training steps (e.g., after each epoch), the current model is applied to the validation set, and the average similarity score between the model's inference results and the standard answer is calculated;
[0006] S2, dynamic adjustment trigger judgment: compare the average similarity score with the threshold. When the average similarity score is greater than or equal to the threshold, the dynamic training data adjustment is triggered and the process goes to step S3. If the average similarity score is less than the threshold, the dynamic training data adjustment is not triggered and the process goes to the next round of training. The setting of the threshold should take into account the initial performance and expected goals of the model and can usually be determined through experiments and experience.
[0007] S3, dynamic training data adjustment: rotating, cropping and / or scaling the image data in the training data, reorganizing and / or rewriting the text data in the training data, and speed changing and / or pitch adjusting the voice data in the training data;
[0008] S4. Reuse the dynamically adjusted training data for model training, and repeat the whole process until the pre-training is completed.
[0009] When the score on the validation set is high, it indicates that the current model has converged on the training set. If learning continues, "overfitting" problems may occur. At this time, adjusting the training set can improve the generalization of the model to a certain extent.
[0010] The training end condition can be based on the specified number of training steps, or a comprehensive consideration of the number of times the dynamic training data adjustment is triggered and the average similarity score. When the condition is met, the training stops.
[0011] This solution completes the enhancement process through a dynamic data enhancement strategy module, which significantly improves the training effect and performance of the model by real-time monitoring and dynamic adjustment of training data. During the pre-training process, the module flexibly performs data enhancement based on the performance of the model on the validation set to ensure the accuracy and generalization of the model.
[0012] Preferably, the step S1 is specifically as follows:
[0013] S101, model reasoning: Input the samples in the validation set into the current model M to obtain the model reasoning results; Assume that the validation set contains N samples, for each sample X i , the inference result given by model M is M(X i );
[0014] S102, calculate the average similarity score: calculate the similarity between the inference result of the model and the standard answer in the validation set, and then calculate the average similarity score avg_sim of all similarities. This score reflects the overall performance of the model on the validation set. The specific formula is:
[0015]
[0016] Where N is the number of samples in the validation set, M(X i ) is the model M for the i-th validation sample X i The inference result, Y i is the standard answer of the i-th verification sample, and sim is the similarity calculation function.
[0017] The similarity calculation method can select an appropriate measurement method according to the specific task. For example, cosine similarity can be used for text data.
[0018] Preferably, the rotation refers to rotating the image by an angle θ (either clockwise or counterclockwise), where θ is determined by the following formula:
[0019] θ=θ base +α·tanh(k1(avg_sim-0.5))
[0020]
[0021] Among them, θ base (θ base ≥0) is the base rotation angle, α(α>0) is the rotation adjustment ratio, which is used to control the amplitude of the angle adjustment, k1 is the rotation gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. The higher the similarity, the greater the adjustment amplitude. The range of avg_sim is [0,1], the range of tanh(k(avg_sim-0.5)) is (-1,1), and the theta function is an increasing function. Rotating the image can increase the diversity of the data and help the model learn to recognize objects at different angles.
[0022] Preferably, the cropping refers to randomly retaining a portion of the image with a cropping ratio p and deleting the rest, and the cropping ratio p is determined by the following formula:
[0023] p=p base -β·tanh(k2(avg_sim-0.5))
[0024]
[0025] Among them, p base (p base ≥0) is the basic cropping ratio, β (β>0) is the cropping adjustment ratio, which is used to control the amplitude of the cropping ratio adjustment, k2 is the cropping gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. If there is no target in the remaining image after the annotated object is cropped, then this image will be used as the background for training. The higher the similarity, the smaller the remaining part. In extreme cases, the similarity is 1, and p is close to 0, so it is not retained. Cropping images can simulate different scenes and perspectives and enhance the robustness of the model.
[0026] Preferably, the scaling refers to scaling the image at a scaling ratio s, and the scaling ratio s is determined by the following formula:
[0027] s=s base +γ·tanh(k3(avg_sim-0.5))
[0028]
[0029] Among them, s base (s base≥0) is the basic scaling ratio, γ is the scaling adjustment ratio, which is used to control the amplitude of the scaling adjustment, k3 is the scaling gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. The scaling direction is related to the similarity and the positive or negative value of γ. γ is greater than 0 for positive correlation and γ is less than 0 for negative correlation, but both positive and negative correlations will be magnified and reduced. At the same time, the maximum method ratio is set to 2, because infinite magnification is meaningless. Scaling images can cope with changes in different resolutions and distances, and improve the model's adaptability to various image inputs.
[0030] Preferably, the reorganization refers to randomly selecting a part of the sentences in the text and reorganizing them using a large language model to generate a new sentence structure to make the text data more diverse. The proportion of reorganized sentences r is determined by the following formula:
[0031] r=r base +δ·tanh(k4(avg_sim-0.5))
[0032]
[0033] Among them, r base (r base ≥0) is the basic recombination ratio, δ(δ>0) is the recombination adjustment ratio, which is used to control the amplitude of the recombination ratio adjustment, k4 is the recombination gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. Reorganization mainly refers to adjusting the sentence structure of the original text data, such as the order of precedence.
[0034] Preferably, the rewriting refers to randomly selecting a part of the text for synonym replacement and sentence adjustment, or using a large language model for rewriting to generate diversified text expressions to help the model understand and process different language expressions. The proportion c of rewritten sentences is determined by the following formula:
[0035] c=c base +ε·tanh(k5(avg_sim-0.5))
[0036]
[0037] Among them, c base (c base ≥0) is the basic rewriting ratio, ε(ε>0) is the rewriting adjustment ratio, which is used to control the amplitude of the rewriting ratio adjustment, k5 is the rewriting gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. Rewriting means rewriting the original training data and changing the expression while maintaining the original semantics.
[0038] Preferably, the speed change refers to adjusting the speech playback speed according to the speed change amplitude v to simulate speech inputs of different speech speeds and improve the adaptability of the model to speech speed changes. The speed change amplitude v is determined by the following formula:
[0039] v=v base +ζ·tanh(k6(avg_sim-0.5))
[0040]
[0041] Among them, v base (v base ≥0) is the basic speed ratio, ζ is the speed adjustment ratio, which is used to control the amplitude of the speed adjustment, k6 is the speed gradient coefficient, which is used to control the gradient of the adjustment function and is usually set to 10, and avg_sim is the average similarity score of the model on the validation set.
[0042] Preferably, the pitch adjustment refers to changing the voice pitch according to the pitch amplitude f to simulate different speakers and emotional states and enhance the model's ability to recognize and understand voice data. The pitch amplitude f is determined by the following formula:
[0043] f=f base +η·tanh(k7(avg_sim-0.5))
[0044]
[0045] Among them, f base (f base ≥0) is the basic pitch ratio, η is the pitch adjustment ratio, which is used to control the amplitude of pitch adjustment, k7 is the pitch gradient coefficient, which is used to control the gradient of the adjustment function and is usually set to 10, and avg_sim is the average similarity score of the model on the validation set.
[0046] Preferably, the validation set data is obtained by:
[0047] A1. Data collection;
[0048] A2. Data annotation;
[0049] A3. Multimodal data fusion: Semantically align the collected data of each modality, and then construct question-answer pairs for each modality of data. The questions are related to the content of the modality data or other modality data, and the answers are the corresponding annotated content or generated based on the annotated content.
[0050] This process is mainly completed by the data set collection and annotation module. The main function of this module is to systematically collect and annotate the validation set according to the pre-training goals to evaluate the performance of the multimodal large model during the training process, thereby providing a basis for subsequent dynamic data adjustments. The specific steps of this module are as follows:
[0051] (1) Goal-oriented data collection:
[0052] The dataset collection and annotation module first determines the type and scope of data to be collected based on the specific goals of the training data. For different application scenarios, this module can flexibly collect multimodal data such as images, text, and speech. For example, for image recognition tasks, this modality collects image data containing various objects, scenes, and labels; for natural language processing tasks, it collects text data covering a variety of contexts and expressions.
[0053] (2) High-quality data annotation:
[0054] The collected data needs to be annotated and verified to ensure the quality and accuracy of the validation set. For image data, annotation includes objects, bounding boxes, scene descriptions, etc. For text data, annotation involves semantic analysis, sentiment analysis, etc. The annotation process can be manual or using existing large models.
[0055] (3) Fusion of multimodal data
[0056] In order to better evaluate the performance of a large multimodal model, the validation set needs to cover data combinations of different modalities. This module converts the above annotations into corresponding multimodal question-answering data. The specific process is as follows:
[0057] 1. Data alignment: semantically align the collected data of each modality. For example, for an image, there is a corresponding text description or voice interpretation.
[0058] 2. Question-answer pairing: For each modality of data, construct a question-answer pairing. The question can be about the content of the modality data or a question related to other modal data. The answer is the corresponding annotation content or the answer generated based on the annotation content.
[0059] The substantial effect of the present invention is that the model can continuously adapt to new data features and task requirements during the training process, thereby significantly improving the pre-training effect of the model. In addition, the method of the present invention is simple to operate and easy to implement, reducing the complexity of manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is a flow chart of a dynamic data enhancement method of the present invention. DETAILED DESCRIPTION
[0061] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.
[0062] Embodiment: A dynamic data enhancement method for multimodal large model pre-training in this embodiment is as follows: Figure 1 As shown, the following steps are included:
[0063] S1. Validation set application and similarity calculation: During the pre-training process, after a certain number of training steps (e.g., after each epoch), the current model is applied to the validation set, and the average similarity score between the model's inference results and the standard answer is calculated;
[0064] S2, dynamic adjustment trigger judgment: compare the average similarity score with the threshold. When the average similarity score is greater than or equal to the threshold, the dynamic training data adjustment is triggered and the process goes to step S3. If the average similarity score is less than the threshold, the dynamic training data adjustment is not triggered and the process goes to the next round of training. The setting of the threshold should take into account the initial performance and expected goals of the model. It can usually be determined through experiments and experience. In this embodiment, it is set to 0.35.
[0065] S3, dynamic training data adjustment: rotating, cropping and / or scaling the image data in the training data, reorganizing and / or rewriting the text data in the training data, and speed changing and / or pitch adjusting the voice data in the training data;
[0066] S4. Reuse the dynamically adjusted training data for model training, and repeat the whole process until the pre-training is completed.
[0067] When the score on the validation set is high, it indicates that the current model has converged on the training set. If learning continues, "overfitting" problems may occur. At this time, adjusting the training set can improve the generalization of the model to a certain extent.
[0068] The training end condition can be based on the specified number of training steps, or a comprehensive consideration of the number of times the dynamic training data adjustment is triggered and the average similarity score. When the condition is met, the training stops.
[0069] This solution completes the enhancement process through a dynamic data enhancement strategy module, which significantly improves the training effect and performance of the model by real-time monitoring and dynamic adjustment of training data. During the pre-training process, the module flexibly performs data enhancement based on the performance of the model on the validation set to ensure the accuracy and generalization of the model.
[0070] The step S1 is specifically as follows:
[0071] S101, model reasoning: Input the samples in the validation set into the current model M to obtain the model reasoning results; Assume that the validation set contains N samples, for each sample X i , the inference result given by model M is M(X i );
[0072] S102, calculate the average similarity score: calculate the similarity between the inference result of the model and the standard answer in the validation set, and then calculate the average similarity score avg_sim of all similarities. This score reflects the overall performance of the model on the validation set. The specific formula is:
[0073]
[0074] Where N is the number of samples in the validation set, M(X i ) is the model M for the i-th validation sample X i The inference result, Y i is the standard answer of the i-th verification sample, and sim is the similarity calculation function.
[0075] The similarity calculation method can select an appropriate measurement method according to the specific task. For example, cosine similarity can be used for text data.
[0076] The rotation refers to rotating the image by an angle of θ (either clockwise or counterclockwise), where θ is determined by the following formula:
[0077] θ=θ base +α·tanh(k1(avg_sim-0.5))
[0078]
[0079] Among them, θ base (θ base ≥0) is the base rotation angle, α(α>0) is the rotation adjustment ratio, which is used to control the amplitude of the angle adjustment, k1 is the rotation gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. The higher the similarity, the greater the adjustment amplitude. The range of avg_sim is [0,1], the range of tanh(k1(avg_sim-0.5)) is (-1,1), and the theta function is an increasing function. Rotating the image can increase the diversity of the data and help the model learn to recognize objects at different angles.
[0080] The cropping refers to randomly retaining a portion of the image with a cropping ratio p and deleting the rest. The cropping ratio p is determined by the following formula:
[0081] p=p base-β·tanh(k2(avg_sim-0.5))
[0082]
[0083] Among them, p base (p base ≥0) is the basic cropping ratio, β (β>0) is the cropping adjustment ratio, which is used to control the amplitude of the cropping ratio adjustment, k2 is the cropping gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. If there is no target in the remaining image after the annotated object is cropped, then this image will be used as the background for training. The higher the similarity, the smaller the remaining part. In extreme cases, the similarity is 1, and p is close to 0, so it is not retained. Cropping images can simulate different scenes and perspectives and enhance the robustness of the model.
[0084] The scaling refers to scaling the image according to the scaling ratio s, and the scaling ratio s is determined by the following formula:
[0085] s=s base +γ·tanh(k3(avg_sim-0.5))
[0086]
[0087] Among them, s base (s base ≥0) is the basic scaling ratio, γ is the scaling adjustment ratio, which is used to control the amplitude of the scaling adjustment, k3 is the scaling gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. The scaling direction is related to the similarity and the positive or negative value of γ. γ is greater than 0 for positive correlation and γ is less than 0 for negative correlation, but both positive and negative correlations will be magnified and reduced. At the same time, the maximum method ratio is set to 2, because infinite magnification is meaningless. Scaling images can cope with changes in different resolutions and distances, and improve the model's adaptability to various image inputs.
[0088] The reorganization refers to randomly selecting a part of the sentences in the text and reorganizing them using a large language model to generate a new sentence structure to make the text data more diverse. The proportion of reorganized sentences r is determined by the following formula:
[0089] r=r base +δ·tanh(k4(avg_sim-0.5))
[0090]
[0091] Among them, r base (r base≥0) is the basic recombination ratio, δ(δ>0) is the recombination adjustment ratio, which is used to control the amplitude of the recombination ratio adjustment, k4 is the recombination gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. Reorganization mainly refers to adjusting the sentence structure of the original text data, such as the order of precedence.
[0092] The rewriting refers to randomly selecting a part of the text for synonym replacement and sentence adjustment, or using a large language model for rewriting to generate diversified text expressions to help the model understand and process different language expressions. The proportion of rewritten sentences c is determined by the following formula:
[0093] c=c base +ε·tanh(k5(avg_sim-0.5))
[0094]
[0095] Among them, c base (c base ≥0) is the basic rewriting ratio, ε(ε>0) is the rewriting adjustment ratio, which is used to control the amplitude of the rewriting ratio adjustment, k5 is the rewriting gradient coefficient, which is used to control the gradient of the adjustment function, usually set to 10, and avg_sim is the average similarity score of the model on the validation set. Rewriting means rewriting the original training data and changing the expression while maintaining the original semantics.
[0096] The speed change refers to adjusting the speech playback speed according to the speed change amplitude v to simulate speech inputs of different speech speeds and improve the adaptability of the model to speech speed changes. The speed change amplitude v is determined by the following formula:
[0097] v=v base +ζ·tanh(k6(avg_sim-0.5))
[0098]
[0099] Among them, v base (v base ≥0) is the basic speed ratio, ζ is the speed adjustment ratio, which is used to control the amplitude of the speed adjustment, k6 is the speed gradient coefficient, which is used to control the gradient of the adjustment function and is usually set to 10, and avg_sim is the average similarity score of the model on the validation set.
[0100] The pitch adjustment refers to changing the voice pitch according to the pitch amplitude f to simulate different speakers and emotional states and enhance the model's ability to recognize and understand voice data. The pitch amplitude f is determined by the following formula:
[0101] f=f base+η·tanh(k7(avg_sim-0.5))
[0102]
[0103] Among them, f base (f base ≥0) is the basic pitch ratio, η is the pitch adjustment ratio, which is used to control the amplitude of pitch adjustment, k7 is the pitch gradient coefficient, which is used to control the gradient of the adjustment function and is usually set to 10, and avg_sim is the average similarity score of the model on the validation set.
[0104] The validation set data is obtained in the following way:
[0105] A1. Data collection;
[0106] A2. Data annotation;
[0107] A3. Multimodal data fusion: Semantically align the collected data of each modality, and then construct question-answer pairs for each modality of data. The questions are related to the content of the modality data or other modality data, and the answers are the corresponding annotated content or generated based on the annotated content.
[0108] This process is mainly completed by the data set collection and annotation module. The main function of this module is to systematically collect and annotate the validation set according to the pre-training goals to evaluate the performance of the multimodal large model during the training process, thereby providing a basis for subsequent dynamic data adjustments. The specific steps of this module are as follows:
[0109] (1) Goal-oriented data collection:
[0110] The dataset collection and annotation module first determines the type and scope of data to be collected based on the specific goals of the training data. For different application scenarios, this module can flexibly collect multimodal data such as images, text, and speech. For example, for image recognition tasks, this modality collects image data containing various objects, scenes, and labels; for natural language processing tasks, it collects text data covering a variety of contexts and expressions.
[0111] (2) High-quality data annotation:
[0112] The collected data needs to be annotated and verified to ensure the quality and accuracy of the validation set. For image data, annotation includes objects, bounding boxes, scene descriptions, etc. For text data, annotation involves semantic analysis, sentiment analysis, etc. The annotation process can be manual or using existing large models.
[0113] (3) Fusion of multimodal data
[0114] In order to better evaluate the performance of a large multimodal model, the validation set needs to cover data combinations of different modalities. This module converts the above annotations into corresponding multimodal question-answering data. The specific process is as follows:
[0115] 1. Data alignment: semantically align the collected data of each modality. For example, for an image, there is a corresponding text description or voice interpretation.
[0116] 2. Question-answer pairing: For each modality of data, construct a question-answer pairing. The question can be about the content of the modality data or a question related to other modal data. The answer is the corresponding annotation content or the answer generated based on the annotation content.
[0117] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.
[0118] Although the terms such as average similarity, rotation, and rewriting are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.
Claims
1. A dynamic data enhancement method for multimodal large model pre-training, characterized in that: The following steps are involved: S1. Validation set application and similarity calculation: During the pre-training process, after a certain number of training steps, the current model is applied to the validation set, and the average similarity score between the model's reasoning results and the standard answer is calculated; S2, dynamic adjustment trigger judgment: compare the average similarity score with the threshold. When the average similarity score is greater than or equal to the threshold, dynamic training data adjustment is triggered and the process goes to step S3. If the average similarity score is less than the threshold, dynamic training data adjustment is not triggered and the process goes to the next round of training. S3, dynamic training data adjustment: rotating, cropping and / or scaling the image data in the training data, reorganizing and / or rewriting the text data in the training data, and speed changing and / or pitch adjusting the voice data in the training data; S4. Reuse the dynamically adjusted training data for model training, and repeat the whole process until the pre-training is completed.
2. A dynamic data enhancement method for multimodal large model pre-training according to claim 1, characterized in that: The step S1 is specifically as follows: S101, model reasoning: input the samples in the validation set into the current model M to obtain the reasoning results of the model; S102, calculate the average similarity score: calculate the similarity between the inference result of the model and the standard answer in the validation set, and then calculate the average similarity score avg_sim of all similarities. The specific formula is: Where N is the number of samples in the validation set, M(X i ) is the model M for the i-th validation sample X i The inference result, Y i is the standard answer of the i-th verification sample, and sim is the similarity calculation function.
3. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The rotation refers to rotating the image by an angle θ, where θ is determined by the following formula: θ=θbase+α·tanh(k1(avg_sim-0.5)) Among them, θ base is the base rotation angle, α is the rotation adjustment scale factor, k1 is the rotation gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
4. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The cropping refers to randomly retaining a portion of the image with a cropping ratio p and deleting the rest. The cropping ratio p is determined by the following formula: p=pbase-β·tanh(k2(avg_sim-0.5)) Among them, p base is the base cropping ratio, β is the cropping adjustment ratio, k2 is the cropping gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
5. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The scaling refers to scaling the image according to the scaling ratio s, and the scaling ratio s is determined by the following formula: s=sbase+γ·tanh(k3(avg_sim-0.5)) Among them, s base is the base scaling ratio, γ is the scaling adjustment ratio, k3 is the scaling gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
6. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The reorganization refers to randomly selecting a part of the sentences in the text and reorganizing them using a large language model to generate a new sentence structure. The proportion of reorganized sentences r is determined by the following formula: r=rbase+δ·tanh(k4(avg_sim-0.5)) Among them, r base is the basic recombination ratio, δ is the recombination adjustment ratio, k4 is the recombination gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
7. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The rewriting refers to randomly selecting a part of the text to replace synonyms and adjust the sentence structure, or using a large language model to rewrite. The proportion c of rewritten sentences is determined by the following formula: c=cbase+ε·tanh(k5(avg_sim-0.5)) Among them, c base is the base rewrite ratio, ε is the rewrite adjustment ratio, k5 is the rewrite gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
8. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The speed change refers to adjusting the voice playback speed according to the speed change amplitude v, and the speed change amplitude v is determined by the following formula: v=vbase+ζ·tanh(k6(avg_sim-0.5)) Among them, v base is the basic speed ratio, ζ is the speed adjustment ratio, k6 is the speed gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
9. A dynamic data enhancement method for multimodal large model pre-training according to claim 1 or 2, characterized in that: The pitch adjustment refers to changing the voice pitch according to the pitch amplitude f, and the pitch amplitude f is determined by the following formula: f=fbase+η·tanh(k7(avg_sim-0.5)) Among them, f base is the basic pronunciation ratio, η is the pronunciation adjustment ratio, k7 is the pronunciation gradient coefficient, and avg_sim is the average similarity score of the model on the validation set.
10. The dynamic data enhancement method for multimodal large model pre-training according to claim 1, characterized in that: The validation set data is obtained in the following way: A1. Data collection; A2. Data annotation; A3. Multimodal data fusion: Semantically align the collected data of each modality, and then construct question-answer pairs for each modality of data. The questions are related to the content of the modality data or other modality data, and the answers are the corresponding annotated content or generated based on the annotated content.
