Model self-lifting method, system and equipment based on data weight balance and medium
By identifying and adjusting the head-to-tail data imbalance in the self-improvement iteration of the visual reasoning model, and using a rebalancing strategy to optimize the distribution of training data, the problem of data imbalance in traditional methods is solved, thereby improving the model's reasoning ability and stability.
Patent Information
- Application Number
- CN202511539210.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-13
AI Technical Summary
In traditional visual reasoning model self-improvement methods, the imbalance between the beginning and end of the training data makes it difficult for the model to achieve efficient and stable improvement in visual reasoning ability. This manifests as imbalance between the beginning and end of the data, imbalance in the distribution of difficulty levels, and imbalance in response length.
During the self-improvement iteration process, by identifying the imbalance between the head and tail data, rebalancing strategies such as threshold pruning, repetition filling, adaptive weighted sampling, and guided resampling are adopted to adjust the data, ensuring that the training data is more evenly distributed on the inference trajectory corresponding to the head and tail queries.
It significantly improves the model's reasoning ability and generalization, ensuring that the model can both consolidate its ability to process high-accuracy head data and fully learn from low-accuracy tail data, thereby improving the efficiency and stability of the model's self-improvement.
Smart Images
Figure CN121525841A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of visual language model self-improvement, and particularly relates to a model self-improvement method and system based on data rebalancing, equipment and a medium. BACKGROUND
[0002] In the field of artificial intelligence, visual reasoning tasks aim to enable models to solve complex problems by combining image information and logical reasoning capabilities. The self-improvement method enables large visual language models to achieve performance improvement through an iterative exploration and learning process, without relying on large-scale high-quality labeled data sets, while ensuring data distribution space alignment and application in various visual reasoning scenarios.
[0003] Traditional visual reasoning model self-improvement methods mainly follow a three-stage iterative process of "Exploration-Filtering-Learning". The specific process is as follows: first, in the exploration stage, the model performs K times of trajectory sampling on the input query to generate multiple possible reasoning trajectories; second, in the filtering stage, a pre-set reward model or reward function is used to determine the correctness of the sampled trajectories, and the correct trajectories are selected and retained, while the incorrect trajectories are removed; finally, in the learning stage, the model updates the parameters using the filtered correct trajectory data as training samples, and gradually optimizes the reasoning ability through repeated iterations of the above three stages.
[0004] However, the traditional self-improvement method does not set any data distribution adjustment module between the "filtering" and "learning" stages, and the filtered correct trajectory data is directly used for model training. As the training data is skewed towards simple data at the head, the model's learning of difficult data at the tail is insufficient. These defects directly lead to the difficulty of achieving efficient and stable visual reasoning capability improvement for the model, which is manifested in head-tail data imbalance, difficulty level distribution imbalance, and response length imbalance. SUMMARY
[0005] The purpose of the present application is to provide a model self-improvement method and system based on data rebalancing, which alleviates the data imbalance phenomenon of visual reasoning models in self-improvement iteration, improves data quality and utilization efficiency, and optimizes the self-improvement effect of the model.
[0006] To achieve the above purpose, the present application provides a model self-improvement method based on data rebalancing, comprising: in each iteration, using the current model to sample a batch of queries multiple times to generate multiple reasoning trajectory data for each query; selecting correct trajectories from the multiple reasoning trajectory data to obtain the number of correct trajectories for each query; identify a head-tail imbalance phenomenon based on the number of correct trajectories of each query; when the head-tail imbalance phenomenon is identified, adopt a rebalancing strategy to adjust the data to obtain rebalanced data; use the rebalanced data as training data to perform self-improvement training on the model.
[0007] Optionally, the identifying of the head-tail imbalance phenomenon based on the number of correct trajectories of each query comprises: identifying each piece of data with a number of correct trajectories within a first preset range as head data, and identifying each piece of data with a number of correct trajectories within a second preset range as tail data. When the proportion of the head data in the iteration process continues to increase and the proportion of the tail data in the iteration process continues to decrease, the head-tail imbalance phenomenon is identified.
[0008] Optionally, the rebalancing strategy comprises at least one of a distribution readjustment strategy and a trajectory resampling strategy. The distribution readjustment strategy comprises at least one of a threshold clipping strategy and a repeated filling strategy; and the trajectory resampling strategy comprises at least one of an adaptive weighted resampling strategy and a guided resampling strategy.
[0009] Optionally, the threshold clipping strategy comprises: counting the number k of correct trajectories of each query in K samplings; for a query with a number k of correct trajectories greater than a preset number L of correct trajectories, randomly retaining L correct trajectories and deleting the remaining k-L correct trajectories; wherein k is less than or equal to K, and K, k and L are positive integers greater than zero; The repeated filling strategy comprises: counting the number k of correct trajectories of each query in K samplings; for a query with a number k of correct trajectories less than K, generating the same correct trajectories as the existing correct trajectories until the number k of correct trajectories is equal to K.
[0010] Optionally, the adaptive weighted resampling strategy comprises: counting the number k of correct trajectories of each query in K samplings, performing K-k times of resampling using the model to obtain correct trajectory data obtained after resampling; wherein k is less than or equal to K, and K and k are positive integers greater than zero; The guided resampling strategy comprises: decomposing the correct trajectories of the tail data into multiple steps according to the lengths of the correct trajectories; performing resampling from the middle step of each correct trajectory using the model to obtain correct trajectory data obtained after resampling.
[0011] Optionally, before the self-improvement training of the model using the rebalanced data as training data, further comprising: using the model to correct errors in the error trajectory, regenerate a correct trajectory, and obtain self-correction data as part of the training data.
[0012] Optionally, after the self-improvement training of the model using the rebalanced data as training data, further comprising: calculating the performance difference of the model under the conditions of image input and no image input for the same difficult problem to evaluate the improvement of the real visual reasoning ability of the model.
[0013] To achieve the above object, the present application further provides a model self-improvement system based on data rebalancing, comprising: an exploration module for generating multiple reasoning trajectory data for each query by using a current model to sample a batch of queries multiple times in each iteration; a filtering module for filtering correct trajectories from the multiple reasoning trajectory data to obtain the number of correct trajectories for each query; a rebalancing module for identifying head-tail imbalance based on the number of correct trajectories for each query; when identifying head-tail imbalance, using a rebalancing strategy to adjust the data to obtain rebalanced data; a training module for self-improvement training of the model using the rebalanced data as training data.
[0014] To achieve the above object, the present application further provides a terminal device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the model self-improvement method based on data rebalancing according to any one of the above.
[0015] To achieve the above object, the present application further provides a computer readable storage medium comprising a stored computer program, wherein when the computer program is running, the device where the computer readable storage medium is located executes the model self-improvement method based on data rebalancing according to any one of the above.
[0016] Compared with existing technologies, the model self-improvement method based on data rebalancing provided by this invention can accurately identify and effectively alleviate the imbalance between the head and tail data in the model self-improvement iteration. It adjusts the selected correct trajectory data through four rebalancing strategies: threshold pruning, repetition padding, adaptive weighted sampling, and guided resampling. This makes the training data more evenly distributed on the inference trajectories corresponding to the head and tail queries, ensuring that the model can consolidate its ability to process high-accuracy head data and fully learn from low-accuracy tail data. This significantly improves the comprehensiveness and generalization of the model's inference ability, and enhances the efficiency and stability of the model self-improvement. Attached Figure Description
[0017] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a model self-improvement method based on data rebalancing provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of a model self-improvement system based on data rebalancing provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] See Figure 1 , Figure 1 This is a flowchart of a model self-improvement method based on data rebalancing provided by an embodiment of the present invention. The model self-improvement method based on data rebalancing includes steps S1 to S4: S1. In each iteration, the current model is used to sample a batch of queries multiple times to generate multiple inference trajectory data for each query. S2. Filter out the correct trajectories from the multiple inference trajectory data to obtain the number of correct trajectories for each query; S3. Based on the number of correct trajectories for each query, identify head-to-tail imbalance phenomena; when head-to-tail imbalance phenomena are identified, a rebalancing strategy is used to adjust the data to obtain rebalanced data. It should be noted that the model used in this embodiment of the invention is a visual reasoning model. Traditional self-improvement methods for visual reasoning models mainly follow a three-stage iterative process of "exploration-filtering-learning". Specifically, the model first completes trajectory exploration through several self-sampling iterations, then filters the sampled trajectories based on feedback from the reward model or reward function, retaining only the correct trajectories, and finally uses the filtered data for training to achieve the learning stage. By repeatedly iterating the above process, model performance improvement can be achieved without relying on external data. However, the inventors observed that self-improvement faces the phenomenon of head-to-tail data imbalance during the iteration process. Therefore, this embodiment of the invention proposes inserting a head-to-tail data re-balancing step, i.e., step S3, between the filtering and learning stages of traditional self-improvement.
[0021] S4. Use the rebalanced data as training data to perform self-improvement training on the model.
[0022] It is worth noting that the embodiments of the present invention observed the phenomenon of uneven data distribution during the self-improvement iteration process in the visual reasoning scenario, namely the head-tail imbalance phenomenon identified in step S3, and defined it as the "Matthew effect".
[0023] It should be noted that this invention borrows the "Matthew effect," a social phenomenon proposed by sociologist Robert K. Merton, to represent the imbalance in the distribution of head and tail data during the self-improvement process. That is, the number of easy samples (head data) dominating the correct trajectory gradually increases, while the proportion of difficult samples (tail data), which are already scarce, decreases.
[0024] In one optional embodiment, identifying head-to-tail imbalance based on the number of correct trajectories for each query includes: The query for the number of correct trajectories within a first preset range is identified as head data, and the query for the number of correct trajectories within a second preset range is identified as tail data. When the proportion of head data in the iteration process continues to increase while the proportion of tail data in the iteration process continues to decrease, it is identified as a head-tail imbalance phenomenon.
[0025] Through the above steps, this embodiment of the invention establishes a Matthew effect recognition mechanism, using the number of correct trajectories in self-generated data as an indicator of data distribution. For example, in each iteration, for each piece of data (i.e., each query), the self-improvement method samples K trajectories. This embodiment of the invention counts the number of correct trajectories for each piece of data and uses this as the basis for determining the beginning and end of the data.
[0026] Preferably, the dominant data with the number of correct trajectories between 3 / 4K and K is defined as head data, while the scarce data with the number of correct trajectories between 0 and 1 / 4K is defined as tail data. When the proportion of high-accuracy head data continues to increase while the proportion of low-accuracy tail data continues to decrease, it is identified as a "Matthew effect," i.e., an imbalance between the head and tail.
[0027] In one alternative embodiment, the changes in data distribution can also be observed based on the difficulty level and response length of each query to identify head-to-tail imbalance.
[0028] Specifically, from the perspective of difficulty level, while ensuring the even distribution of each difficulty level as much as possible, the data was pre-divided into 5 difficulty levels, and the trend of the proportion of data at each difficulty level during the iteration process was monitored. For example, based on the performance of the Qwen2-VL-7B-Instruct model on the MMPR dataset, when the lowest difficulty level (easy data) accounted for more than half of the total, while the highest difficulty level (difficult data) only accounted for a preset percentage (e.g., 1.5%), it was determined that a head-to-tail imbalance had occurred.
[0029] Specifically, regarding response length, the inventors first observed that the model-generated responses were generally shorter than manually labeled data, and this phenomenon intensified with iteration. Furthermore, the inventors compared the response length decay of data at different difficulty levels, finding that the decay was not significant for low-difficulty data, while a substantial decay occurred for high-difficulty data. Therefore, when a slight decay in response length was observed for low-difficulty data, while a significant decay occurred for high-difficulty data, it was determined that a head-to-tail imbalance had occurred.
[0030] In one optional embodiment, the rebalancing strategy includes at least one of a distribution readjustment strategy and a trajectory resampling strategy; The distribution readjustment strategy includes at least one of threshold clipping (TC) and repeat-based padding (RP); the trajectory resampling strategy includes at least one of adaptive-weighted resampling (AR) and guided resampling (GR).
[0031] It is worth noting that the embodiments of the present invention propose four effective rebalancing strategies from two perspectives: distribution-reshaping and trajectory-reampling, so as to select the appropriate rebalancing strategy for data adjustment according to actual needs.
[0032] In one optional embodiment, the threshold pruning strategy includes: Count the number of correct trajectories, k, for each query across K samples; For queries where the number of correct trajectories k is greater than the preset threshold L, L correct trajectories are randomly retained, and the remaining kL correct trajectories are deleted; where k is less than or equal to K, and K, k, and L are all positive integers greater than zero.
[0033] Preferably, when K=16, setting L=4 can achieve the best balance effect.
[0034] For example, in a query with 16 samples, 2 trajectories are correct and 14 are incorrect. Since there are only 2 correct trajectories in this query, which is less than 4, both correct trajectories will be retained. If there are 6 correct trajectories and 10 incorrect trajectories in 16 samples, since there are more than 4 correct trajectories in this query, both random correct trajectories will be deleted.
[0035] It is worth noting that the threshold pruning strategy in this embodiment of the invention sets an upper limit L for the number of correct trajectories and randomly truncates correct trajectories exceeding the threshold, thereby reducing the absolute number of simple head data. This generates an adjusted training dataset, thus keeping the difficult tail data unchanged while reducing the head data. This method is simple to implement, indirectly increasing the proportion of difficult tail data by reducing the number of simple head data, achieving an improvement effect through deletion, and requires less training data with high training efficiency.
[0036] In one optional embodiment, the repeated filling strategy includes: Count the number of correct trajectories, k, for each query across K samples; For queries where the number of correct trajectories k is less than K, generate correct trajectories that are identical to existing correct trajectories until the number of correct trajectories k equals K; where k is less than or equal to K, and K and k are both positive integers greater than zero.
[0037] For example, in a query with 16 samples, there are 2 correct trajectories and 14 incorrect trajectories. This query is considered to have fewer correct trajectories than the target number of correct trajectories (16 occurrences). Therefore, the number of correct trajectories is increased to 16. Thus, the 2 correct trajectories from this query will each be repeated 8 times and added to the new training dataset, where each query has K correct samples.
[0038] It is worth noting that the duplication strategy ensures that all queries appear with the same frequency K in the training data. By supplementing queries with insufficient correct samples through the duplication mechanism, the proportion of difficult tail data can be significantly increased in a simple way, while maintaining the original quality of the data.
[0039] It should be noted that while the aforementioned distribution resizing strategy is simple and easy to implement, methods such as deletion or duplication may affect data diversity or lead to overfitting. Therefore, this invention proposes two efficient trajectory resampling techniques: an adaptive weighted sampling strategy and a guided resampling strategy, to achieve stable performance improvements with minimal computational overhead.
[0040] In one optional embodiment, the adaptive weighted sampling strategy includes: Count the number of correct trajectories k for each query in K samplings, and use the model to perform Kk resampling to obtain the correct trajectory data after resampling; where k is less than or equal to K, and K and k are both positive integers greater than zero.
[0041] For example, if 2 trajectories are correct and 14 trajectories are incorrect in 16 samples in a query, then the query needs to be resampled 16-2=14 times. The resampled dataset is then filtered to obtain the correct trajectory data after resampling. The initially filtered dataset and the resampled filtered dataset are then merged as the training dataset for the learning phase.
[0042] It's worth noting that the adaptive weighted sampling strategy can dynamically adjust the number of resampling attempts for each query based on the sampling distribution, achieving difficulty-oriented hierarchical resampling. Since the overall accuracy of the model on the training dataset is typically around 50%, and the number of resampling attempts in the adaptive weighted sampling strategy is the same as the number of sampling errors (Kk times), the adaptive weighted sampling strategy only requires about 50% of the resampling overhead on average, significantly improving resampling efficiency compared to brute-force sampling.
[0043] In one alternative embodiment, the guided resampling strategy includes: Decompose the correct trajectory of the tail data into multiple steps according to the length of the correct trajectory; The model is used to resample from the intermediate step of each correct trajectory to obtain the correct trajectory data after resampling.
[0044] For example, in a query with 16 samples, 2 trajectories are correct and 14 are incorrect. According to the above definition, this query is tail data. First, the correct trajectory of the tail data is decomposed into steps. For the correct trajectory r, it is decomposed into a sequence of steps r(1), r(2), ..., r(S) on average according to the trajectory length (with words as the statistical unit). Then, the model is made to start from each intermediate step s and explore to generate subsequent responses. That is, the input is q, r(1), r(2), ..., r(s-1), and the model only needs to start exploring from the s-th step. Then, the obtained resampled dataset is filtered to retain the resampled trajectory with the correct final result. Finally, the original filtered dataset and the guided resampled filtered dataset are merged as training data.
[0045] For example, for two correct trajectory data a and b, they are divided into multiple steps a1, a2, a3, a4 and b1, b2, b3, b4 according to their length, respectively. Then, they are resampled, that is, the input prompts are a1, a1+a2, a1+a2+a3 and b1, b1+b2, b1+b2+b3 respectively, so that the model can generate subsequent inference steps respectively.
[0046] It is worth noting that the guided resampling strategy decomposes the correct trajectory into S consecutive steps, enabling the model to resample from different intermediate inference steps, thereby achieving efficient guided data generation. The guided strategy reduces the sampling difficulty, allowing the model to quickly locate efficient exploration directions in a wide sampling space. At the same time, this strategy only requires resampling a portion of the tail data once to generate high-quality hard samples, greatly improving resampling efficiency.
[0047] In practice, the four rebalancing strategies can be used independently or in combination, with the optimal configuration chosen based on specific data distribution characteristics and performance requirements. For example, if the coverage of high-difficulty level data is high (e.g., above 10%) in the sampled correct trajectories, but the number of correct trajectories is low, then the distribution rebalancing strategies (TC and RP) and guided resampling (GR) can be combined to improve sampling diversity and further enhance the balance of data distribution. The choice of different distribution rebalancing strategies needs to consider computational overhead and performance requirements. If lower overhead is preferred, threshold pruning (TC) should be chosen, while if higher performance is preferred, repeat padding (RP) should be chosen. In scenarios where used independently, repeat padding (RP) and guided resampling (GR) methods show the best overall performance in visual reasoning scenarios.
[0048] In an optional embodiment, before using the rebalanced data as training data to perform self-improvement training on the model, the method further includes: The model is used to correct erroneous trajectories and regenerate correct trajectories to obtain self-corrected data, which is then used as part of the training data.
[0049] In one alternative embodiment, after using the model to correct erroneous trajectories, regenerating correct trajectories to obtain self-corrected data, and using the self-corrected data as part of the training data, the method further includes: The rebalancing data and the self-correction data are merged, and the correct trajectory of the merged data is filtered by length. The filtered merged data is then used as training data.
[0050] It is worth noting that, in order to achieve better performance, in addition to the four resampling strategies mentioned above, this embodiment of the invention also applies self-correction technology to the rebalancing strategy, while establishing length quality control to filter out data missing in the inference process. Specifically, the self-correction technology appends the erroneous trajectory to the model's prompts, while simultaneously indicating to the model, "There is an error in this trajectory; please correct the error and regenerate a correct inference path." Through self-correction technology, the sampled erroneous trajectories can be utilized efficiently, improving data utilization efficiency and thus increasing the amount of tail data, thereby achieving rebalancing to a certain extent. Furthermore, regarding the "Matthew effect" of response length, in order to allow the model to generate trajectories with thought chains as much as possible, data with excessively short response lengths (less than 10 tokens) can be filtered out, thereby achieving length quality control.
[0051] In an optional embodiment, after using the rebalanced data as training data to perform self-improvement training on the model, the method further includes: The performance difference of the model on the same difficulty level is calculated under conditions of image input and no image input, in order to evaluate the improvement of the model's real visual reasoning ability. Preferably, the same difficulty level problem is a high-difficulty problem.
[0052] To further ensure the high quality and accuracy of the resampled data, this embodiment of the invention verifies the real visual reasoning ability improved by different techniques by comparing the model performance under conditions with and without images in the performance evaluation. Specifically, if the model can correctly answer a question under conditions without images, it is considered that answering the question correctly under conditions with images does not reflect the model's visual reasoning ability. Therefore, this embodiment of the invention uses the performance difference of the model on high-difficulty data under conditions with and without images as the evaluation criterion, thereby verifying the real visual reasoning ability improved by different techniques. From the results, without self-boosting, the performance difference between with and without images is close to 0, indicating that the original model's visual reasoning ability is not strong, and it is difficult to correctly solve high-difficulty problems even when seeing images. However, the self-boosting method achieved a difference of 4.1 percentage points, indicating that the self-boosting method does improve the model's visual reasoning ability. The rebalancing strategy proposed in this embodiment of the invention further expands this difference. For example, the repeated filling strategy (RP) achieves a difference of 18.8 percentage points, demonstrating a significant improvement in real visual reasoning ability.
[0053] Meanwhile, experiments were conducted on different architecture models, including Qwen2-VL-7B-Instruct and InternVL2.5-4B, in this embodiment of the invention. The results show that the self-boosting method provided by this embodiment of the invention achieves a significant performance improvement.
[0054] First, this embodiment of the invention observes that simply increasing the number of samples in self-boosting does not bring about a high-performance improvement: on the Qwen2-VL-7B-Instruct model, increasing the number of samples from K=8 to K=16 only improved the average performance of self-boosting by 0.38 percentage points after 5 iterations; and given double the computational overhead, this slight performance improvement is inefficient. Therefore, the self-boosting method provided in this embodiment of the invention was verified. The results show that, in terms of overall average performance, the self-boosting method provided in this embodiment of the invention achieves an improvement of 3.86 points compared to ordinary self-boosting.
[0055] Secondly, unlike ordinary self-improvement which encounters a performance bottleneck in the later stages, the head and tail data rebalancing strategy of this invention can continuously drive the model performance improvement. Even after 5 iterations, the performance still shows an upward trend, effectively breaking through the performance ceiling of traditional self-improvement.
[0056] Meanwhile, the self-improvement method provided in this invention successfully reduces the proportion of easy data at the head and increases the proportion of difficult data at the tail, achieving a reasonable distribution of training data and alleviating the extreme head-tail imbalance. For example, in the Qwen2-VL-7B-Instruct model, the repetition padding strategy (RP) reduces the proportion of the lowest difficulty level 1 from 51.1% to 24.8%, while significantly increasing the proportion of the lowest difficulty level 5 from 1.5% to 6.6%. Furthermore, unlike existing static adjustment methods, this invention can continuously monitor and adjust the data distribution during iteration, dynamically counteracting the increasing trend of the Matthew effect.
[0057] Moreover, the embodiments of this invention employ distribution resizing strategies, including threshold pruning (TC) and repetition padding (RP), which simply change the distribution of the sampled data without consuming additional computational resources, saving significant overhead. On the other hand, a trajectory resampling strategy is adopted to improve data diversity, employing efficient resampling methods that significantly reduce computational overhead: Adaptive weighted sampling (AR) requires only about 50% of the resampling computational resources, saving half the overhead compared to brute-force sampling; Guided resampling (GR) only requires resampling each data point once, greatly improving efficiency compared to the multiple resampling steps of traditional methods. Through intelligent resampling strategies, computational resources are concentrated on the most difficult samples requiring improvement, achieving precise allocation and efficient utilization of computational resources.
[0058] In addition, the concept of self-correction was applied to the iterative process of self-improvement, avoiding the resource consumption of additional comment models and reducing system complexity.
[0059] It is worth noting that the self-improvement method provided in this embodiment of the invention exhibits strong cross-model adaptability, high cross-dataset consistency, and high cross-configuration robustness. It performs excellently on models with different architectures, such as Qwen2-VL-7B-Instruct and InternVL2.5-4B, and achieves significant improvements on multiple datasets including MMPR, MathVerse, and WeMath. It maintains stable performance improvements under different sampling numbers (K=8, K=16) and different experimental configurations.
[0060] In summary, the model self-improvement method based on data rebalancing provided by this invention can accurately identify and effectively alleviate the imbalance between the head and tail data in the model self-improvement iteration. It adjusts the selected correct trajectory data through four rebalancing strategies: threshold pruning, repetition padding, adaptive weighted sampling, and guided resampling. This makes the training data more evenly distributed across the inference trajectories corresponding to the head and tail queries, ensuring that the model can both consolidate its ability to process high-accuracy head data and fully learn from low-accuracy tail data. This significantly improves the comprehensiveness and generalization of the model's inference ability, enhancing the efficiency and stability of model self-improvement.
[0061] Based on the above method items, the present invention provides corresponding system items embodiments.
[0062] See Figure 2 , Figure 2 This is a structural block diagram of a model self-improvement system based on data rebalancing provided in an embodiment of the present invention. The model self-improvement system based on data rebalancing includes: Exploration module 21 is used to sample a batch of queries multiple times using the current model in each iteration to generate multiple inference trajectory data for each query; Filtering module 22 is used to filter out the correct trajectory from the multiple inference trajectory data to obtain the number of correct trajectories for each query; The rebalancing module 23 is used to identify head-to-tail imbalance based on the number of correct trajectories in each query; when head-to-tail imbalance is identified, a rebalancing strategy is used to adjust the data to obtain rebalanced data. Training module 24 is used to perform self-improvement training on the model using the rebalanced data as training data.
[0063] It should be noted that the model self-improvement system based on data rebalancing provided in this embodiment of the invention is used to execute all the process steps of the model self-improvement method based on data rebalancing in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.
[0064] This invention also provides a terminal device, such as... Figure 3 The diagram shown is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, it implements the model self-improvement method based on data rebalancing as described in any of the above embodiments.
[0065] In addition, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the model self-improvement method based on data rebalancing as described in any of the above embodiments.
[0066] When the processor 31 executes the computer program, it implements the steps in the above-described embodiment of the model self-improvement method based on data rebalancing, for example... Figure 1 All steps of the data rebalancing-based model self-improvement method shown. Alternatively, when the processor 31 executes the computer program, it implements the functions of each module in the above-described data rebalancing-based model self-improvement system embodiment, for example... Figure 2 The diagram illustrates the functions of each module in the data rebalancing-based model self-improvement system.
[0067] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.
[0068] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 31 can be any conventional processor. The processor 31 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.
[0069] The memory 32 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 32 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), and a flash card, etc., or the memory 32 can also be other volatile solid-state storage devices.
[0070] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural block diagram shown is merely a structural example of the terminal device described above and does not constitute a limitation on the structure of the terminal device. The terminal device may include more or fewer components than shown, or combine certain components, or use different components.
[0071] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A model self-improvement method based on data rebalancing, characterized in that, include: In each iteration, the current model is used to sample a batch of queries multiple times to generate multiple inference trajectory data for each query; The correct trajectories are selected from the multiple inference trajectory data to obtain the number of correct trajectories for each query; Based on the number of correct trajectories for each query, head-to-tail imbalance is identified; when head-to-tail imbalance is identified, a rebalancing strategy is used to adjust the data to obtain rebalanced data. The rebalanced data is used as training data to perform self-improvement training on the model.
2. The model self-improvement method based on data rebalancing as described in claim 1, characterized in that, The identification of head-and-tail imbalance based on the number of correct trajectories in each query includes: Each data point with the correct trajectory count within a first preset range is identified as head data, and each data point with the correct trajectory count within a second preset range is identified as tail data. If the proportion of the head data in the iteration process continues to increase while the proportion of the tail data in the iteration process continues to decrease, it is identified as a head-tail imbalance phenomenon.
3. The model self-improvement method based on data rebalancing as described in claim 1, characterized in that, The rebalancing strategy includes at least one of the distribution readjustment strategy and the trajectory resampling strategy; The distribution readjustment strategy includes at least one of a threshold pruning strategy and a repeat padding strategy; the trajectory resampling strategy includes at least one of an adaptive weighted sampling strategy and a guided resampling strategy.
4. The model self-improvement method based on data rebalancing as described in claim 3, characterized in that, The threshold pruning strategy includes: Count the number of correct trajectories, k, for each query across K samples; For queries where the number of correct trajectories k is greater than the preset threshold L, L correct trajectories are randomly retained, and the remaining kL correct trajectories are deleted; where k is less than or equal to K, and K, k, and L are all positive integers greater than zero; The repeated filling strategy includes: Count the number of correct trajectories, k, for each query across K samples; For queries where the number of correct trajectories k is less than K, generate correct trajectories that are identical to existing correct trajectories until the number of correct trajectories k equals K.
5. The model self-improvement method based on data rebalancing as described in claim 3, characterized in that, The adaptive weighted sampling strategy includes: Count the number of correct trajectories k for each query in K samplings, and use the model to perform Kk resampling to obtain the correct trajectory data after resampling; where k is less than or equal to K, and K and k are both positive integers greater than zero; The guided resampling strategy includes: Decompose the correct trajectory of the tail data into multiple steps according to the length of the correct trajectory; The model is used to resample from the intermediate step of each correct trajectory to obtain the correct trajectory data after resampling.
6. The model self-improvement method based on data rebalancing as described in claim 1, characterized in that, Before using the rebalanced data as training data to perform self-improvement training on the model, the method further includes: The model is used to correct erroneous trajectories and regenerate correct trajectories to obtain self-corrected data, which is then used as part of the training data.
7. The model self-improvement method based on data rebalancing as described in claim 1, characterized in that, After using the rebalanced data as training data to perform self-improvement training on the model, the method further includes: Calculate the performance difference of the model on the same difficulty under conditions with and without image input, in order to evaluate the improvement of the model's real visual reasoning ability.
8. A model self-improvement system based on data rebalancing, characterized in that, include: The exploration module is used to sample a batch of queries multiple times in each iteration using the current model, generating multiple inference trajectory data for each query; The filtering module is used to filter out the correct trajectories from the multiple inference trajectory data to obtain the number of correct trajectories for each query; The rebalancing module is used to identify head-to-tail imbalance based on the number of correct trajectories for each query; when head-to-tail imbalance is identified, a rebalancing strategy is used to adjust the data to obtain rebalanced data. The training module is used to train the model by using the rebalanced data as training data.
9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data rebalancing-based model self-improvement method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the model self-improvement method based on data rebalancing as described in any one of claims 1 to 7.