Adaptive fine-tuning system for a database-based speech recognition model and control method thereof

Abstract

The present invention relates to an adaptive fine-tuning system for a database-based speech recognition model and a control method thereof. To this end, the present invention defines 17 or more learning strategies in a learning strategy table (finetune_plan) as records in 23 columns, automatically selects learning data in hours based on voice data quality metrics (CER, voice quality score, noise type) and usage history (usage_count), and performs multiple consecutive fine-tuning without code changes through session-based automatic chaining and equal distribution of the retraining pool. For every training session, 77 items of multidimensional metrics (performance indicators, data structure, quality distribution, training settings, strategy information, time / state) are automatically recorded in the database to support immediate analysis via SQL queries and the formulation of future strategies. Experimental results showed that for 10.57 million call center voice records (15,178 hours), the Whisper-large-v3 model achieved a CER of 0.0249 (minimum 0.0075) over 239 rounds, while the Whisper-small model achieved a 69% improvement in CER (0.1594→0.0500) over 377 rounds, demonstrating universality regardless of model size. The present invention structurally prevents overfitting by limiting the maximum value of usage_count (preferably 4 times) and distributing it evenly per round, enhances real-world robustness with a 4-stage noise introduction strategy, and guarantees reproducibility using the same session / seed combination. It provides scalability that allows the same framework to be applied to various speech recognition models other than Whisper, such as Conformer, wav2vec 2.0, and HuBERT.

Description

Technology Field

[0001] The present invention relates to a fine-tuning technology for an AI and machine learning (ML)-based speech-to-text (STT) model. More specifically, it integrates the automatic selection of training data, automatic switching of multiple training strategies, management and even distribution of data reuse history, and recording of multidimensional metrics of training results by utilizing quality metrics (Character Error Rate, sound quality score, noise type) and training strategy information of voice data stored in a relational database (RDBMS). In particular, it relates to a fine-tuning technology for a speech-to-text model that accumulates voice data quality indicators (CER, audio_score) and usage history (usage_count) in a database, automatically selects data according to 17 or more training strategies defined in a training strategy table (finetune_plan), automatically chains a previous training model as the base model for the next session on a session-by-session basis, and evenly distributes reused data by round through a retrain_pool table, thereby enabling the switching of training strategies solely through database settings without code modification. Background Technology

[0003] Unless otherwise indicated in this specification, the contents described in this identification item are not prior art for the claims of this application, and are not recognized as prior art simply because they are described in this identification item.

[0004] As is well known, fine-tuning refers to the process of taking a pre-trained model that has already learned from vast amounts of data and is equipped with intelligence, and further training it to suit a specific task or domain.

[0005] However, even the conventional fine-tuning that is useful as described above has the following problems.

[0006] First, there are limitations to manual management. Specifically, in conventional speech recognition model fine-tuning techniques, settings required for training—such as hyperparameters, data paths, learning rates, and epochs—were managed by directly hardcoding them within the source code or entering them into separate configuration files (YAML, JSON). This approach involved the inconvenience of having to manually edit and redeploy the source code whenever the training strategy was to be changed. For instance, even simple changes like shifting the learning rate from 0.00003 to 0.00001 or increasing the number of epochs from 5 to 10 required going through the entire software development cycle, including code modification, code review, testing, and deployment. Furthermore, because version control regarding previously trained settings was not systematically maintained, it was difficult to reproduce training settings at a specific point in time, and there was a problem where consistency could not be guaranteed when repeating the same experiment. In practical environments, it is common for multiple researchers to attempt fine-tuning on the same model with different settings; however, in source code-based configuration management, such parallel experimentation causes code conflicts, significantly reducing development efficiency.

[0007] Second, there are issues regarding data redundancy and overfitting. Specifically, in conventional technology, training data is managed in file formats such as CSV or JSON, making it virtually impossible to track how many times specific voice data has been used in training (usage_count). Consequently, some high-quality data is repeatedly used in every training session, leading to overfitting to the corresponding data patterns. Overfitted models exhibit a low error rate on the training data but display a phenomenon of generalization failure, where performance degrades sharply when recognizing new voices in actual operating environments. Conversely, the lack of data usage history also resulted in data waste, where some high-quality data is never utilized for training. This problem is even more severe in environments with large-scale voice datasets (e.g., over 10 million records), where the utilization rate of the total data frequently remains below 30%.

[0008] Third, there is the control of training volume based on time versus count. In other words, conventional fine-tuning techniques selected training data based on the number of records (count). For instance, the training volume was controlled in a manner such as "using 10,000 records for training." However, since voice data varies in file length, the actual audio duration (hours) can differ significantly even with the same 10,000 records. If a large number of short audio files (1–3 seconds) are included, 10,000 records could amount to only 5 hours, whereas if a large number of long audio files (20–30 seconds) are included, the same 10,000 records could reach 80 hours. This count-based control method made it difficult to predict the actual size of the training volume, leading to significant errors in establishing GPU resource allocation plans and estimating training time. Furthermore, there was a problem in that quantitative consistency of data between training sessions was not guaranteed, making comparative analysis of training results difficult.

[0009] Fourth, there is a lack of strategy management. Specifically, in the fine-tuning of speech recognition models, applying different strategies depending on the early, mid, and late stages of training is effective for performance optimization. For instance, a stepwise strategy is required where rapid convergence is achieved using a high learning rate (0.00003) in the early stages, generalization performance is enhanced by mixing reused data in the mid stages, and fine-tuning is performed using a low learning rate (0.000005) in the late stages. However, conventional technology lacked the means to systematically manage and automatically switch between these multi-stage strategies. In practice, there may be more than 17 training strategies; however, implementing them using conditional statements (if-else) within the source code drastically increases code complexity and makes maintenance difficult. Furthermore, the absence of a systematic framework to compare and analyze the execution history and performance of each strategy made it difficult to determine which strategy is most effective under specific conditions. As the number of possible strategy combinations increases exponentially during experimental design, the limitations of manual management became evident.

[0010] Fifth, there are difficulties in tracking training results. Specifically, conventional technologies recorded training results in text logs or TensorBoard. Since log files are unstructured text files, they require programmatic parsing; furthermore, there is a risk of file loss during long-term storage, and systematic comparative analysis across multiple training sessions was virtually impossible. While TensorBoard is suitable for visualization, it does not support structured data queries (SQL-based conditional search, aggregation, and sorting), making it impossible to perform complex conditional analyses, such as "the average learning rate of sessions with a CER of 0.05 or less and a reused data ratio of 30% or more." In particular, there was a complete lack of means to comprehensively manage and correlate multidimensional metrics—such as the quality distribution of data used in each training session (ratio of excellent / good / fair / poor), the ratio of new to reused data, the ratio of noise data, actual training time, and GPU utilization.

[0011] Sixth, there are limitations in processing noise data. Specifically, voice data from actual call center environments contains various types of noise, such as background noise, low-quality network noise, echo, reverb, crosstalk, wind noise, and mechanical noise. Conventional technologies either treated all data equally for training without classifying these noise types, or conversely, selected only clean voice data for training. In the former case, heavily noisy data hindered training and degraded overall performance, while in the latter case, a domain mismatch problem occurred where performance deteriorated significantly when recognizing voice containing noise in actual operating environments. The ideal approach would be to establish basic performance using clean data and then gradually introduce noise data to strengthen the robustness of the model; however, conventional technologies lacked a means to automate this stepwise introduction of noise.

[0013] In order to solve the aforementioned problems, the following prior art documents were developed in the past; however, a major problem still arose in that the problems of the aforementioned prior art could not be solved all at once. Prior art literature

[0015] Korean Registered Patent Publication No. 2932886 (February 25, 2026) has been registered. Korean Registered Patent Publication No. 2897338 (December 3, 2025) has been registered. Korean Registered Patent Publication No. 2771713 (February 18, 2025) has been registered. Korean Registered Patent Publication No. 2895581 (December 1, 2025) has been registered. Korean Registered Patent Publication No. 2808637 (May 12, 2025) has been registered. The problem to be solved

[0016] The present invention has been devised to resolve the various problems of the prior art as described above, and its purpose is as follows.

[0017] First, it provides a method to automatically select training data by accumulating voice data quality metrics (CER, voice quality score, noise type) in a relational database and dynamically applying quality criteria according to the learning strategy. In this process, the amount of training is quantitatively controlled based on actual voice time (hours) rather than the number of data records, and overfitting caused by overuse is prevented by tracking the usage count of each data point. Through this, large-scale voice data of 10.57 million records can be systematically managed, and optimal data can be automatically configured for every training session.

[0018] Second, the system defines learning strategies (hyperparameters, data composition ratios, quality criteria) as records in a database table (finetune_plan), thereby providing a system that allows adding, modifying, or disabling learning strategies solely through database settings without changing source code. This system supports more than 17 types of learning strategies and maximizes the flexibility of experiment design by allowing the free combination of detailed strategies in the DB, ranging from the experimental stage (Type 1-10) to full-scale learning (Type 11-14) and advanced strategies (Type 15-17).

[0019] Third, it provides a continuous learning system that automatically loads the model from the previous training round as the base model for the next round through a session-based automatic chaining mechanism, and allows training to resume from a specific session after interruption. Through this, long-term continuous learning of 377 rounds or more can be performed automatically without human intervention.

[0020] Fourth, it provides a method to manage reusable data in cycle / round units through the retrain pool (stt_retrain_pool) table and perform reused training that is not biased toward specific data using the extract_reused_data_evenly algorithm. It structurally prevents overfitting by limiting the usage_count to a preset maximum value (preferably 4 times).

[0021] Fifth, it provides a method for managing learning results that automatically records 77 items of metrics (performance indicators, data structure, quality distribution, learning settings, strategy information, etc.) in a database for every learning session, allowing for the immediate analysis of performance trends of multiple learning sessions using SQL queries and their use in establishing next learning strategies.

[0022] Sixth, the robustness of speech recognition in actual operating environments is improved through a learning strategy that gradually expands the types of noise (Stage 1: use only clean → Stage 2: clean + mild noise → Stage 3: add medium noise → Stage 4: include all noise). This gradual introduction of noise enables the model to adapt to noisy environments while maintaining its basic performance in clean speech. means of solving the problem

[0024] To achieve this objective, the present invention includes the following steps.

[0025] (a) A learning strategy loading step (refer to the strategy loader (110) in FIG. 1 and step S110 in FIG. 2), which is a step of querying information on the learning strategy type (train_type), session number (session), new data usage (new_hours), reused data usage (reused_hours), noise data usage (noise_hours), learning rate (learning_rate), number of epochs (num_train_epochs), maximum allowable CER (max_cer), minimum sound quality score (min_score), data strategy name (data_strategy), and usage status (use_yn) from the learning strategy table (finetune_plan) of the relational database. The learning strategy table consists of 23 columns and manages all settings required for learning as a single record, and only active strategies with a use_yn column of 'Y' are queried in session order and determined as execution targets. The strategy loader (110) converts the retrieved records into a Python dictionary format and passes them to subsequent modules, and applies a lock file mechanism to prevent duplicate execution for the same session number.

[0026] (b) A data automatic selection step (see data selector (120) of FIG. 1, steps S120 to S133 of FIG. 2 and FIG. 3), wherein training data is automatically selected from the voice data metadata table (stt_dataset) of the database, and (b-1) data in which the CER is less than or equal to max_cer and the voice quality score is greater than or equal to min_score according to the quality criteria (max_cer, min_score) defined in the learning strategy is filtered, (b-2) only data in which the usage count (usage_count) is less than a preset maximum value (4 times) is selected to prevent excessive use of specific data, and (b-3) a time-based quantitative selection is performed by randomly shuffling the selected data based on a seed and accumulating the audio duration of each data until the target time (target_hours) defined in the learning strategy is reached. The above time-based screening fundamentally resolves variations in learning amount caused by imbalances in data counts and guarantees reproducibility under identical conditions by using the same seed.

[0027] (c) A step for even distribution of reusable data (refer to the retraining pool management unit (160) in FIG. 1 and FIG. 4), wherein if the type of the learning strategy is a type that includes reusable data (train_type 11, 12, 16, 17), the reusable data stored in the retraining pool table (stt_retrain_pool) in cycle / round units is queried, and in each round, the target reuse time (reused_hours) is divided by the number of rounds (N) to be extracted evenly, and the used_flag of the extracted data is updated. The equal distribution algorithm ensures that the model can evenly learn data patterns at various training points by extracting data at the same rate in all rounds so as not to be biased toward data in a specific round. It includes a fallback logic that supplements extraction from another round if the extraction time per round is insufficient for the available data in that round.

[0028] (d) A data splitting step (refer to step S130 in Fig. 2), wherein the entire training data selected in steps (b) and (c) is split into training (train, 80%), validation (validation, 10%), and evaluation (evaluation, 10%) ratios, and the splitting information (data_split) for each data is placed and updated in the database. The splitting ratios can be adjusted according to the training strategy, the validation data is used to monitor overfitting in real time during training, and the evaluation data is used to measure final performance after training is completed. When splitting the data, speaker-unit splitting may be applied so that data of the same speaker is not included in both training and evaluation simultaneously.

[0029] (e) A model automatic loading step (refer to step S140 in FIG. 2), wherein the session number of the training strategy is checked, and if a model saved in the previous session exists, the corresponding model (. / models / session_{N-1} / ) is loaded as the base model, and if it does not exist, the original pre-trained model (e.g., openai / whisper-large-v3) is loaded to perform automatic chaining between sessions. Through this, the training of each session can inherit the results of the previous session, enabling cumulative performance improvement, and if performance degradation occurs in a specific session, it is possible to roll back to the model prior to that session and attempt retraining with a different strategy.

[0030] (f) A fine-tuning execution step (refer to the fine-tuning execution engine (150) of FIG. 1 and step S150 of FIG. 2), wherein fine-tuning is performed on the model loaded in step (e) using the training data and validation data split in step (d) with the hyperparameters (learning_rate, num_train_epochs, per_device_train_batch_size) defined in the learning strategy. During training, the validation loss for the validation data is calculated at every epoch to determine whether to stop early, and the optimal checkpoint is automatically saved. After training is complete, the model is saved to the session path (. / models / session_{N} / ), and the tokenizer and processor settings are saved along with the checkpoint file to form a complete model package capable of independent inference.

[0031] (g) A performance evaluation step (refer to the performance evaluation section (170) in FIG. 1 and step S160 in FIG. 2), wherein the Character Error Rate (CER) and Word Error Rate (WER) of the fine-tuned model are calculated using the evaluation data divided in step (d), and the performance is compared with that of the original pre-trained model. When calculating the CER, Korean-specific normalization (removal of spaces, removal of punctuation, conversion of numbers into Hangul) is applied, and the Levenshtein distance is calculated by dividing the reference text length. The evaluation result is converted into a session-specific improvement rate (improvement_rate = (previous_CER - current_CER) / previous_CER × 100) to quantitatively measure the learning effect.

[0032] (h) A learning result recording step (refer to the relational database (140) in FIG. 1 and step S170 in FIG. 2), wherein the results of the fine-tuning are automatically recorded in a learning result table (stt_learning_report) as 77 metrics, including performance indicators (CER, WER, loss), data composition information (ratio of new / reused data, quality distribution), learning settings (learning_rate, batch_size, epochs), strategy information (strategy_name, strategy_type), and time information (learning start / end, actual time taken). The 77 metrics consist of performance metrics (10), data composition metrics (12), quality distribution metrics (8), learning setting metrics (8), strategy metrics (6), time / state metrics (10), and other metrics (23).

[0033] (i) As a usage history update step (refer to step S180 in Fig. 2), the usage count of each data used for training in step (f) is increased by 1, and the current session number is added to the used round information (used_in_round) to update the database. This update is performed in batch units to minimize DB load even on large datasets, and data consistency is guaranteed through transaction rollback in the event of update failure.

[0034] (j) A sequential execution determination step (refer to the session orchestrator (180) in FIG. 1 and step S190 in FIG. 2), which checks whether the next session exists in the learning strategy table; if it exists, returns to step (a) to automatically start the learning of the next session, and if it does not exist, unlocks the lock file and terminates the execution. This automatic traversal mechanism enables hundreds of consecutive sessions to be executed automatically without human intervention, and ensures operational stability by allowing the session to be resumed from the interrupted session via the start_session parameter even in the event of an abnormal termination.

[0035] Meanwhile, the present invention is equipped with a hybrid strategy for time-based data selection.

[0036] That is, in the time-based quantitative screening (b-3) above, the target time (target_hours) is determined by a hybrid strategy. Specifically, the larger value between the percentage-based time (total time of all available data × percentage / 100) and the minimum guaranteed time for each stage (3 hours for iterations 1–10, 5 hours for iterations 11–20, and 10 hours for iterations 21–25) is determined as the target time. This hybrid strategy implements adaptive learning amount control, which guarantees a minimum amount of learning even when data is scarce in the early stages of learning, while gradually increasing the amount of learning as data accumulates.

[0037] In addition, the present invention is equipped with relearning pool management (equal distribution based on cycle / round).

[0038] In other words, the retraining pool table (stt_retrain_pool) stores data that showed good performance in previous training in cycle / round units. Each cycle represents a complete training period, and each round represents a segmented group of data within that cycle. The even distribution algorithm (extract_reused_data_evenly) ensures that data from all rounds participates evenly in retraining by extracting (reused_hours / N) hours of data from each round when there are N rounds. This enables balanced retraining that is not biased by data patterns at a specific point in time.

[0039] Furthermore, the present invention is equipped with session chaining (session N → session N+1 automatic connection).

[0040] In other words, automatic session chaining is a mechanism where the model saved upon the completion of Session N (. / models / session_{N} / ) is automatically loaded as the base model when Session N+1 begins training. This allows each session to inherit the training results from the previous session, enabling cumulative performance improvement and eliminating the need for manual model path specification. Session chaining supports the `start_session` parameter, allowing training to resume from a specific session upon interruption and enabling rollback to a previous session to retry with a different strategy if the performance of a particular session is unsatisfactory.

[0041] In addition, the present invention is equipped with a stepwise introduction of noise (Stage 1 to 4).

[0042] In other words, noise-adaptive learning is structured in stages. In Stage 1 (clean), basic recognition performance is established using only clean, noise-free speech data. In Stage 2 (mild), data containing mild background noise is added. In Stage 3 (moderate), moderate-level noise, such as echo / reverb and wind noise, is added. In Stage 4 (all), training is performed under conditions identical to the real environment by including all noise types, such as music overlay and traffic noise. The number of noise types and the stage configuration are scalable depending on the domain, and each stage transition is controlled by the noise_hours and data_strategy columns of the finetune_plan table.

[0043] In addition, the present invention is equipped with 77 metric automatic recordings.

[0044] In other words, 77 multidimensional metrics are automatically recorded in the stt_learning_report table upon the completion of each learning session. These metrics are classified into six groups (performance indicators, data composition, quality distribution, learning settings, strategy information, and time / status) and can be analyzed immediately via SQL queries. For example, the query "SELECT session, cer, new_data_ratio, reused_data_ratio FROM stt_learning_report WHERE cer < 0.05 ORDER BY session" allows for the analysis of the data composition of sessions that achieved a CER of less than 0.05 to derive the optimal strategy. Additionally, the improvement rate per session is automatically calculated by comparing the previous_CER column with the current CER, and the causes of performance degradation in sessions (changes in data composition, learning rate changes, etc.) can be traced back. Effects of the invention

[0046] As described in detail above, the present invention provides the following effects.

[0047] First, there is a groundbreaking improvement in speech recognition performance. Experimental results according to the present invention showed that the Whisper-large-v3 model achieved a 32% performance improvement, increasing from an initial Round 1 average CER of 0.0366 to an average CER of 0.0249 in Rounds 201-239, and recorded a minimum CER of 0.0075. The Whisper-small model achieved a 69% performance improvement, increasing from an initial Round 1-50 average CER of 0.1594 to an average CER of 0.0500 in Rounds 201-239. This demonstrates that the system of the present invention can be effectively applied regardless of model size.

[0048] Second, it involves the systematic management of large-scale voice data. The present invention manages 10.57 million records (approximately 15,178 hours) of call center voice data using a relational database, tracking the quality metric (CER, voice quality score), usage history (usage_count), and noise type of each data in real time. Compared to conventional file-based management, conditional data selection is possible with a single line of SQL query, and fast querying is possible even in large datasets through index-based searching.

[0049] Third, the training preparation time is significantly reduced. In conventional manual fine-tuning methods, manual tasks such as data preparation, setting changes, and result recording were required for every session, but this invention almost eliminates such manual work through DB-based automation. In actual operation, 377 rounds of continuous training were automatically performed without human intervention, and the time between sessions (data selection, model loading, etc.) averaged 3 minutes per session, which was less than 2% of the total training time.

[0050] Fourth, there is a structural guarantee to prevent overfitting. By limiting usage_count to within a preset maximum value and applying equal distribution per round, overfitting caused by excessive use of specific data is structurally prevented. Experimental results confirmed that even during 377 rounds of continuous training, no sharp increase in validation_loss (signs of overfitting) occurred, and the CER remained stable.

[0051] Fifth, the stability of long-term continuous learning is verified. The stability of the system of the present invention has been verified in 377 rounds of continuous learning (including 55 automatic strategy switching). Mechanisms such as lock file-based duplicate execution prevention, transaction-based data consistency guarantee, and session-based model checkpoint storage ensure reliability in a long-term operating environment.

[0052] Sixth, there is the automatic switching of 55 training strategies. The 55 training strategies defined in the finetune_plan table are automatically switched according to the session order, and strategies can be added, modified, or deactivated simply by editing DB records without changing the source code. This maximizes the flexibility of experiment design and completely eliminates the software deployment cycle associated with strategy changes.

[0053] Seventh, it guarantees reproducibility. Using the same session number and seed value allows for the reproduction of identical data selection and learning results. This is essential for verifying experimental results, publishing papers, and proving experimental reproducibility during patent applications, and it significantly improves experimental reliability compared to conventional non-deterministic learning methods.

[0054] The above effects are not mutually independent but are organically connected to generate synergy. For example, systematic data management (second) enables the prevention of overfitting (fourth), and the recording of 77 metrics (fifth) allows for the comparative analysis of the effectiveness of 55 strategies (sixth), thereby achieving the final results of a 32% improvement in CER for Whisper-large-v3 and a 69% improvement in CER for Whisper-small (first). In particular, achieving consistent performance improvement with the same system regardless of model size is a key basis for proving the universality of the present invention.

[0055] Compared to conventional technology, the present invention enables immediate switching by modifying DB records without code modification or distribution when changing strategies, enables immediate analysis of learning results with SQL queries, and provides the effect of maintaining stable performance without overfitting in 377 rounds of continuous learning.

[0057] Hereinafter, a preferred embodiment of the present invention for achieving these effects will be described in detail with reference to the attached drawings. Brief explanation of the drawing

[0059] FIG. 1 is a DB-based speech recognition model adaptive fine-tu according to an embodiment of the present invention. This is a system block diagram showing the overall configuration of the Ning system. FIG. 2 illustrates a session-based learning execution flow according to an embodiment of the present invention. It is Seodo. FIG. 3 is a DB-based training data automatic selection algorithm according to an embodiment of the present invention This is a diagram showing the detailed flow of the system. FIG. 4 is a retraining pool-based data equal distribution algorithm according to an embodiment of the present invention. This is a diagram showing the theory. FIG. 5 shows the configuration of a learning strategy by Train Type and a session according to an embodiment of the present invention. This is a diagram showing the automatic chaining process. Specific details for implementing the invention

[0060] The applied to the present invention is configured as shown in FIGS. 1 to 5.

[0061] In the following description of the present invention, if it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.

[0062] Furthermore, the terms described below are established considering their functions in the present invention, and since these may vary depending on the producer's intent or practice, their definitions should be based on the content throughout this specification.

[0063] In addition, the size and thickness of each component shown in the drawings are depicted arbitrarily for convenience of explanation, so the present invention is not necessarily limited to what is shown in the drawings.

[0064] First, the present invention relates to an adaptive fine-tuning system (100) for a database-based speech recognition model and is configured as follows.

[0065] That is, the present invention is provided with a strategy loader (110) that queries a learning strategy from a learning strategy table of a relational database.

[0066] In addition, the present invention is provided with a data selector (120) that automatically selects training data according to quality criteria and usage limit defined in the learning strategy in a voice data metadata table, and performs time-based quantitative selection by accumulating the actual voice time of each data and extracting it until a target time is reached.

[0067] And the present invention is provided with a quality evaluation unit (130) that evaluates the quality of voice data and assigns a quality grade.

[0068] In addition, the present invention is provided with a relational database (140) that stores learning strategies, voice data metadata, learning results, and retraining pool information.

[0069] The present invention is provided with a fine-tuning execution engine (150) that performs fine-tuning of a speech recognition model using the selected data and stores the learned model in a session-specific path.

[0070] In addition, the present invention is equipped with a relearning pool management unit (160) that evenly distributes reused data by round in a relearning pool table.

[0071] And the present invention is provided with a performance evaluation unit (170) that calculates the CER and WER of a fine-tuned model.

[0072] In addition, the present invention is equipped with a session orchestrator (180) that sequentially executes the strategy loader, data selector, fine-tuning execution engine, retraining pool management unit, and performance evaluation unit on a session basis, and manages automatic model chaining between sessions.

[0073] Meanwhile, the strategy loader (110) applied in the present invention supports 17 or more types of learning strategies, and includes experimental stage strategies (Type 1-10), full-scale learning strategies (Type 11-14), and advanced strategies (Type 15-17).

[0074] In this case, it is desirable that the above learning strategy be managed solely by adding, modifying, or deactivating records in the above learning strategy table without changing the code.

[0075] Meanwhile, the relational database (140) applied to the present invention is equipped with a finetune_plan table (141) that defines a learning strategy with 23 columns; a stt_dataset table (142) that stores voice data quality metrics and usage history; a stt_learning_report table (143) that records the results of each learning session with 77 metrics; and a stt_retrain_pool table (144) that manages reusable data in cycle / round units.

[0077] Meanwhile, the present invention can be modified in various ways and can take various forms when applying the above-mentioned components.

[0078] And it should be understood that the present invention is not limited to the specific form mentioned in the above detailed description, but rather should be understood to include all variations, equivalents, and substitutions within the spirit and scope of the invention as defined by the appended claims.

[0080] The effects of the adaptive fine-tuning system and control method of the database-based speech recognition model of the present invention configured as described above are as follows.

[0081] FIG. 1, applied to the present invention for this purpose, is a system block diagram showing the overall configuration of a DB-based speech recognition model adaptive fine-tuning system (100) according to one embodiment of the present invention. The system (100) includes a strategy loader (110), a data selector (120), a quality evaluation unit (130), a relational database (140), a fine-tuning execution engine (150), a retraining pool management unit (160), a performance evaluation unit (170), and a session orchestrator (180), and the data flow and control flow between each component are illustrated by arrows. The relational database (140) is composed of four tables: finetune_plan (141), stt_dataset (142), stt_learning_report (143), and stt_retrain_pool (144).

[0082] FIG. 2 is a flowchart illustrating a session-based learning execution flow according to an embodiment of the present invention. It illustrates a flow in which a session orchestrator (180) sequentially queries active sessions from a finetune_plan table and automatically repeats the steps of strategy loading (S110) → data selection (S120) → data splitting (S130) → model loading (S140) → fine-tuning (S150) → performance evaluation (S160) → result recording (S170) → usage history update (S180) → next session determination (S190) for each session. It includes logic for creating / unlocking a lock file and recovery in case of an error.

[0083] FIG. 3 is a diagram showing the detailed flow of a DB-based automatic training data selection algorithm according to an embodiment of the present invention. It consists of four steps: quality-based filtering (S131), time-based quantitative selection (S132), data composition branching by train_type (S133), and data splitting (S134). The SQL queries and algorithm logic applied in each step are illustrated in detail in the form of a flowchart. It also includes branching conditions for hybrid time determination logic (percentage vs. minimum guaranteed time).

[0084] FIG. 4 is a diagram illustrating a retraining pool-based data equal distribution algorithm according to an embodiment of the present invention. The steps of data grouping by cycle / round (S161), equal extraction by round (S162), updating the usage flag (S163), and increasing the usage_count (S164) are illustrated in a flowchart. It includes logic for equal extraction for (target_hours / N) hours in each round when N rounds exist, and a fallback mechanism for supplementing from another round when available data is insufficient in a specific round.

[0085] FIG. 5 is a diagram illustrating the configuration of learning strategies by Train Type and the automatic session chaining process according to an embodiment of the present invention. The data configuration (new / reused / noise ratio), application conditions, and transition rules between sessions for 17 learning strategies are diagrammed. A three-layer structure of experimental stages (Type 1-10), full-scale learning stages (Type 11-14), and advanced strategy stages (Type 15-17), and a chaining structure in which the output model of Session N is automatically connected to the input model of Session N+1 are visually illustrated.

[0086] (First Example)

[0087] FIG. 1 is an overall configuration diagram of a DB-based speech recognition model adaptive fine-tuning system according to a first embodiment of the present invention. Referring to FIG. 1, the system (100) of the present invention includes a strategy loader (110), a data selector (120), a reusable data equal distribution step (S125), a quality evaluation unit (130), a relational database (140), a fine-tuning execution engine (150), a retraining pool management unit (160), a performance evaluation unit (170), and a session orchestrator (180). The detailed operation of each component is described below.

[0088] The strategy loader (110) is a module that queries the learning strategy of the current session from the learning strategy table (finetune_plan, 141) of the relational database (140). The finetune_plan table (141) consists of 23 columns: id, phase, session, train_cycle, use_yn, train_type, train_hours, num_train_epochs, learning_rate, total_train_hours, cumulative_original_hours, cumulative_train_hours, data_strategy, notes, reused_hours, new_hours, noise_hours, reused_percentage, early_hours, recent_hours, etc_hours, max_cer, and min_score, and each record defines a complete learning strategy.

[0089] The strategy loader (110) can retrieve a list of active strategies by executing the query "SELECT * FROM finetune_plan WHERE use_yn = 'Y' ORDER BY session ASC" and, if the start_session parameter is specified, add the condition "AND session >= {start_session}" to start execution from a specific session. The retrieved strategy information is converted into a Python dictionary and passed to subsequent modules such as a data selector (120) and a fine-tuning execution engine (150).

[0090] The detailed schema of the above finetune_plan table (141) is as shown in the following [Table 1].

[0091] Column name Type explanation id INT Primary Key (Auto-increment) phase VARCHAR Learning stage name (e.g., "Experiment", "Main Learning", "Advanced") session INT Session number (sequential execution unit, 1~N) train_cycle INT Learning cycle number use_yn CHAR(1) Usage status (Y: Active, N: Inactive) train_type INT Types of learning strategies (1~17) train_hours DECIMAL Total training data time num_train_epochs INT Number of epochs (e.g., 5, 10) learning_rate DECIMAL Learning rate (e.g., 0.00001) total_train_hours DECIMAL Total study time (in hours) cumulative_original_hours DECIMAL Cumulative original data time cumulative_train_hours DECIMAL Cumulative learning time data_strategy VARCHAR Data Strategy Name (e.g., "progressive", "balanced") notes TEXT Remarks / Memo reused_hours DECIMAL Reusable data time new_hours DECIMAL New data time noise_hours DECIMAL Noise data time reused_percentage DECIMAL Data Reuse Rate (%) early_hours DECIMAL Initial round data time recent_hours DECIMAL Recent round data time etc_hours DECIMAL Other data time max_cer DECIMAL Maximum allowable CER (e.g., 0.4) min_score INT Minimum sound quality score (e.g., 70)

[0092] As a specific example, when a record is stored with train_type=11, session=280, new_hours=10.0, reused_hours=5.0, learning_rate=0.00001, num_train_epochs=5, max_cer=0.4, min_score=70, the strategy loader (110) looks up this setting and constructs a complete training plan with a learning rate of 0.00001 for 5 epochs using new 10 hours + reused 5 hours of data with an incremental data reinforcement strategy in session 280. By changing the use_yn column to 'N', the strategy can be disabled and excluded from execution, and the execution order can be rearranged by changing the session value.

[0093] FIG. 2 illustrates a session-based learning execution flow. The session orchestrator (180) is a top-level control module that manages the lifecycle of the entire learning process.

[0094] The execution flow of the session orchestrator (180) is as follows: (1) Create a lock file ( / tmp / whisper_finetuning.lock) to prevent duplicate execution. (2) Query records with use_yn='Y' in the finetune_plan table in session order. (3) For each session, sequentially execute strategy loading (S110) → data selection (S120) → data splitting (S130) → model loading (S140) → fine-tuning (S150) → performance evaluation (S160) → result recording (S170) → usage history update (S180). (4) After a session is completed, check for the existence of the next session (S190) and execute it automatically in succession. (5) Release the lock file and terminate when all sessions are completed or an error occurs.

[0095] If an error occurs, the error information for the session is recorded in the stt_learning_report table along with status="FAILED" and error_message, and the next session is terminated without execution. Subsequently, the session can be retried by specifying the start_session parameter as the session.

[0096] FIG. 3 illustrates the detailed flow of a DB-based automatic training data selection algorithm. A data selector (120) automatically selects training data from a voice data metadata table (stt_dataset, 142). The stt_dataset table (142) stores information such as file path, audio length, transcribed text, quality metric, and usage history for each voice file, and consists of a total of 10.57 million records.

[0097] The detailed schema of the above stt_dataset table (142) is as shown in the following [Table 2].

[0098] Column name Type explanation id BIGINT Primary Key (Auto-increment) file_path VARCHAR absolute path of the audio file audio_duration DECIMAL Audio length (in seconds, 2 decimal places) transcription TEXT Transcription text (correct answer label) cer DECIMAL Character Error Rate (0.0~1.0) wer DECIMAL Word Error Rate (0.0~1.0) audio_score INT Sound quality score (0~100) noise_type VARCHAR Noise types (clean, background_noise, echo_reverb, etc.) noise_level VARCHAR Noise intensity (clean / mild / moderate / severe) usage_count INT Number of times learning is used (default 0, apply maximum configurable value) used_in_round VARCHAR Used round history (JSON array) data_split VARCHAR Data splitting (train / validation / evaluation) speaker_id VARCHAR Speaker identifier recording_date DATE Recording date quality_grade VARCHAR Quality Grade (excellent / good / fair / poor)

[0099] The data screening process is performed in 5 steps as follows.

[0100] First (S131), it is a quality-based filtering step. Based on max_cer and min_score received from the strategy loader (110), the query "SELECT * FROM stt_dataset WHERE cer <= {max_cer} AND audio_score >= {min_score} AND usage_count < 4" is executed to retrieve data that meets the quality criteria. For example, if max_cer=0.4 and min_score=70, only data with a CER of 0.4 or less, an audio quality score of 70 or more, and a usage count of less than 4 is selected.

[0101] Second (S132), the time-based quantitative screening step. After randomly shuffling the filtered data based on a seed, the audio_duration of each data is cumulatively summed and sequentially extracted until the target_hours (target time) is reached. At this time, the target_hours is determined by the new_hours value of the finetune_plan, and if a hybrid strategy is applied, it is determined by the larger value between the percentage-based time and the minimum guaranteed time.

[0102] Third (S133), it is a train_type branching step. Depending on the train_type, it branches to use only new data (Type 1-10, 13), mix reused data from the retraining pool (Type 11, 12, 16, 17), mix various data using balanced sampling (Type 14), or add noise data (Type 15). In Type 1-10, reused_hours is forced to 0 so only new data is used; in Type 13, max_cer is forced to 0.05 so only high-quality data is selected; in Type 14, various data distributions are reflected through balanced sampling; in Type 15, noise data is additionally selected for the time specified in noise_hours; and in Type 16, retraining data for each round is evenly extracted and mixed with new data through the equal distribution algorithm (extract_reused_data_evenly) of the retraining pool management unit (160).

[0103] Fourth, the reused data mixing step. If train_type is a type that includes reuse, reused data corresponding to reused_hours is received from the retraining pool management unit (160) and combined with new data. The total time of the combined data is adjusted so that it does not exceed train_hours.

[0104] Fifth (S134), the data splitting step. The final selected data is split in the ratio of train (80%):validation (10%):evaluation (10%), and the data_split column for each data set is batch updated. During splitting, speaker_id-based group splitting can be applied to ensure that data for the same speaker is included in both the train and evaluation sets simultaneously.

[0105] Referring further to FIG. 1, the quality evaluation unit (130) is a module that evaluates the quality of each voice data stored in the stt_dataset table (142) and assigns a quality grade. The quality evaluation criteria are determined by a composite condition of CER and audio_score and are classified into four grades: excellent (CER ≤ 0.05, audio_score ≥ 90), good (CER ≤ 0.15, audio_score ≥ 75), fair (CER ≤ 0.30, audio_score ≥ 60), and poor (CER > 0.30 or audio_score < 60).

[0106] The quality evaluation unit (130) also classifies the noise types of voice data. In this embodiment, it is classified into six types: clean, background_noise, echo_reverb, music_overlay, wind_noise, and traffic_noise. Acoustic characteristics such as spectral characteristics (spectral centroid, flatness, bandwidth), energy distribution, and harmonic-percussive ratio are analyzed to calculate a score for each type, and then classified into the type with the highest score. The number of noise types can be expanded depending on the domain to be analyzed, and this information is utilized in the noise stepwise introduction strategy (Type 15).

[0107] Referring again to FIG. 1, the relational database (140) is implemented based on MySQL and is the core data store of the system. It consists of four main tables: a learning strategy table (finetune_plan, 141), a voice data metadata table (stt_dataset, 142), a learning result table (stt_learning_report, 143), and a retraining pool table (stt_retrain_pool, 144). Below, the detailed schemas of the stt_learning_report table (143) and the stt_retrain_pool table (144) will be described.

[0108] The above stt_learning_report table (143) is a table that records the results of each learning session in 77 metrics. The detailed schema consists of 6 groups as shown in the following [Table 3].

[0109] Group / Column Name Number List of major columns ① Performance Indicators 10 cer, wer, val_cer, val_wer, best_cer, best_wer, training_loss, validation_loss, previous_cer, target_cer ② Data Organization 12 sample_count, total_audio_hours, new_data_count, reused_data_count, new_data_ratio, reused_data_ratio, recent_data_ratio, noise_data_ratio, avg_data_usage_count, total_data_hours, new_hours_actual, reused_hours_actual ③ Quality Distribution 8 excellent_ratio, excellent_hours, good_ratio, good_hours, fair_ratio, fair_hours, poor_ratio, poor_hours ④ Learning Settings 8 learning_rate, batch_size, num_train_epochs, gradient_accumulation_steps, warmup_steps, optimizer_type, scheduler_type, model_architecture ⑤ Strategic Information 6 strategy_name, strategy_type, train_type, data_strategy, adaptation_reason, ratio_config_version ⑥ Time / Status 10+ session, round, learn_start, learn_end, actual_training_minutes, status, success_flag, error_message, gpu_memory_used, config_json ⑦ Others 23 seed, model_path, checkpoint_path, total_steps, eval_steps, save_steps, fp16_enabled, max_input_length, feature_extractor, tokenizer_name, language, task, augmentation_applied, noise_augmentation_type, early_stopping_patience, best_model_checkpoint, total_params, trainable_params, frozen_params, disk_usage_gb, created_at, updated_at, remarks

[0110] The above stt_learning_report table (143) automatically inserts one record when each session of learning is completed, and failed sessions are also recorded with status="FAILED" and error_message, allowing for failure tracking.

[0111] The above stt_retrain_pool table (144) is a table that manages reusable data. The detailed schema is as shown in the following [Table 4].

[0112] Column name Type explanation id BIGINT Primary Key (Auto-increment) cycle_id INT Cycle ID (training cycle unit) round_id INT Round ID (Detailed group within the cycle) dataset_id BIGINT foreign key of the stt_dataset table file_path VARCHAR Voice file path audio_duration DECIMAL Audio length (seconds) cer DECIMAL The CER of the data audio_score INT Sound quality score of the data used_flag BOOLEAN Usage complete (True / False) used_in_session INT Session number used inserted_at DATETIME Full registration date

[0113] Referring further to FIG. 1, the fine-tuning execution engine (150) is a core module that performs the actual training of the Whisper model. It is implemented based on the Seq2SeqTrainer of the Hugging Face Transformers library and performs model loading, data preprocessing, training, and checkpoint saving in batches.

[0114] The above model path determination rule is as follows. When the current session is session=N, if a model exists in the path . / models / session_{N-1} / , that model is loaded as the base model; otherwise, the original pre-trained model (e.g., openai / whisper-large-v3) is loaded. After training is complete, the resulting model is saved in the path . / models / session_{N} / and automatically connected as the base model for the next session.

[0115] During the above training, hyperparameters such as learning_rate, num_train_epochs, per_device_train_batch_size, gradient_accumulation_steps, and warmup_steps are passed from the finetune_plan table and set in TrainingArguments. fp16 (half-precision) training is enabled by default to reduce GPU memory usage and improve training speed. Evaluation of the validation data is performed at every eval_step, and if the validation loss does not improve, the training is terminated early after the early_stopping_patience epoch.

[0116] Meanwhile, FIG. 4 illustrates a data equal distribution algorithm based on a retraining pool. The retraining pool management unit (160) manages reused data through the stt_retrain_pool table (144).

[0117] The above equal distribution algorithm is performed in the following four steps (S161~S164).

[0118] In S161, the entire pool data corresponding to cycle_id is grouped by round_id. For example, if there is a pool with cycle_id=1 containing data from a total of 20 rounds from round 1 to 20, it is classified into 20 groups.

[0119] In S162, data is extracted evenly for (target_reused_hours / N_rounds) hours in each round. For example, if target_reused_hours=5.0 and N_rounds=20, 0.25 hours (15 minutes) are extracted in each round. If the available data for a specific round is less than 0.25 hours, the entire data for that round is extracted, and the shortfall is supplemented from other rounds.

[0120] In S163, the used_flag of the extracted data is updated to True, and the used_in_session is updated to the current session number. In S164, the usage_count for the corresponding data in the stt_dataset table is increased by 1. All of these processes are performed within a single database transaction to ensure data consistency.

[0121] Referring further to FIG. 1, the performance evaluation unit (170) is a module that quantitatively evaluates the recognition performance of a fine-tuned model. It calculates CER (Character Error Rate) and WER (Word Error Rate) as evaluation metrics.

[0122] The above CER calculation is performed after applying the following Korean-specific normalization pipeline: (1) whitespace normalization (converting multiple whitespaces into single whitespaces), (2) punctuation removal (periods, commas, question marks, exclamation marks, etc.), (3) number-to-Hangul conversion (optional), (4) special character removal. The edit distance (Levenshtein distance) is calculated for the normalized text, and the CER is calculated by dividing it by the reference text length. The improvement rate is calculated by comparing the original pre-trained model and the fine-tuned model with the same evaluation data.

[0123] The data flow between the components of the system (100) is as follows. When the session orchestrator (180) requests a session lookup from the strategy loader (110), the strategy loader (110) looks up and returns a training strategy from the finetune_plan table (141). The data selector (120) receives the strategy information and selects data that meets quality standards from the stt_dataset table (142). The retraining pool management unit (160) evenly extracts reused data from the stt_retrain_pool table (144) and transmits it to the data selector (120).

[0124] The fine-tuning execution engine (150) performs training using selected data and a loaded model, and stores the trained model in a session-specific path. The performance evaluation unit (170) calculates CER / WER for the trained model and returns the result. Finally, the session orchestrator (180) records the training result in the stt_learning_report table (143), updates the history of the data used, and checks for the existence of the next session to continue the cycle.

[0125] The session orchestrator (180) implements an exclusive lock mechanism using the / tmp / whisper_finetuning.lock file. When training starts, a lock file is created, and if the lock file already exists, it is determined that another process is training, and execution is stopped. When training is completed or terminates with an error, the lock file is deleted to allow for the next execution. To prepare for cases where the lock file remains due to abnormal termination (kill, system reboot, etc.), the process ID (PID) is recorded in the lock file, and if the process with the corresponding PID does not exist, it is determined to be a stale lock and automatically released.

[0127] (Second Embodiment)

[0128] Figure 5 shows the configuration of training strategies by Train Type. The second embodiment describes the detailed configuration of 17 training strategies defined in the finetune_plan table (141). [Table 5] Each training strategy (train_type) has a unique data configuration method, quality criteria, and hyperparameter combination, and is automatically switched per session.

[0129] Ty pe Strategy Name Data configuration explanation 1 Debugging check New only (small quantity) Small-scale training for pipeline verification. Checking if the entire code path is working correctly. 2 Tokenizer experiment New only Tokenizer Optimization Experiment. Verification of the Effect of Adding Korean-Specific Tokens 3 Pipeline verification New only Check the overall data flow (DB → Preprocessing → Training → Evaluation → Recording). 4 Learning rate experiment New only Optimal learning rate search (0.00001~0.0001 range grid search) 5 Tokenizer adjustment New only Type 2 Result-Based Tokenizer Final Adjustment 6 Check parameters New only Searching for the optimal combination of batch size, gradient accumulation, etc. 7 Preparation for transition New only Final verification and baseline establishment before transitioning to full-scale learning 8 Full-scale learning New only Training on full-scale data. Data organization based on target time. 9 Loss monitoring New only Analysis of training / validation loss trends and setting early termination criteria 10 Performance verification New only Check achievement rate against CER / WER targets and reset standards 11 Gradual data enhancement New + Reusable (per round) Search for the optimal ratio by gradually increasing the reused_percentage in each iteration 12 Data Optimization New + Reuse (Equal) Application of equal distribution algorithm. Data is reused evenly across all rounds. 13 High-quality refinement New only (CER≤0.05) Select only excellent-grade data for intensive fine-tuning 14 High-end tuning New + Various Samples Diversity reinforcement learning. Intentionally mixing quality distributions 15 Noise enhancement New + Superior Performance + Noise 4-stage noise introduction. Enhanced real-world robustness. 16 Equal distribution reuse New + Reuse (per round) Reuse with even distribution per round. Prevents bias. 17 Advanced Strategy New + Reusable (Variable) Application of special strategies per round. Mix of early and recent rounds, priority of higher rounds, etc.

[0130] The above 17 strategies are broadly classified into three layers. Layer 1 (Type 1-10) is the experimental stage, where pipeline validation, hyperparameter exploration, and baseline establishment are performed using only new data. Layer 2 (Type 11-14) is the full-scale learning stage, where various data composition strategies, such as mixing reused data and high-quality refinement, are applied. Layer 3 (Type 15-17) is the advanced strategy stage, where sophisticated learning techniques, such as noise augmentation, evenly distributed reuse, and special strategies for each iteration, are applied.

[0131] A detailed explanation of the key strategies is as follows.

[0132] Type 11 (Incremental Data Enhancement): This strategy starts the proportion of reused data at 10% in the initial session and gradually increases it to 50%. The reuse rate for each session is determined by the reused_percentage column, and generalization performance is enhanced by gradually mixing in previous training data after the model has sufficiently adapted to new data patterns. Experimental results showed that optimal CER improvement was observed at a reuse rate of 30–40%.

[0133] Type 13 (High-Quality Refining): This strategy trains by selecting only excellent-grade data with a CER ≤ 0.05 and an audio_score ≥ 90. Although the amount of data is small (about 15% of the total), its quality is extremely high, making it effective for fine-tuning the model's precision in the final stage. By setting the learning rate to a very low 0.000005, it seeks subtle performance improvements while maintaining existing training results.

[0134] Type 15 (Noise Augmentation): This strategy mixes new data (60%), top-performing data with the lowest CER from previous training (20%), and noise data (20%). Noise data is added for the time specified in the noise_hours column, and the noise stage (stage_1–stage_4) is specified in the data_strategy column. This progressively strengthens robustness in noisy environments while maintaining the model's clean speech recognition performance.

[0135] Type 16 (Evenly distributed reuse): This strategy utilizes the extract_reused_data_evenly algorithm of the retraining pool management unit (160) to evenly reuse data from all previous rounds. This enables balanced learning that is not biased toward data at any given time point and is particularly effective in preventing overfitting during long-term learning. Experimental results showed an average CER of 0.0557 and a minimum CER of 0.0075 over 203 sessions.

[0136] Type 17 (Advanced Strategy): This strategy applies a specialized data composition for each round. For example, in a specific round, data from the last 2-3 rounds is limited to less than 30%, in another round, 20% of the total accumulated data is randomly extracted, and in yet another round, data from the top 40% of rounds is extracted based on the weighted sum of the average CER and sound quality score per round (audio_score × 0.6 + (1 - CER) × 0.4).

[0137] Since all of the above strategies are defined as records in the finetune_plan table (141), when adding a new strategy, you just need to add a record to the database and add an execution function to the TrainTypeExecutor module. You can disable the corresponding strategy by setting the use_yn column to 'N', and you can freely rearrange the execution order by changing the session order.

[0139] (Third Embodiment)

[0140] The third embodiment describes the detailed configuration and utilization method of the 77 metrics recorded in the stt_learning_report table (143). At the completion of each learning session, the 77 metrics are recorded in the database with a single INSERT query, which enables SQL-based multidimensional learning history analysis.

[0141] First, performance metrics (10)

[0142] Performance metrics quantify the recognition accuracy of the trained model. cer (Character Error Rate) represents the final evaluation CER, wer (Word Error Rate) represents the final evaluation WER, val_cer represents the validation data CER, val_wer represents the validation data WER, best_cer represents the lowest CER achieved during training, best_wer represents the lowest WER achieved during training, training_loss represents the final training loss, validation_loss represents the final validation loss, previous_cer represents the CER of the previous session (for comparison between sessions), and target_cer represents the target CER.

[0143] Second, data composition metrics (12)

[0144] Data composition metrics record the quantitative composition of the data used in the session. They include sample_count (total number of data items), total_audio_hours (total audio time), new_data_count (number of new data items), reused_data_count (number of reused data items), new_data_ratio (ratio of new data), reused_data_ratio (ratio of reused data), recent_data_ratio (ratio of recent data), noise_data_ratio (ratio of noise data), avg_data_usage_count (average number of data uses), total_data_hours (total data time), new_hours_actual (actual new time), and reused_hours_actual (actual reused time).

[0145] Third, quality distribution metrics (8)

[0146] Quality distribution metrics record the composition of the data used by quality grade. They include excellent_ratio, excellent_hours, good_ratio, good_hours, fair_ratio, fair_hours, poor_ratio, and poor_hours. This information allows for the analysis of the correlation between CER and data quality distribution.

[0147] Fourth, learning setting metrics (8)

[0148] The training settings metric records the hyperparameters applied to the session. It includes learning_rate, batch_size, num_train_epochs, gradient_accumulation_steps, warmup_steps, optimizer_type, scheduler_type, and model_architecture.

[0149] Fifth, strategic information metrics (6)

[0150] The strategy information metric records meta-information of the training strategy applied to the session. It includes strategy_name, strategy_type, train_type, data_strategy, adaptation_reason, and ratio_config_version.

[0151] Sixth, usage query examples

[0152] The 77 metrics mentioned above are utilized for various analyses through SQL queries. Below, examples of representative usage queries are explained.

[0153] Query 1 - Analysis of CER trends by session: With the query "SELECT session, cer, previous_cer, new_data_ratio, reused_data_ratio FROM stt_learning_report ORDER BY session", you can grasp the changes in CER by session and the data composition ratio at a glance.

[0154] Query 2 - Deriving the optimal strategy: The optimal strategy can be derived by comparing the average CER and minimum CER for each strategy using the query "SELECT train_type, strategy_name, AVG(cer) as avg_cer, MIN(cer) as min_cer, COUNT(*) as session_count FROM stt_learning_report GROUP BY train_type, strategy_name ORDER BY avg_cer".

[0155] Query 3 - Analysis of the cause of performance degradation: You can identify sessions with increased CER (performance degradation) using the query "SELECT session, cer, previous_cer, (cer - previous_cer) as cer_change, new_data_ratio, excellent_ratio, learning_rate FROM stt_learning_report WHERE cer > previous_cer ORDER BY cer_change DESC" and determine the cause by analyzing the data configuration and learning settings of the sessions.

[0156] Query 4 - Correlation between Quality Distribution and Performance: The correlation between the ratio of excellent / good grade data and CER can be analyzed using the query "SELECT excellent_ratio, good_ratio, AVG(cer) as avg_cer FROM stt_learning_report GROUP BY ROUND(excellent_ratio, 1), ROUND(good_ratio, 1) ORDER BY avg_cer".

[0157] Query 5 - Performance statistics by learning rate: The optimal learning rate can be derived by comparing the average CER and minimum CER by learning rate using the query "SELECT learning_rate, COUNT(*) as cnt, AVG(cer) as avg_cer, MIN(cer) as min_cer, AVG(training_loss) as avg_loss FROM stt_learning_report GROUP BY learning_rate ORDER BY avg_cer".

[0158] Query 6 - Reuse Ratio and Overfitting Analysis: You can identify overfitting intervals by comparing CER and loss values ​​for each reuse ratio interval using the query "SELECT ROUND(reused_data_ratio, 1) as ratio_group, AVG(cer) as avg_cer, AVG(validation_loss) as avg_val_loss, AVG(training_loss) as avg_train_loss FROM stt_learning_report GROUP BY ratio_group ORDER BY ratio_group".

[0159] Query 7 - Calculation of improvement rate per session: By using the query "SELECT session, cer, previous_cer, ROUND((previous_cer - cer) / previous_cer * 100, 2) as improvement_pct FROM stt_learning_report WHERE previous_cer > 0 ORDER BY improvement_pct DESC LIMIT 20", you can retrieve the 20 sessions with the highest improvement rates and analyze the strategies and data composition of those sessions.

[0160] Seventh, metric-based automatic notifications

[0161] Metrics in the stt_learning_report table can be integrated with an automated notification system to immediately notify operators when specific conditions occur. For example, notifications are sent if the CER increases by more than 10% compared to the previous session, if the validation_loss exceeds twice the training_loss, or if training terminates abnormally (status=FAILED). This enables immediate response to abnormal situations even during long-term unattended training.

[0162] Eighth, metric time series analysis

[0163] The 77 metrics form time-series data with the session column as the time axis. By applying a moving average to the CER time series, short-term fluctuations can be smoothed and long-term trends identified, and the onset of overfitting can be detected early by analyzing the gap trend between training_loss and validation_loss. This time-series analysis can be performed directly within the database using SQL window functions.

[0165] (Fourth Example)

[0166] The fourth embodiment describes the experimental results obtained by applying the system of the present invention to actual call center voice data. The following experimental results quantitatively verify the effects of the present invention.

[0167] Experimental environment

[0168] The base models used in the experiment were two models, OpenAI Whisper-large-v3 and Whisper-small, and training was performed in an NVIDIA GPU environment. The training data consisted of call center voice data, with a total of 15,178 hours (approximately 10.57 million records) registered in the stt_dataset table. Whisper-large-v3 underwent 1,107 training sessions over Rounds 1-239, and Whisper-small underwent 1,264 training sessions over Rounds 1-377. Additionally, experiments were conducted on Whisper-medium (93 sessions) and Whisper-large-v3-turbo (22 sessions) to verify the universality of the present invention.

[0169] Experiment 1: Whisper-large-v3 model training results

[0170] The experimental results for the Whisper-large-v3 model are as follows. Training was performed for 705 sessions in Rounds 1-50, achieving an average CER of 0.0366. In Rounds 51-100, an average CER of 0.0360 was maintained with 167 sessions, and in Rounds 101-200, the average CER improved to 0.0333 with 195 sessions. Finally, in Rounds 201-239, an average CER of 0.0249 was achieved with 40 sessions, recording a 32% performance improvement (0.0366 → 0.0249) over the entire training period. The minimum CER for the entire period was 0.0075, achieved in the Type 16 (Equal Distribution Reuse) strategy.

[0171] Experiment 2: Whisper-small model training results

[0172] The experimental results for the Whisper-small model are as follows. Training was performed for 584 sessions in Round 1-50, recording an average CER of 0.1594. In Round 51-100, an average CER of 0.0539 was achieved with 101 sessions; in Round 101-200, an average CER of 0.0612 was achieved with 199 sessions; and in Round 201-239, an average CER of 0.0500 was achieved with 78 sessions. This corresponds to a 69% performance improvement (0.1594 → 0.0500) compared to the initial state. Despite the model size of Whisper-small being approximately 1 / 10 that of large-v3, the system of the present invention achieved a level of recognition accuracy that is practical for use in the call center domain.

[0173] Of particular note is that the Whisper-small model continued to train even after Round 240, maintaining stable performance in the Round 240-300 (174 sessions, avg CER 0.0687) and Round 301+ (128 sessions, avg CER 0.0832) ranges. This demonstrates that the usage_count-based overfitting prevention and retraining pool equal distribution algorithm of the present invention works effectively even in small models.

[0174] Experiment 3: Performance Comparison by Model

[0175] The results of applying the system of the present invention to four Whisper models are compared as shown in [Table 6] below. Through this, the model scale-independent universality of the present invention was confirmed.

[0176] model Number of sessions Average CER Minimum CER Round note Whisper-large-v3 1,107 0.0355 0.0075 1-239 Achieve minimum CER Whisper-small 1,264 0.1179 0.0001 1-377 Maximum improvement rate 69% Whisper-medium 93 0.0581 0.0409 1-41 Medium-scale verification Whisper-large-v3-turbo 22 0.0942 0.0304 1-23 Lightweight large model

[0177] The following points are noteworthy from the experimental results above. First, Whisper-large-v3 achieved an average CER of 0.0355 and a minimum CER of 0.0075 over 1,107 sessions, demonstrating the highest performance in absolute recognition accuracy. Second, Whisper-small achieved a 69% improvement in CER over 1,264 sessions despite having a model size approximately 1 / 10 that of large-v3, confirming that the automation of the learning strategy of the present invention brings about greater performance improvement in small models. Third, training was performed on the same system for the Whisper-medium and large-v3-turbo models by changing only the model settings in the finetune_plan table without modifying the training code, thereby verifying the model-independent design of the present invention.

[0178] Experiment 4: Performance Comparison by Strategy (Based on Whisper-large-v3)

[0179] The results of comparing the performance of the major learning strategies applied to the Whisper-large-v3 model are shown in [Table 7].

[0180] Strategy Average CER Minimum CER Number of sessions note Type 16 (Equal distribution reuse) 0.0557 0.0075 203 Achieve minimum CER Type 17 (Advanced Strategy) 0.0492 0.0216 193 Lowest average CER Integrated Optimization (Mixed) 0.0571 0.0178 468 Multi-strategy blend Specialized learning (domain) 0.0612 0.0154 464 Domain specialization

[0181] In the above results, the Type 16 (Equal Distribution Reuse) strategy achieved a minimum CER of 0.0075, recording the lowest CER. This demonstrates that the equal distribution algorithm of the retraining pool management unit (160) is effective in preventing overfitting while maximizing model performance. The Type 17 (Advanced Strategy) showed the lowest average among the strategies with an average CER of 0.0492, confirming that the specialized data configuration for each iteration contributes to the stability of long-term learning.

[0182] Experiment 5: Comparison of CER Trends by Model

[0183] The CER trends for each round interval of the two models, Whisper-large-v3 and Whisper-small, are compared as shown in the following [Table 8].

[0184] panel Large-v3 CER Small CER Number of sessions note Round 1-50 0.0366 0.1594 L:705 S:584 Early learning Round 51-100 0.0360 0.0539 L:167 S:101 Small rapid improvement Round 101-200 0.0333 0.0612 L:195 S:199 Entering the stabilization phase Round 201-239 0.0249 0.0500 L:40 S:78 Achieving optimal performance Round 240-300 - 0.0687 S:174 Small standalone learning Round 301-377 - 0.0832 S:128 Long-term stability verification Overall improvement rate 32% improvement 69% improvement Total 2,488 Proven Model-Independent Effect

[0185] The following points are noteworthy from the above experimental results.

[0186] First, although Whisper-large-v3 already had excellent pre-training quality and a low initial CER of 0.0366, the system of the present invention achieved an additional 32% improvement to a CER of 0.0249 in Round 201-239. This implies that there is room for domain-specific fine-tuning even in high-performance models, and that the systematic approach of the present invention effectively brings out that potential.

[0187] Second, Whisper-small achieved a 69% improvement, increasing its CER from 0.1594 in Round 1-50 to 0.0500 in Round 201-239 despite having a model size of about 1 / 10, confirming that the automation of the learning strategy of the present invention brings about a greater relative improvement effect in small models. This suggests that the system of the present invention can provide effective performance improvement even in environments where computing resources are limited.

[0188] Third, Whisper-small continued training independently until Round 377 even after Round 240, verifying the long-term stability of the system over a total of 377 rounds. Although the average CER increased slightly to 0.0832 after Round 301, this was analyzed as natural convergence due to training data saturation rather than overfitting, and it remained stable without a sharp increase in validation_loss.

[0189] Fourth, the model-independent architecture of the present invention was verified by the fact that training was performed on all four models (large-v3, small, medium, large-v3-turbo) solely through the settings of the finetune_plan table without changing the training code. It demonstrated scalability, where only a train_type record and an execution function need to be added when adding a new model.

[0190] The above experimental results quantitatively demonstrate that the database-based adaptive fine-tuning system of the present invention possesses the following technical advantages over conventional technology.

[0191] First, achieving a CER of 0.0075 (99.25% recognition accuracy) in Whisper-large-v3 is a commercially viable level in the call center domain and is capable of providing sufficient accuracy without post-processing in real-time call transcription systems. Whisper-small also achieved a CER of 0.0500 (95% recognition accuracy), demonstrating its applicability in lightweight deployment environments.

[0192] Second, the fact that no overfitting occurred during 377 rounds of continuous training means that the usage_count maximum value limit (4 times in this embodiment) and the round-by-round equal distribution algorithm effectively prevent data bias. In conventional technology, performance degradation due to overfitting was common during continuous training of 100 rounds or more, but the present invention structurally prevents this.

[0193] Third, achieving consistent performance improvement with the same system regardless of model size implies that the database-based strategy management and data selection algorithms of the present invention are a universal framework not dependent on a specific model architecture. This suggests scalability that allows for the same applicability to other STT models, such as Conformer and wav2vec, in the future.

[0194] Fourth, the automatic switching of 17 learning strategies dramatically improved the speed of strategy testing. In the conventional method, code modification and manual re-execution were required for every strategy change, taking an average of half a day to test one strategy; however, in the present invention, strategy switching is possible within minutes with only DB record changes.

[0195] In addition, the analysis of data quality grades according to the present invention analyzed the distribution of data quality grades used in 377 rounds of training, and found that excellent grade data accounted for approximately 18% of the total, good grade for approximately 42%, fair grade for approximately 31%, and poor grade for approximately 9%. In sessions where a CER of 0.05 or less was achieved, the proportion of excellent grade data was found to be over 25% on average, confirming that the proportion of high-quality data has a significant impact on performance.

[0196] In addition, in the training session (Type 15) containing noisy data, a pattern was observed in which the CER temporarily increased initially but recovered quickly in subsequent sessions to achieve an even lower CER. This suggests that the stepwise introduction of noise has a regularization effect that improves the model's generalization performance.

[0197] In addition, regarding the correlation between the learning rate and performance of the present invention, an analysis of the CER distribution by learning_rate revealed that the lowest average CER was achieved at a learning rate of 0.00001. At a learning rate of 0.00003, the initial convergence speed was fast but the final CER was somewhat high, while at a learning rate of 0.000005, convergence was slow, resulting in a small improvement in performance within the same epoch. These performance characteristics by learning rate were derived by analyzing the learning_rate and cer columns of the stt_learning_report table using SQL queries.

[0198] In addition, regarding the relationship between the reuse ratio and overfitting of the present invention, an analysis of the relationship between the reused_data_ratio and validation_loss revealed that when the reuse ratio exceeded 50%, the validation_loss tended to increase. On the other hand, in the reuse ratio range of 20–40%, both training_loss and validation_loss decreased steadily, confirming that this range represents the optimal balance point between overfitting and underfitting. This analysis was performed using the reused_data_ratio, training_loss, and validation_loss columns of the stt_learning_report table.

[0199] In addition, to verify the cumulative learning effect of session chaining by automatic session chaining of the present invention, independent learning (based on a single session, original model) and chaining learning (based on a previous session model) were compared using the same data. The average CER of independent learning was 0.0821, while the average CER of chaining learning was 0.0548, showing that chaining learning performed 33.2% better. This demonstrates that the accumulation of learning results between sessions contributes significantly to improving model performance.

[0200] In addition, the operational stability of the system of the present invention was such that there were zero system failures during a continuous learning period of 377 rounds. The lock file-based duplicate execution prevention mechanism successfully blocked three concurrent execution attempts. Two database transaction rollbacks occurred due to temporary network disconnection, but were successfully recovered through automatic retries. Model checkpoints were successfully saved per session, allowing for responses without data loss even when rollback to a specific session and retries were required.

[0201] The GPU memory usage of the present invention was stable at an average of 22GB, and the training time was an average of 2.5 hours per session. The cumulative training time for a total of 377 sessions was approximately 943 hours (39 days), but the time between sessions (data selection, model loading, etc.) averaged 3 minutes per session, which corresponded to less than 2% of the total training time, confirming that the system overhead was extremely low.

[0203] (Fifth Example)

[0204] The fifth embodiment illustrates the scalability of the system of the present invention, enabling it to be applied to various environments and models.

[0205] First, support for various speech recognition models

[0206] The system of the present invention can be applied to various speech recognition models, such as Conformer, wav2vec 2.0, HuBERT, and Branchformer, in addition to the Whisper model. When changing the model, only the model load / save interface of the fine-tuning execution engine (150) needs to be replaced, and the remaining components, such as the strategy loader (110), data selector (120), retraining pool management unit (160), and performance evaluation unit (170), operate independently of the model. It is also possible to compare performance between models by recording the model name used in the model_architecture column of the stt_learning_report table.

[0207] Second, multilingual support

[0208] The system of the present invention can be applied to fine-tuning multilingual speech recognition models, including not only Korean but also English, Japanese, and Chinese. The normalization pipeline of the performance evaluation unit (170) can be modified to suit the target language, and the CER / WER calculation logic is independent of language. A language-specific learning strategy can be managed by adding a language column to the finetune_plan table.

[0209] Third, distributed learning environment

[0210] The system of the present invention can be extended to a multi-GPU or distributed learning environment in addition to a single GPU environment. A distributed learning library (DeepSpeed, FSDP, etc.) can be integrated into the fine-tuning execution engine (150), and distributed learning settings can be managed in the DB by adding columns such as gpu_count and distributed_strategy to the learning strategy table. The session orchestrator (180) can also execute multiple experiments in parallel by automatically managing GPU resource allocation.

[0211] Fourth, real-time learning monitoring

[0212] The stt_learning_report table (143) can be linked to a real-time dashboard. The progress of recent training sessions can be displayed in real time by periodically executing the query "SELECT session, cer, training_loss, validation_loss, status FROM stt_learning_report ORDER BY session DESC LIMIT 10" in a web-based monitoring system. When a performance drop (cer > previous_cer) is detected, an alert can be sent to enable immediate response.

[0213] Fifth, automatic strategy recommendation

[0214] It is possible to extend the system by analyzing the accumulated training history data in the stt_learning_report table to automatically recommend the next training strategy that is optimal for the current model state. For example, a rule-based recommendation system can be implemented that recommends switching the training strategy to Type 13 (High-Quality Refining) if the CER reduction rate of the last 5 sessions slows down, and recommends reducing the learning rate if the validation_loss is on an upward trend. Furthermore, meta-learning-based automatic strategy optimization using accumulated metric data is also possible.

[0215] Sixth, A / B testing support

[0216] Automated A / B testing is possible by registering multiple records with identical data configurations but different hyperparameters in the `finetune_plan` table. For example, two strategies with `learning_rate=0.00001` and `learning_rate=0.00003` can be executed sequentially on the same data, and the optimal learning rate can be determined by comparing the CERs of the two sessions in the `stt_learning_report` table. This process is fully automated, enabling systematic hyperparameter exploration without human intervention. Industrial applicability

[0218] The technical concept of the adaptive fine-tuning system for a database-based speech recognition model and the control method thereof according to the present invention is capable of repeatedly producing identical results in practice, and in particular, by implementing such an invention, it can promote technological advancement and contribute to industrial development, thus having sufficient value for protection.

[0219] No content Explanation of the symbols

[0220] <Explanation of symbols for major parts of the drawing> 100; Speech recognition model adaptive fine-tuning system 110; Plan Loader 120; Data Selector 130; Quality Evaluator 140; Relational database (MySQL) 141; finetune_plan table - Learning strategy table (23 columns) 142; stt_dataset table - Voice data metadata table (10.57 million records) 143; stt_learning_report table - Training results table (77 metrics) 144; stt_retrain_pool table - Retraining pool table 150; Finetuning Execution Engine 160; Retrain Pool Manager 170; Performance Evaluation Department (CER / WER Evaluator) 180; Session Orchestrator

Claims

Claim 1 The present invention relates to an adaptive fine-tuning system (100) for a database-based speech recognition model, comprising: a strategy loader (110) that queries a learning strategy from a learning strategy table of a relational database; a data selector (120) that automatically selects training data from a voice data metadata table according to quality criteria and usage limit defined in the learning strategy, and performs time-based quantitative selection by accumulating the actual voice time of each data and extracting it until a target time is reached; a quality evaluation unit (130) that evaluates the quality of voice data and assigns a quality grade; a relational database (140) that stores learning strategies, voice data metadata, learning results, and retraining pool information; a fine-tuning execution engine (150) that performs fine-tuning of the speech recognition model using the selected data and stores the trained model in a session-specific path; a retraining pool management unit (160) that evenly distributes reused data by round from a retraining pool table; and a performance evaluation unit (170) that calculates the CER and WER of the fine-tuned model. An adaptive fine-tuning system for a database-based speech recognition model, characterized by including: a session orchestrator (180) that sequentially executes the above strategy loader, data selector, fine-tuning execution engine, retraining pool management unit, and performance evaluation unit on a session basis, and manages automatic model chaining between sessions. Claim 2 An adaptive fine-tuning system for a database-based speech recognition model according to claim 1, wherein the strategy loader (110) supports 17 or more types of learning strategies, including experimental phase strategies (Type 1-10), full-scale learning strategies (Type 11-14), and advanced strategies (Type 15-17). Claim 3 An adaptive fine-tuning system for a database-based speech recognition model according to claim 1, wherein the relational database (140) includes: a finetune_plan table (141) that defines a learning strategy in 23 columns; a stt_dataset table (142) that stores quality metrics and usage history of speech data; a stt_learning_report table (143) that records the results of each learning session in 77 metrics; and a stt_retrain_pool table (144) that manages reusable data in cycle / round units. Claim 4 The method of control using an adaptive fine-tuning system (100) of a database-based speech recognition model according to claim 1 comprises: (a) a learning strategy loading step (S110) for querying information on a learning strategy type, session number, new data usage, reused data usage, noise data usage, learning rate, number of epochs, maximum allowable CER, and minimum voice quality score from a learning strategy table of a relational database; (b) a data automatic selection step (S120) for filtering data from a voice data metadata table of the database in which, according to a quality standard defined in the learning strategy, the CER is less than or equal to the maximum allowable CER and the voice quality score is greater than or equal to the minimum voice quality score, selecting only data for which the number of uses is less than a preset maximum value, and extracting data until a target time is reached by accumulating the actual voice time of each data; (c) a reused data equal distribution step (S125) for extracting reused data stored in a retraining pool table in round units in case the type of the learning strategy includes reused data, in each round; and (d) a data splitting step for dividing the entire selected learning data into training, verification, and evaluation data. Step (S130); (e) Model automatic loading step (S140) which loads the model as the base model if a model saved in the previous session exists, and if it does not exist, loads the original pre-trained model; (f) Fine-tuning execution step (S150) which performs fine-tuning with hyperparameters defined in the learning strategy and saves the trained model to a session-specific path; (g) Performance evaluation step (S160) which calculates the Character Error Rate (CER) and Word Error Rate (WER) of the fine-tuned model; (h) Learning result recording step (S170) which automatically records 77 metrics in a learning result table; (i) Usage history update step (S180) which updates the usage count of each data used in training; and (j) Continuous execution determination step (S190) which returns to step (a) and automatically executes continuously if a next session exists.A control method for an adaptive fine-tuning system of a database-based speech recognition model characterized by including Claim 5 A control method for an adaptive fine-tuning system of a database-based speech recognition model according to claim 4, wherein the (b) automatic data selection step (S120) is characterized by forcing the reusable data usage to 0 for learning strategy types 1 to 10 to use only new data, selecting only high-quality data with a CER of 0.05 or less for learning strategy type 13, and using a complex mixture of new data, high-performance data, and noise data for learning strategy type 15. Claim 6 A control method for an adaptive fine-tuning system of a database-based speech recognition model according to claim 4, wherein the (b) data automatic selection step (S120) applies a hybrid strategy in which the larger value between the percentage-based time and the step-by-step minimum guaranteed time is determined as the target time, and the step-by-step minimum guaranteed time is 3 hours for the 1st to 10th rounds, 5 hours for the 11th to 20th rounds, and 10 hours for the 21st round or more. Claim 7 A control method for an adaptive fine-tuning system of a database-based speech recognition model according to claim 4, wherein the (c) reusable data equal distribution step (S125) includes a fallback logic for extracting data for each round by dividing the target reuse time by N for N rounds stored in a retraining pool table, and for supplementing data extraction from another round when there is a shortage of available data in a specific round, and updating a flag indicating the completion of use of the extracted data. Claim 8 The control method of the adaptive fine-tuning system of the database-based speech recognition model according to claim 4 further comprises a multi-stage strategy of introducing noise data in stages, wherein in the initial stage only clean speech data without noise is used, and in subsequent stages noise types such as background noise, echo / reverb, wind sound, music overlap, and traffic noise are gradually added to strengthen the real-environment robustness of the model in stages.

Images