Adaptive Inference Method and System by Dynamic Switching of sLLM

Abstract

The present invention relates to an adaptive inference method by dynamic switching of a small language model, wherein the method comprises: a first step of receiving user input; a second step of preprocessing the input received in the first step; a third step of selecting a domain based on the result of the preprocessing in the second step; a fourth step of determining whether the domain selected in the third step matches the current small language model; a fifth step of selecting another small language model if it is determined in the fourth step that it does not match the current small language model; a sixth step of loading the small language model selected in the fifth step; and a seventh step of delivering context to the small language model selected in the sixth step.

Description

Technology Field

[0001] The present invention relates to an adaptive inference method and system that dynamically switches between multiple domain-specific small language models (sLLMs) to rapidly respond to user requests and provide accurate responses even when the query is changed to one belonging to a different domain during the user query process. Background Technology

[0003] Recently, large language models such as GPT-4, Claude, LLaMA, and Gemini have demonstrated outstanding performance in various natural language processing tasks, but these large language models are revealing several serious problems in actual service operation environments.

[0004] First, large language models have tens of billions to trillions of parameters, requiring massive computational resources and memory. For example, GPT-4 requires tens of gigabytes of GPU memory during the inference process, which is difficult for small and medium-sized service providers or individual users to handle.

[0005] Second, in terms of cost-efficiency, large language models incur excessive computational costs even for simple domain-specific tasks. For instance, since all parameters of the entire large language model are activated for tasks such as simple symptom inquiries in the medical field or basic legal interpretation, there is a problem of inefficient resource usage relative to the actual amount of computation required.

[0006] Third, there is the issue of response delay. The inference process of large language models takes a significant amount of time due to complex attention mechanisms and multi-layered transformer structures, which can impair the user experience in services requiring real-time responses.

[0007] Fourth, there are serious issues regarding environmental costs and energy consumption. The operation of large language models entails high power consumption, which is not environmentally friendly and leads to increased long-term operating costs.

[0008] Due to the problems associated with these large language models, small language models specialized for specific domains are being introduced and are actually being used; however, while conventional small language models partially resolve the issues of the aforementioned large language models, they show limitations, particularly in terms of multitasking.

[0009] Small language models are generally limited to fewer than 7 billion parameters, which limits their ability to effectively represent and process knowledge from multiple domains simultaneously. Consequently, performance degrades rapidly when user queries cross multiple domains.

[0010] For example, when a query related to medicine changes to the legal field, it is difficult to achieve satisfactory performance with a single small language model.

[0011] Furthermore, general small language models that are not specialized for a specific domain show significantly lower performance compared to large language models in tasks requiring expertise.

[0012] Conventional adaptive technologies have the following limitations.

[0013] Limitations of the LoRA (Low-Rank Adaptation) method

[0014] LoRA is a parameter-efficient fine-tuning technique proposed by Microsoft in 2021 that fixes existing weights and adds an adapter composed of the product of two low-dimensional matrices.

[0015] This approach offers the advantages of memory efficiency and fast learning speed, but it is fundamentally constrained by the representational capabilities of a single base model. Furthermore, due to low-dimensional ranks, it has limitations in adapting to complex domains, and because the architecture or fundamental characteristics of the base model cannot be changed, it is difficult to reflect fundamental differences between domains.

[0016] Limitations of the Adapter method

[0017] The Adapter method is a transfer learning technique proposed by Google in 2019 that inserts a small feed-forward network between existing Transformer layers. It features a structure of LayerNorm → Down-projection → Non-linearity → Up-projection → Residual, offering the advantage of rapid task-specific adaptation while preserving existing model weights. However, this method cannot alter the fundamental characteristics of the base model, and there are limitations in expressiveness due to size constraints on the adapter. Furthermore, interference issues arise when processing multiple domains simultaneously, and adding new domains can affect the overall model.

[0018] Limitations of the MoE (Mixture of Experts) method

[0019] MoE is a method that scales model capacity through conditional computation by activating only some expert networks based on the input. However, this approach has a structural limitation in that it consists of experts based on the same architecture, making it impossible to mix different architectures optimized for specific domains (e.g., BERT-based question-answering models, GPT-based generative models, T5-based summarization models, etc.). Additionally, since all experts must reside in memory, the overall model size increases, resulting in memory overhead. Furthermore, there are limitations such as load balancing issues caused by the training instability of routing networks, knowledge interference between experts, and catastrophic forgetting phenomena. Moreover, because the expert composition is fixed at the time of training, the flexibility to dynamically add or remove new domain-specific models during service operation is restricted.

[0021] [Prior Art Literature]

[0022] Korean Patent No. 10-2349402 (Published January 11, 2022)

[0023] Korean Patent Publication No. 10-2025-0079453 (Published June 4, 2025) The problem to be solved

[0025] The present invention aims to overcome the limitations of the aforementioned prior art by providing a method for dynamic switching between small language models. means of solving the problem

[0027] The present invention relates to an adaptive inference method by dynamic switching of a small language model, wherein the method comprises: a first step of receiving user input; a second step of preprocessing the input received in the first step; a third step of selecting a domain based on the result of the preprocessing in the second step; a fourth step of determining whether the domain selected in the third step matches the current small language model; a fifth step of selecting another small language model if it is determined in the fourth step that it does not match the current small language model; a sixth step of loading the small language model selected in the fifth step; and a seventh step of delivering context to the small language model selected in the sixth step.

[0028] The adaptive inference method according to the present invention may further include a step 5-1 of determining whether a small language model selected in step 5 exists in a cache; and a step 5-2 of loading the selected small language model from the cache if it is determined in step 5-1 that it exists in the cache, and loading the selected small language model from the disk if it is not.

[0029] The adaptive inference method according to the present invention may further include a 5-3 step of unloading the current small language model when another small language model is selected and loaded in the 5th step.

[0030] The adaptive inference method according to the present invention may further include an eighth step of predicting a subsequent small language model; and a ninth step of pre-loading the subsequent small language model predicted in the eighth step in the background.

[0031] The third step may include: a third-1 step of extracting domain-specific keywords included in the input preprocessing result and selecting candidate domains; a third-2 step of performing a similarity comparison between the input text and the domain template; a third-3 step of calculating a confidence score based on the keyword score and the similarity comparison score; and a third-4 step of selecting a domain based on the confidence score from the third-3 step.

[0032] Step 5 may include Step 5-4, which extracts a list of small language models corresponding to the selected domain; Step 5-5, which calculates a score based on past performance history and similarity with the current context; Step 5-6, which checks resource constraints and extracts loadable small language models; and Step 5-7, which selects a final small language model considering performance scores and resource efficiency.

[0033] Step 8 may include Step 8-1, which extracts a domain switching pattern from a recent predetermined number of conversations; Step 8-2, which constructs a probability matrix of switching from the current domain to another domain; Step 8-3, which selects a domain whose switching probability is greater than or equal to a predetermined threshold; and Step 8-4, which predicts the next small language model selected according to a predetermined priority among the small language models corresponding to the domain selected in Step 8-3.

[0034] The adaptive reasoning method according to the present invention can be executed by a computer program recorded on a computer-readable recording medium, and the present invention also includes within the scope of rights a computer-readable recording medium on which such a computer program is recorded.

[0035] The present invention also provides an adaptive inference system that dynamically switches between a plurality of small language models, wherein the adaptive inference system includes a domain classification module; a model selection engine; a context bridge; and a model cache manager.

[0036] The domain classification module preprocesses user input and selects a domain based on the preprocessing results; the model selection engine determines whether the selected domain matches the current small language model and, if it is determined that it does not match the current small language model, executes the selection of another small language model; the context bridge delivers the context to the selected small language model; and the model cache manager loads the selected small language model into memory and unloads the existing small language model. Effects of the invention

[0038] According to the present invention, high performance at the level of a specialized language model can be provided in each domain through domain-specific optimization, and in particular, significantly faster response speeds compared to large language models are achieved by utilizing the fast inference speed of small language models. Therefore, high accuracy and expertise in the corresponding domain can be guaranteed through domain-specific learning.

[0039] According to the present invention, memory usage can be significantly reduced by selectively loading only the necessary small language models, thereby enabling the reduction of computational costs and the implementation of an environmentally friendly artificial intelligence system. Brief explanation of the drawing

[0041] FIG. 1 is a diagram illustrating an example of an environment in which an adaptive inference method according to the present invention is executed. Figure 2 is a block diagram of a domain classification model. Figure 3 is a block diagram of a model selection engine. Figure 4 is a block diagram of a context bridge. Figure 5 is a block diagram of the model cache manager. FIG. 6 is a flowchart of an adaptive inference method according to the present invention. FIG. 7 is a flowchart of the domain detection method of the present invention. FIG. 8 is a flowchart of the model selection method of the present invention. FIG. 9 is a flowchart of the predictive preloading method of the present invention. Specific details for implementing the invention

[0042] The present invention will be described below with reference to the attached drawings.

[0043] In this specification, expressions such as "transmission (transfer) from A to B" or "A receiving from B" include transmission (transfer) or reception involving an intermediate medium, and do not refer only to direct transmission (transfer) or reception from A to B. In the description of the invention, the order of each step should be understood as non-restrictive, unless the preceding step must logically and temporally be performed prior to the subsequent step. That is, except for such exceptional cases, the essence of the invention is not affected even if the process described as the subsequent step is performed prior to the process described as the preceding step, and the scope of rights should be defined regardless of the order of the steps.

[0044] In this specification, “A or B” is defined to mean not only selectively referring to either A or B, but also including both A and B. Furthermore, in this specification, the term “include” has a meaning that encompasses the inclusion of additional components in addition to the elements listed as being included.

[0045] This specification describes only the minimum components necessary for explaining the invention and does not mention components unrelated to the essence of the invention. Furthermore, the mentioned components should not be interpreted in an exclusive sense, but in a non-exclusive sense, which may include other unmentioned components.

[0046] In this specification, "module," "unit," "part," or "node" refers to a logical combination of general-purpose hardware and software that performs the functions thereof.

[0047] The expressions “first,” “second,” or similar expressions used in this specification are used to distinguish identical or similar components or to distinguish the names of components or steps constituting the present invention, and do not imply an order or a plural.

[0048] The method according to the present invention may be executed by an electronic computing device (also referred to as a "terminal" in this specification), such as a computer, tablet PC, mobile phone, portable computing device, or fixed computing device. It should also be understood that one or more methods or forms of the present invention may be executed by at least one processor. The processor may be installed in a computer, tablet PC, mobile device, portable computing device, etc. Memory configured to store computer program instructions may be installed in such a device, and the processor may be specifically programmed to execute program instructions in which a program is stored, thereby executing one or more processes as described in this specification. It should also be understood that the information and methods described in this specification may be executed by a computer, tablet PC, mobile device, portable computing device, etc., comprising one or more additional components and a processor. Furthermore, control logic may be implemented on a non-volatile computer-readable medium containing program instructions executable by a processor, a control unit, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. Additionally, computer-readable recording media may be distributed across networked computers and may be stored and executed in a distributed manner, for example, by a remote server or cloud.

[0049] FIG. 1 illustrates an example of an environment in which an adaptive inference method according to the present invention is executed.

[0050] The adaptive inference system (1) according to the present invention includes a domain classification module (11), a model selection engine (12), a context bridge (13), and a model cache manager (14), and utilizes a plurality of small language models (2) provided for each domain.

[0051] The domain classification module (11) analyzes the domain related to the user's input in real time. The domain classification module (11) includes a text preprocessor (111), a domain embedding generator (112), a classification neural network (113), and a confidence calculator (114).

[0052] The text preprocessor (111) refines and standardizes the user's raw input and converts it into a form suitable for subsequent processing.

[0053] Cleaning may include, for example, the following processes.

[0054] ● Remove HTML tags, XML tags (e.g.: , etc)

[0055] ● Normalize or remove special characters and emojis

[0056] ● Remove duplicate spaces and unnecessary newline characters

[0057] ● Masking or removing noise data such as URLs and email addresses

[0058] ● Filtering promotional text and spam patterns

[0059] Standardization can include, for example, the following processes.

[0060] ● Case-uniform conversion (generally unified to lowercase)

[0061] ● Unicode Normalization (Unified into NFC or NFD methods)

[0062] ● Expansion of abbreviations and acronyms (e.g., "AI" → "Artificial Intelligence", "ML" → "Machine Learning")

[0063] ● Standardization of synonyms and related terms (e.g., "medical," "medical," "healthcare" → standardized to "medical")

[0064] ● Standardize number notation (e.g., "1,000" → "thousand", "10%" → "ten percent")

[0065] Transformation can include, for example, the following processes.

[0066] ● Sentence-level segmentation and tokenization

[0067] ● Korean Morphological Analysis (e.g., "좋었다" → "좋다 / adjective + 았 / past tense + 다 / final ending")

[0068] ● Part-of-speech tagging and named entity recognition

[0069] ● Removal of stop words (particles, endings, etc.: "eun", "neun", "i", "ga", "eul", "reul", etc.)

[0070] ● Stem extraction and headword conversion

[0071] The domain embedding generator (112) quantifies semantic characteristics by converting information preprocessed by the text preprocessor (111) into embeddings in a high-dimensional vector space. The vectors converted by the domain embedding generator (112) can be used for similarity comparison with a domain template (described later).

[0072] The classification neural network (113) selects the most suitable domain among possible domains based on the generated embeddings.

[0073] The reliability calculator (114) quantifies the certainty of the model selection by improving the reliability score for the classification result.

[0074] The model selection engine (12) selects an optimal small language model for a domain selected by the domain classification module (11), and includes a model performance matrix (121), a priority determination algorithm (122), and an alternative model recommendation unit (123) as shown in FIG. 3.

[0075] The model performance matrix (121) manages performance indicators of small language models available for each domain. Performance indicators may include performance and function indicators (benchmark score, accuracy, confusion, etc.) and efficiency indicators (inference time, inference speed, model size, memory usage, power consumption, etc.).

[0076] The priority determination algorithm (122) assigns priority to candidate models by taking into account the current situation and requirements.

[0077] The priority determination algorithm (122) can be configured as follows, for example, based on the multiple criteria decision-making (MCDM) method.

[0078] 1. Overall Score Calculation Formula:

[0079] * Final Score = α × Performance Score + β × Efficiency Score + γ × Availability Score + δ × Frequency Score + ε × Domain Fit Score

[0080] (where α + β + γ + δ + ε = 1)

[0081] 2. Detailed calculation method for each score:

[0082] ● Performance score (0~100):

[0083] - Benchmark scores (BLEU, ROUGE, BERTScore, etc.)

[0084] - Domain-specific expertise evaluation scores

[0085] - Past User Satisfaction Evaluation

[0086] ● Efficiency score (0~100):

[0087] - Reciprocal of reasoning time × 100 (Faster is higher score)

[0088] - Reciprocal of memory usage × 100 (lower score means higher score)

[0089] - Power consumption efficiency

[0090] ● Availability score (0 or 100):

[0091] - Whether it can be executed with current system resources (binary score)

[0092] - Check License and Permissions

[0093] - Verify model file integrity

[0094] ● Usage frequency score (0~100):

[0095] - Score based on usage over the past 7 days

[0096] - Apply user preference weights

[0097] - Consider usage patterns by time of day

[0098] ● Domain Fit Score (0~100):

[0099] - Match between the selected domain and the model-specific region

[0100] - Cross-domain performance evaluation results

[0101] - Training data quality index by domain

[0102] 3. Weight Dynamic Adjustment Rules:

[0103] ● Real-time service: α=0.2, β=0.4, γ=0.2, δ=0.1, ε=0.1 (Emphasis on efficiency)

[0104] ● Professional Task: α=0.4, β=0.1, γ=0.2, δ=0.1, ε=0.2 (Performance-focused)

[0105] ● Resource Constraint Environment: α=0.1, β=0.2, γ=0.5, δ=0.1, ε=0.1 (Emphasis on Availability)

[0106] The alternative model recommendation unit (123) suggests an appropriate alternative when the use of the highest priority small language model is not possible.

[0107] The following situations can be considered as cases where the use of a top-priority small language model is not possible.

[0108] ● Insufficient Memory: When the available system memory is less than the minimum memory required for model loading

[0109] ● GPU / CPU resource limit: When the current system's computational power does not meet the model's minimum requirements

[0110] ● Model file error: If the model file is corrupted or missing

[0111] ● License Restrictions: If you do not have permission to use the model or the license has expired

[0112] ● Network connection error: When unable to download the model from the remote repository

[0113] ● Version Compatibility Issues: Version mismatch between the current system environment and the model

[0114] In cases where the use of the highest priority small language model is not possible as described above, the alternative model recommendation unit (123) can recommend an appropriate alternative model according to a multi-layered decision process such as the following.

[0115] 1. Selection of Next-Ranked Model: Select the next-best model in order of scores calculated by the priority determination algorithm.

[0116] 2. Resource-Adaptive Selection: Prioritize lightweight models suited to current system resources.

[0117] ○ Quantized model (INT8, INT4 versions)

[0118] ○ Pruned lightweight version model

[0119] ○ A model compressed into Knowledge Distillation

[0120] 3. Domain Similarity-Based Selection: Replacing with a model from an adjacent domain that has characteristics similar to the target domain.

[0121] ○ Medical → Life Sciences, Bio Domain

[0122] ○ Legal → Administrative, Policy Domain

[0123] ○ IT → Engineering, Technology Domain

[0124] 4. Hybrid Approach: Combining multiple small models to compensate for performance deficiencies.

[0125] ○ Application of ensemble techniques

[0126] ○ Multi-model voting method

[0127] ○ Configuration of step-by-step processing pipelines

[0128] 5. General-purpose model fallback: Use a general-purpose small language model as a last resort.

[0129] The context bridge (13) performs the role of maintaining the conversational context when switching small language models and includes a context compression unit (131), a state transfer unit between models (132), and a continuity verification unit (133) as shown in FIG. 4.

[0130] The context compression unit (131) compresses key information from the previous conversation and converts it into a form that can be efficiently transmitted to a new model.

[0131] Extraction of key information from the previous conversation can be performed, for example, as follows.

[0132] ● Entity-based screening

[0133] - Extract key entities mentioned in the conversation (person names, place names, organization names, technical terms, etc.) and assign priorities based on frequency and importance.

[0134] ● Utilizing attention weights

[0135] - Identify important tokens and sentences by utilizing weights calculated from the Transformer model's attention mechanism

[0136] ● TF-IDF-based importance calculation

[0137] - Extract core keywords by calculating TF-IDF (Term Frequency-Inverse Document Frequency) scores for each dialogue turn.

[0138] ● Apply temporal decay

[0139] - Apply a time-based decay function that assigns higher weight to recently mentioned content (e.g., use an exponential decay function).

[0140] ● Maintain thematic consistency

[0141] - Identify the topic flow of the conversation through LDA (Latent Dirichlet Allocation) or BERT-based topic modeling, and prioritize the preservation of information related to consistent topics.

[0142] The process of compressing key information and converting it into a form that can be efficiently transmitted to a new model can be executed, for example, as follows.

[0143] ● Embedded Vector Compression

[0144] - Reduce the dimensionality of the original conversation embedding vectors to 1 / 5 to 1 / 10 of the original size using PCA (Principal Component Analysis) or an auto-encoder.

[0145] ● Structured Metadata

[0146] - Converts into structured data in JSON format, including key entities, keywords, emotional states, conversational intent, etc.

[0147] ● JSON format example

[0148] {

[0149] "entities": ["Patient Name", "Hospital Name", "Department"],

[0150] "keywords": ["Symptoms", "Prescription", "Treatment"],

[0151] "intent": "medical consultation",

[0152] "emotion": "concern",

[0153] "summary": "Consultation regarding abdominal pain symptoms"

[0154] }

[0155] ● Token Sequence Compression

[0156] - Generate a compressed token sequence by selecting only tokens with an importance score above a threshold.

[0157] ● Hierarchical Summary

[0158] - Provides granular contextual information by generating a three-level hierarchical summary at the sentence, paragraph, and whole-dialogue levels.

[0159] The inter-model state transfer unit (132) converts the context compressed by the context compression unit (131) into a form that can be understood by the small language model being converted (token-based context, vector-based context, key-value-based context, etc.) and transmits it.

[0160] The continuity verification unit (133) checks whether the logical consistency of the conversation is maintained even after the small language model is converted. The process of checking whether the logical consistency of the conversation is maintained can be performed, for example, as follows.

[0161] Verification of Topic Continuity:

[0162] ● Measure the cosine similarity between the previous conversation topic and the current response topic to check if a similarity of 0.7 or higher (arbitrary value) is maintained.

[0163] ● Topic vectors are generated using SBERT (Sentence-BERT) or Universal Sentence Encoder

[0164] ● Generate a warning signal and notify the user when abrupt topic switching is detected.

[0165] Entity consistency verification

[0166] ● Verify the consistency of key entities (people, places, concepts, etc.) mentioned in the conversation.

[0167] ● Extract entities and track reference relationships using Named Entity Recognition (NER) technology

[0168] ● Verify whether there are different representations or conflicting information regarding the same entity

[0169] Temporal consistency verification

[0170] ● Verify the logical validity of the chronological order and causal relationships in the flow of the conversation

[0171] ● Verify tense agreement and consistency of time expressions

[0172] ● Check if the logical link between the past, present, and future is maintained

[0173] Verification of emotional and tone consistency

[0174] ● Track changes in emotional state throughout the conversation using an emotion analysis model

[0175] ● Detect sudden emotional changes or inconsistencies in tone

[0176] ● Check if the speaker's personality or tone of voice is maintained consistently

[0177] Verification of semantic contradictions

[0178] ● Check for logical contradictions or conflicting statements

[0179] ● Check consistency between propositions through a reasoning rule-based verification system

[0180] ● Set a review flag for responses with low confidence

[0181] The model cache manager (14) executes the loading / unloading of memory-efficient small language models and includes an LRU cache policy unit (141), a predictive preloading unit (142), and a memory usage monitoring unit (143) as shown in FIG. 5.

[0182] The LRU cache policy unit (141) determines the small language models to be removed from memory based on the Least Recently Used algorithm.

[0183] The predictive preloading unit (142) analyzes the user's conversation pattern and preloads the model that is likely to be needed next. The predictive preloading method will be described later with reference to FIG. 9.

[0184] The memory usage monitoring unit (143) tracks the memory status of the system in real time and supports appropriate cache management decisions.

[0185] Figure 6 shows an overall flowchart of the adaptive inference method according to the present invention.

[0186] When user input is received in step (600), the domain classification module (11) preprocesses the input (step 605) and selects and classifies the domain related to the input (step 610).

[0187] Figure 7 illustrates a flowchart of a domain selection method. In step (700), domain-specific keywords are extracted from user input, and candidate domains are selected.

[0188] In step (710), the cosine similarity between the domain template of the selected candidate domain and the input text is compared, and in step (720), the confidence score is calculated. The confidence score can be calculated through the weighted average of the keyword score and the similarity score.

[0189] The calculation of the reliability score can be performed, for example, as follows.

[0190] Basic Confidence Score Calculation Formula:

[0191] Trust Score = w₁ × Keyword Score + w₂ × Similarity Score + w₃ × Context Score + w₄ × Confidence Score

[0192] (Here, the sum of all weights (w₁ + w₂ + w₃ + w₄) = 1)

[0193] Calculation method for each component

[0194] - Keyword Score (0~100)

[0195] Keyword Score = (Σ(Domain-Specific Keywords) 가중치 ) / Total number of keywords) × 100 → Domain-specific keywords 가중치 = TF-IDF value × Domain specificity coefficient

[0196] - Similarity score (0~100)

[0197] Similarity score = cosine_similarity(input_embedding_vector, domain_template_vector) × 100 → cosine_similarity(A,B) = (A·B) / (∥A∥ ​​× ∥B∥)

[0198] - Contextual score (0~100)

[0199] Context Score = α × Similarity + β × Topic Consistency → Similarity = cosine_similarity (Current Input, Summary of Previous N Turns) → Topic Consistency = 1 - topic_drift_penalty

[0200] - Confidence Score (0~100)

[0201] Confidence Score = (Highest Score - Next Highest Score) / Highest Score × 100 → Entropy-based Confidence = (1 - H(P)) × 100 → Here, H(P) = -Σ(p i × log(p i ))

[0202] Weight Setting Example (by Domain Type)

[0203] 1. General case

[0204] w₁=0.3, w₂=0.4, w₃=0.2, w₄=0.1

[0205] 2. Full-text domain

[0206] w₁=0.4, w₂=0.3, w₃=0.2, w₄=0.1 (Keyword emphasis)

[0207] 3. Interactive Domain

[0208] w₁=0.2, w₂=0.3, w₃=0.4, w₄=0.1 (context-conscious)

[0209] In step (730), a domain with a confidence score of 0.8 or higher, for example, is selected as the final domain.

[0210] In step (615), it is determined whether the selected domain matches the currently operating small language model. If it is determined to match, the process proceeds to step (665) to perform inference.

[0211] If the result is determined to be "No" in step (615), proceed to step (620) and execute the (small language) model selection algorithm.

[0212] Figure 8 illustrates a flowchart of the selection process for a small language model.

[0213] In step (800), a list of small language models corresponding to the domain selected in step (610) is extracted. In step (810), a performance score based on past performance history and similarity with the current context is calculated, and additionally, in step (820), loadable small language models are filtered by checking resource constraints.

[0214] Finally, in step (830), the optimal small language model is determined by considering the performance score and resource efficiency.

[0215] In step (625), the memory status is checked, and in step (630), it is determined whether the small language model determined in step (830) exists in the cache. If it is determined that the small language model exists in the cache, the process proceeds to step (635) to load the determined small language model from the cache, and if it is determined that it does not exist, the process proceeds to step (640) to load the selected small language model from the disk.

[0216] After step (645), the previously loaded small language model is unloaded (step 650), and the small language model loaded from disk is loaded into memory (step 655).

[0217] In step (660), context is delivered to the newly loaded small language model. As previously mentioned, the context bridge (13) generates a context that is converted into a form that the newly loaded small language model can understand, and this context is delivered to the newly loaded small language model.

[0218] Next, in step (665), the newly loaded small language model performs inference, and in step (670), a response is generated.

[0219] According to the present invention, a subsequent small language model is predicted in the next step (675).

[0220] Figure 9 shows a flowchart of the predictive preloading process.

[0221] In step (900), the pattern of the last N conversations is analyzed to extract the domain switching pattern. Then, in step (910), a matrix of domain switching probabilities, i.e., from the current domain to another domain, is constructed.

[0222] In step (915), a threshold value, for example, a conversion probability domain of 0.3 or higher is selected.

[0223] The transition probability matrix can be constructed as follows.

[0224] P(domain j | Domain i ) = count(domain i → Domain j ) / count(domain i Transition starting from)

[0225] If an example matrix like the one below exists,

[0226] Medical, Legal, IT, Finance

[0227] Medical 0.60 0.25 0.10 0.05

[0228] Law 0.15 0.70 0.05 0.10

[0229] IT 0.05 0.05 0.80 0.10

[0230] Finance 0.10 0.20 0.15 0.55

[0231] If the current domain is "medical", select the "medical" domain, which is a conversion probability domain with a threshold of 0.3 or higher.

[0232] In step (920), a small language model of the selected domain is predicted. For example, the following prediction method can be considered.

[0233] - Examples of models belonging to the medical domain

[0234] 1. Medical-BERT-Ko

[0235] 2. BioBERT-Korean

[0236] 3. Healthcare-GPT-sLLM

[0237] 4. MedicalQA-T5-small

[0239] Step 4: Final Decision on the Target Model for Prediction

[0240] All small language models belonging to the selected domain may be used as prediction targets, and priority may be assigned based on the following prediction criteria.

[0241] ● Performance ranking within the domain

[0242] ● System resource usage efficiency

[0243] ● Past usage frequency and preference

[0244] In step (680), if there is free memory space, the predicted small language model is loaded in the background.

[0245] Finally, the response is returned in step (685).

[0246] According to a preferred embodiment of the present invention, the size of a small language model can be reduced through quantization and pruning to shorten the loading time.

[0247] According to another preferred embodiment of the present invention, a parallel loading technique can be used to simultaneously load multiple small language models using multiple threads.

[0248] According to another preferred embodiment of the present invention, a progressive loading technique can be used to reduce the initial response time by preferentially loading the core layers of a small language model, such as an embedding layer and an initial transformer block.

[0249] Although the present invention has been described above with reference to the attached drawings, the scope of the present invention is determined by the claims set forth below and should not be interpreted as being limited to the aforementioned embodiments and / or drawings. Furthermore, it should be clearly understood that improvements, changes, and modifications to the invention described in the claims that are obvious to those skilled in the art are also included within the scope of the present invention.

Claims

Claim 1 An adaptive inference method by dynamic switching of a small language model executed by an adaptive inference system comprising a domain classification module, a model selection engine, a context bridge, and a model cache manager, wherein the domain classification module preprocesses user input in a first step; the domain classification module selects a domain based on the preprocessing result of the first step in a second step; the domain classification module determines whether the domain selected in the second step matches the current small language model in a third step; the model selection engine executes the selection of another small language model if it is determined in the third step that it does not match the current small language model in a fourth step; the model cache manager determines in a fourth-1 step whether the small language model selected in the fourth step exists in the cache in a cache in a fourth step; the model cache manager loads the small language model selected from the cache if it is determined in the fourth-1 step that it exists in the cache, and loads the small language model selected from the disk otherwise; and the context bridge [retrieves] key information based on entity-based screening, utilization of attention weights, TF-IDF-based importance calculation, application of temporal decay, and maintenance of topic consistency for the previous conversation An adaptive inference method by dynamic switching of a small language model, comprising: a 5-1 step of extraction; a 5-2 step in which a context bridge compresses the embedding vector of the previous conversation, converts key information into structured metadata, selects only tokens with an importance score above a threshold to generate a compressed token sequence, and hierarchically summarizes the previous conversation to generate context; and a 6 step in which the context bridge transmits the context generated by the 5-1 and 5-2 steps to a small language model selected in the 4th step. Claim 2 delete Claim 3 An adaptive inference method by dynamic switching of a small language model according to claim 1, further comprising a 5-3 step of unloading the current small language model when another small language model is selected and loaded in the 5th step. Claim 4 An adaptive inference method by dynamic switching of a small language model according to claim 1, further comprising an 8th step of predicting a subsequent small language model and a 9th step of pre-loading the subsequent small language model predicted in the 8th step in the background. Claim 5 The adaptive inference method by dynamic switching of a small language model according to claim 1, wherein the second step comprises: a second-1 step of extracting domain-specific keywords included in the preprocessing result of the input and selecting candidate domains; a second-2 step of performing a similarity comparison between the input text and the domain template; a second-3 step of calculating a confidence score based on the keyword score and the similarity comparison score; and a second-4 step of selecting a domain based on the confidence score of the second-3 step. Claim 6 The adaptive inference method by dynamic switching of small language models according to claim 1, wherein the fourth step comprises: a fourth-1 step of extracting a list of small language models corresponding to a selected domain; a fourth-2 step of calculating a score based on similarity with past performance history and current context; a fourth-3 step of checking resource constraints and extracting a loadable small language model; and a fourth-4 step of selecting a final small language model considering performance score and resource efficiency. Claim 7 The adaptive inference method by dynamic switching of small language models according to claim 4, wherein step 8 comprises: step 8-1, which extracts a domain switching pattern from a recent predetermined number of conversations; step 8-2, which constructs a probability matrix of switching from a current domain to another domain; step 8-3, which selects a domain whose switching probability is greater than or equal to a predetermined threshold; and step 8-4, which predicts a small language model selected according to a predetermined priority among small language models corresponding to the domain selected in step 8-3 as the next small language model. Claim 8 A computer-readable recording medium having a computer program recorded thereon that executes the method of any one of claims 1, 3, 4, 5, 6 and 7. Claim 9 A computer program recorded on a computer-readable recording medium to execute the method of any one of claims 1, 3, 4, 5, 6 and 7. Claim 10 In an adaptive inference system that dynamically switches multiple small language models, the adaptive inference system comprises a domain classification module, a model selection engine, a context bridge, and a model cache manager; the domain classification module preprocesses user input and selects a domain based on the preprocessing result; the model selection engine determines whether the selected domain matches the current small language model and, if it is determined that it does not match the current small language model, executes the selection of another small language model; the context bridge extracts key information from the previous conversation based on entity-based screening, utilization of attention weights, TF-IDF-based importance calculation, application of temporal decay, and maintenance of topic consistency; compresses the embedding vector of the previous conversation and converts the key information into structured metadata; generates a compressed token sequence by selecting only tokens with an importance score above a threshold, and generates context by hierarchically summarizing the previous conversation; An adaptive inference system that performs dynamic switching of small language models, delivering context to a selected small language model, and a model cache manager loads the selected small language model from the cache if it is determined that the selected small language model exists in the cache, and loads the selected small language model from the disk otherwise; and loads the selected small language model into memory and unloads the existing small language model.

Images