System and method for automatically generating a configuration for training a machine learning model

Abstract

ABSTRACT SYSTEM AND METHOD FOR AUTOMATICALLY GENERATING A CONFIGURATION FOR TRAINING A MACHINE LEARNING MODEL The present invention discloses a system (100) for automatically generating a configuration for training a machine learning model. The system (100) comprises at least one processor (102), a memory (104) storing a knowledge base of domain-specific hyperparameter templates, one or more GPUs (106-1, …, 106-N) providing hardware-accelerated computation, a user interface (108), and a network interface. The processor (102) is configured to retrieve GPU resource information including available memory, compute capability, and bandwidth; obtain dataset metadata and model identifiers; and compute model memory requirements and maximum feasible batch size. Based on these parameters, the processor selects and validates hyperparameter configurations through simulated GPU memory allocation, and automatically applies fallback adjustments to generate an optimized, hardware-aware configuration executable on the available GPUs, thereby enabling efficient, reliable, and adaptive machine learning model training. FIG. 1 will be the reference figure.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of artificial intelligenceand machine learning systems, and more particularly to a system and method forautomatically generating optimized configurations for training machine learningmodels based on hardware resources, dataset characteristics, and domain-specificrequirements. The invention further relates to adaptive configuration managementframeworks that enable compliant, resource-efficient, and explainable modeltraining workflows.BACKGROUND

[0002] Machine learning (ML) model training has become an integralcomponent of modern computational systems utilized in diverse applications suchas predictive analytics, computer vision, natural language processing, and otherdata-driven tasks. The process of training ML models involves adjusting multiplehyperparameters including learning rate, batch size, optimization algorithms, andthe number of training epochs, each of which significantly influences theconvergence behavior, model accuracy, and computational efficiency. Theselection of these parameters is often empirical and heavily dependent on theexpertise of data scientists or ML engineers, thereby limiting accessibility for nonexpert users.

[0003] In current systems, configuration of ML training environments isgenerally performed either manually or through automated machine learning(AutoML) platforms. These platforms typically employ generic optimizationstrategies such as random search, Bayesian optimization, or evolutionaryalgorithms to identify parameter combinations that yield improved results.However, such approaches rely on multiple iterations of training and validation,which significantly increases computational overhead and training time.Furthermore, these optimization processes often lack contextual awareness offactors such as dataset size, model complexity, or hardware capabilities, leading toinefficient utilization of available resources.

[0004] Existing systems also face persistent technical challenges duringmodel training, including memory allocation failures caused by inappropriatebatch size selection, model divergence resulting from unsuitable learning rateconfigurations, overfitting when operating on limited datasets, and excessivetraining durations due to suboptimal hyperparameter tuning. Additionally, currentAutoML frameworks generally provide limited interpretability of theirconfiguration choices and are often unable to dynamically adapt to the diversity oftasks, domains, and computational environments encountered in practical MLapplications.

[0005] Therefore, there is a need for an improved system and method forconfiguring machine learning training environments that can address thelimitations of current approaches, reduce dependency on manual expertise,prevent training failures, and ensure efficient utilization of availablecomputational resources while maintaining consistency and reliability in modelperformance across varying application domains.SUMMARY

[0006] The present disclosure relates to an automated computingframework capable of intelligently generating a configuration for training machinelearning models without requiring repeated experimentation or manual parametertuning. The disclosure encompasses both a method and a system configured todynamically interpret available computational resources, dataset characteristics,and model architecture parameters to derive optimal hyperparameter settings forefficient and stable model training. The disclosed framework leverages directinteraction with hardware-level drivers and employs resource simulation to pre-validate memory allocations, thereby reducing the occurrence of training failures,unstable convergence, and inefficient utilization of compute resources that arecommon in existing machine learning training workflows.

[0007] In one implementation, the processor of the computing deviceinitiates a hardware assessment sequence that queries each available GPU throughCUDA, ROCm, or equivalent low-level API calls. The assessment identifies keyperformance attributes including total and free GPU memory, the number ofavailable GPUs, their respective compute capabilities, bandwidth, and currentworkload utilization. The detected information establishes a real-timecomputational resource map of the hardware environment in which the training isto be executed. This enables the system to account for resource variability,ensuring that subsequent configuration steps are grounded in actual, rather thantheoretical, compute availability.

[0008] Following the hardware assessment, the processor receivesmetadata describing the dataset and intended task through either the user interfaceor the network interface. Such metadata includes the dataset size, applicationdomain (for example, vision, text, or tabular data), and the intended learning tasksuch as classification, regression, or sequence modeling. Simultaneously, theprocessor obtains a model identifier corresponding to a specific architecture (suchas CNN, RNN, or transformer-based models) and estimates its inherent memoryrequirements. The estimation includes computing memory usage for parameterstorage, optimizer state, and forward activation data. Based on these computedvalues and the available GPU memory, the processor determines a maximumfeasible batch size that can be processed without triggering memory overflow.

[0009] Once the resource constraints and model requirements areestablished, the processor retrieves a set of candidate hyperparameters from adomain-specific knowledge base stored in memory. The knowledge basemaintains empirically optimized templates indexed by task type, dataset scale, andmodel family. The retrieved hyperparameters include, but are not limited to,learning rate, batch size, and number of epochs. These parameters are thenadjusted dynamically by the processor using mathematical relations that maintainproportionality across training scales - for example, adjusting the learning rate inrelation to batch size through a square-root scaling rule. Such computation ensuresthat scaling the batch size does not adversely affect convergence or stability ofgradient descent optimization.

[0010] The candidate configuration is further validated by a simulatedmemory allocation process, wherein the system performs a dummy forward-passor an equivalent simulation routine on the GPUs to estimate peak memoryutilization. This pre-validation enables the detection of possible overflowconditions before the commencement of actual model training, thereby preventingtraining failures. When such a risk condition is identified, the systemautomatically initiates a fallback mechanism that modifies one or moreconfiguration parameters. The fallback strategies may include reducing batch size,increasing gradient accumulation steps to emulate larger batch training, enablingmixed-precision computation modes (FP16 / BF16) when tensor core support isdetected, or redistributing workloads across multiple GPUs through dataparallelism. These corrective actions ensure that the model remains executablewithin available resource constraints while retaining training efficiency.

[0011] In addition to configuration generation, the processor maintainsdetailed logs of all parameter decisions, including justifications linked tohardware limitations, dataset scale, and architecture-specific constraints. Thisaudit capability provides transparency and reproducibility, enabling users andregulatory systems to trace the rationale behind each configuration. In distributedor multi-GPU environments, the processor coordinates inter-GPU workloadallocation and synchronization using data parallelism, gradient checkpointing, andcommunication optimization techniques to balance memory load and computationthroughput effectively. Further, when dataset metadata includes data locality orprivacy constraints, the system ensures compliance by generating configurationsthat preserve data within authorized compute boundaries, thereby maintainingsecurity and integrity of sensitive datasets.

[0012] The disclosed system achieves a computing workflow thatautomatically aligns training configurations with the available hardwareenvironment and dataset characteristics. By pre-validating memory allocations,dynamically scaling hyperparameters, and incorporating adaptive fallbackstrategies, the system ensures that training processes execute without failure orresource wastage. The method substantially reduces the dependency on humanexpertise, eliminates repetitive trial-and-error tuning, and accelerates thepreparation phase of machine learning model training. The integration of real-timehardware querying, simulation-driven validation, and adaptive configurationgeneration collectively enables the system to maintain consistent trainingperformance across diverse computational infrastructures and domains. Theoutcome is a more reliable, efficient, and user-accessible training process capableof achieving balanced utilization of computational resources while maintainingstability and accuracy in trained models.BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 illustrates a schematic diagram of the system automaticallygenerating a configuration for training a machine learning model, in accordancewith an embodiment of the present disclosure.

[0014] FIG. 2 illustrates an exemplary flowchart representing a method forautomatically generating a configuration for training a machine learning model, inaccordance with an embodiment of the present disclosure.DETAILED DESCRIPTION

[0015] Machine learning (ML) model training forms a critical part ofmodern computational systems across various domains, including computervision, natural language processing, and predictive analytics. The process requiresthe careful selection of multiple hyperparameters such as learning rate, batch size,and number of epochs, which significantly affect model accuracy, convergence,and computational efficiency. In current practice, these configurations aredetermined manually or through automated platforms that rely on iterativeoptimisation strategies, often resulting in high computational overhead andprolonged training cycles. Moreover, existing approaches lack dynamic adaptationto factors such as dataset scale, model complexity, and hardware capability,leading to frequent issues like memory overflow, unstable training behavior, andinefficient resource utilisation.

[0016] Accordingly, there exists a requirement for an improvedconfiguration approach that minimises manual intervention, prevents trainingfailures, and ensures optimal utilisation of available computing resources forconsistent model performance across varied environments.

[0017] FIG. 1 illustrates a schematic diagram of the system automaticallygenerating a configuration for training a machine learning model, in accordancewith an embodiment of the present disclosure.

[0018] The system (100) comprises at least one processor (102), amemory (104), one or more GPUs (106-1, 106-2,..., 106-N), and a user interface(108).

[0019] The memory 104 is storing a knowledge base of domain-specifichyperparameter templates. The memory (104) stores a knowledge basecomprising domain-specific hyperparameter templates. Each templateencapsulates empirically tuned configurations corresponding to predefineddomains (e.g., natural language processing, computer vision, or tabular analytics)and task types (e.g., classification, segmentation, or regression). The templatesmay include base learning rate, optimiser type, batch size ranges, number ofepochs, and augmentation settings.

[0020] This knowledge base may be structured as a relational database orrule-based inference engine, allowing the processor (102) to perform templateretrieval through indexed metadata queries. The processor (102) uses acombination of domain mapping and interpolation algorithms to derive near-optimal hyperparameter sets when no exact template match is available.

[0021] The one or more GPUs (106-1, 106-2, ..., 106-N) are electricallyconnected with the processor (102) and provide hardware-accelerated modelexecution capability. The GPUs (106) provide the hardware-acceleratedcomputation backbone for model execution and memory simulation. Each GPU iselectrically coupled to the processor via a high-bandwidth PCIe or NVLinkinterconnect, enabling low-latency communication.

[0022] The GPUs (106) are interrogated to assess free GPU memory,compute throughput (TFLOPs), and inter-GPU communication bandwidth,allowing the system (100) to understand the real-time hardware environment. Thisinformation is essential for determining the maximum feasible batch size and forpreventing out-of-memory conditions during actual training. The processor (102)executes a mathematical model that estimates total model memory consumptionusing the formula:M_total = M_parameters + M_optimizer + M_activations_per_batch x Batch_sizewhere M_activations_per_batch is estimated through priorprofiling data or a simulated forward-pass.

[0023] The user interface (108) is configured to receive user inputs anddisplay recommended configurations, and communicate with remote resources.

[0024] The user interface (108) provides a visual and interactive mediumthrough which users input dataset metadata such as task type, data domain, datasetsize, and optionally the target model architecture identifier. The interface can beimplemented as a graphical dashboard or command-line tool. It further displaysthe system-generated configuration, including hyperparameters, resourceutilization projections, and justification logs explaining each recommendedsetting. This enables transparent interaction between human operators and theconfiguration engine, while maintaining complete autonomy in backend decision-making.

[0025] At least one processor (102) is configured to retrieve hardwareresource information from the GPUs, the hardware resource information includingfree GPU memory, compute capability and bandwidth.

[0026] At least one processor (102) is configured to obtain datasetmetadata and a model identifier from the user interface (108) or the networkinterface.

[0027] At least one processor (102) is configured to compute modelmemory requirements and maximum feasible batch size.

[0028] At least one processor (102) is configured to select recommendedhyperparameters based on the dataset metadata, model memory requirements andhardware resources using the knowledge base stored in the memory (104).

[0029] At least one processor (102) is configured to simulate GPUmemory usage to validate the configuration and automatically implement fallbackadjustments to produce a validated configuration executable on the availableGPUs.

[0030] In one embodiment, the processor (102) is configured to estimateper-sample memory footprint by dividing total predicted activation memory bythe computed tensor shape.

[0031] In another embodiment, the processor (102) is configured toestimates per-sample memory footprint by dividing total predicted activationmemory by computed tensor shape.

[0032] In yet another embodiment, the processor (102) is configured toallocate workloads across multiple GPUs using data parallelism and gradientcheckpointing to optimize memory consumption and compute throughput.

[0033] The processor (102) may include one or more central processingunits (CPUs), tensor processing units (TPUs), or AI accelerators capable ofexecuting instruction sets supporting high-performance computation and driver-level communication with GPU hardware. The processor (102) executes aconfiguration generation engine, implemented as executable instructions stored inthe memory (104), to orchestrate all operations of the system (100).

[0034] The processor (102) initially retrieves real-time hardware resourceinformation from each GPU (106). This includes querying available GPUmemory, compute capability (e.g., CUDA cores, Tensor cores, or streamprocessors), and bandwidth characteristics using low-level driver APIs such asCUDA Management Library (NVML) or ROCm-SMI interfaces. The retrievaloperation may further capture GPU temperature, active processes, and loaddistribution, which serve as contextual parameters for configuration decision-making.

[0035] The network interface allows the system (100) to access remoterepositories, cloud-based datasets, or distributed compute clusters. It may alsoretrieve updated domain templates from a centralised configuration server orcommunicate with federated learning nodes, while adhering to local data securitypolicies.

[0036] In one embodiment, the system (100) incorporates a data isolationarchitecture that ensures secure configuration generation by operating solely onstatistical metadata rather than on raw data values. The processor (102)implements data fingerprinting and anonymised feature summarization techniquesto prevent exposure of identifiable information. This design inherently supportscompliance with major data protection frameworks, including HIPAA, GDPR,CCPA, and PCI-DSS, while enabling fully on-premises deployment for privacy-sensitive or regulated industry applications.

[0037] In another embodiment, the system (100) includes a compliancedocumentation generation module configured to automatically produce auditableconfiguration records, wherein each recommended hyperparameter isaccompanied by parameter-level justification and version metadata. The processor(102) maintains complete configuration trails for each training setup, generatingdocumentation compatible with regulatory review requirements and integratingdomain-specific compliance checks such as fairness audits for financial or creditrisk models.

[0038] In a further embodiment, the system (100) implements privacy-preserving configuration mechanisms by applying differential privacy techniquesduring dataset characterization and utilizing k-anonymity principles whenreferencing cross-domain templates. The processor (102) executes domain-specific regularization strategies to minimize model memorization risks andsupports federated configuration generation workflows, allowing distributedconfiguration synthesis without centralizing sensitive datasets.

[0039] In yet another embodiment, the system (100) is designed to handleextreme resource constraints by detecting CPU-only environments andautomatically applying lightweight or memory-efficient configuration strategies.The processor (102) may activate progressive data loading mechanisms fordatasets exceeding available memory capacity, enable mixed-precision(FP16 / BF16) training where supported, and generate specialized modelconfigurations suitable for edge or mobile deployments.

[0040] In another embodiment, the system (100) supports unconventionaldataset characteristics by automatically identifying severe class imbalances andadjusting training configurations through adaptive loss function selection. Theprocessor (102) dynamically modifies sequence length parameters for extremelylong inputs using context windowing, applies few-shot learning optimizationswhen dataset size is limited, and provides multi-modal configuration balancing fordatasets combining visual, textual, or numerical modalities.

[0041] In a further embodiment, the system (100) manages task-domaincombinations by employing intelligent parameter interpolation techniques thatreference historically validated configurations. The processor (102) appliesnearest-neighbor similarity metrics across task-domain matrices to identify themost comparable configuration, implements transfer learning-based adaptation ofparameter templates, and enforces configuration safety boundaries to ensurestability for emerging or experimental model applications.

[0042] In another embodiment, the system (100) achieves computationalscalability by generating distributed training configurations suitable for multi-GPU and multi-node environments. The processor (102) automatically configuresdata and model parallelism strategies depending on detected hardware resourcesand model complexity, implements gradient checkpointing to enable memory-efficient large-scale model training, and supports dynamic resource allocationacross elastic compute infrastructures.

[0043] In a further embodiment, the processor (102) provides adaptivedata volume scaling by adjusting sampling and batching strategies for extremelylarge datasets. The system (100) optimizes data loading patterns to handle datasetsexceeding physical memory limits, generates configurations suitable forcontinuous or streaming data pipelines, and supports incremental learning setupsfor progressively expanding datasets.

[0044] In yet another embodiment, the system (100) provides modelcomplexity scaling by generating parameter configurations ranging from smallarchitectures to multi-billion parameter transformer models. The processor (102)integrates large-model training optimizations including ZeRO, DeepSpeed, andFully Sharded Data Parallel (FSDP) approaches, and can auto-generatequantization, pruning, and model distillation parameters for deployment-efficientconfigurations in production or constrained environments.

[0045] Through these integrated security, adaptability, and scalabilitymechanisms, the system (100) delivers end-to-end control, allowing domainexperts to configure, validate, and train machine learning models withoutrequiring deep ML expertise. Each generated configuration is explainable,compliant, and resource-optimized, ensuring that even under constrained,distributed, or privacy-critical environments, reliable and efficient trainingperformance is achieved.

[0046] In one embodiment, the system (100) provides an intelligentgraphical user interface (GUI) that facilitates automated dataset creation,validation, and model configuration for machine learning applications. Theinterface is designed to enable both technical and non-technical users to performend-to-end training setup operations without requiring programming or manualparameter tuning.

[0047] In one aspect, the data source configuration module of the system(100) enables a user to initiate dataset creation through an interactive GUI panel.The user may select a desired task and domain corresponding to the data typeintended for training, such as text, image, or tabular data. The interface furtherallows the user to select the data source origin, including local directories, cloudrepositories, or pre-integrated storage systems. Upon selection, the user uploadsdata in a supported format, such as CSV or JSONL. The GUI presents a drag-and-drop mapping interface, enabling the user to align input fields from the uploadeddataset with system-defined schema requirements. The processor (102) performsautomated validation of the uploaded dataset, verifying structure, completeness,and compliance with format constraints. When the data is validated, the system(100) automatically creates a corresponding dataset entity, which is securelystored and made available for subsequent model configuration and trainingworkflows.

[0048] In another embodiment, the system (100) provides a model trainingconfiguration interface that enables a user to fine-tune or retrain a machinelearning model using the dataset created in the previous phase. The interfaceallows the user to input a model name and description and to associate the modelwith a previously created dataset. The system (100) displays configuration optionsallowing the user to choose between multiple modes of operation, including theselection of a specific model architecture, creation of a private model, retrainingof an existing model, or generation of a recommended configuration for userslacking technical expertise. Based on the user's selection, the interfacedynamically adjusts to present relevant model details, metadata, and configurationparameters corresponding to the chosen mode.

[0049] The system (100) automatically computes the trainingconfiguration parameters by analyzing contextual variables derived from theuser's selections and system state. These include the task and domain typeidentified from the dataset, the dataset size, the selected model architecture, andthe available hardware resources detected by the processor (102), such as GPUtype, compute capability, and memory capacity. Using this information, thesystem (100) generates optimized hyperparameters, including batch size, learningrate, gradient accumulation steps, sequence length, number of epochs, andoptimizer configuration. The generated parameters are presented within the GUIfor review. For non-technical users, the interface displays a simplifiedconfiguration view showing the auto-generated parameters in a read-only format.For advanced users, an extended configuration mode is available that allowsmanual modification of computed parameters prior to initiating the trainingprocess.

[0050] Once the configuration is finalized, the user initiates trainingthrough a single command, upon which the processor (102) establishes a unifiedtraining graph linking the dataset, selected model, and hyperparameterconfiguration. During execution, the system (100) maintains and recordscomprehensive configuration metadata, including model identifiers, trainingversion information, timestamps, and resource utilization data. This ensures thateach training session remains auditable and reproducible, supporting compliancewith industry and organizational requirements. Through this unified interface-driven workflow, the system (100) transforms the traditionally complex and error-prone process of dataset preparation and model configuration into a guided,automated, and explainable operation.

[0051] In an exemplary embodiment, the system (100) is utilized in atoken classification scenario within a healthcare domain for training a NamedEntity Recognition (NER) model on approximately 50,000 clinical sentencesusing a single 16 GB GPU. The processor (102) automatically detects availablehardware resources and dataset characteristics, thereafter generating an optimaltraining configuration including a batch size of sixteen (16), gradientaccumulation steps of two (2), a learning rate of 2x10-5, and a maximum sequencelength of two hundred and fifty-six (256). The configuration is accompanied by adomain-specific justification indicating that healthcare datasets containspecialized terminology and generally require extended context windows andlower learning rates to mitigate overfitting. The processor (102) further ensurescompliance with healthcare data protection standards such as HIPAA by operatingexclusively on statistical metadata during configuration generation. Through thisautomated process, the system (100) eliminates the need for manual parametertuning and prevents memory overflow errors during model training, resulting in areduction of experimental iteration time by approximately three to five days whilemaintaining training stability and compliance integrity.

[0052] In another exemplary embodiment, the system (100) is applied to atext regression task in the real estate analytics domain, wherein a dataset ofapproximately five hundred thousand (500,000) property descriptions is used topredict property valuation metrics. The processor (102) automatically configuresthe model training parameters including five (5) epochs, a batch size of thirty-two(32), a learning rate of 1x10-4, and a maximum sequence length of one hundredand twenty-eight (128). The system (100) further generates a data-awareexplanation that the high sample volume allows faster convergence with moderateepochs and a higher learning rate, while shorter sequence lengths are adequate dueto redundancy typically present in real estate text descriptions. The resultingconfiguration enables non-expert users, such as business analysts, to trainproduction-grade regression models without manual optimization, achieving up toforty percent (40%) reduction in training duration and enhanced predictionaccuracy due to domain-specific optimization performed automatically by theprocessor (102).

[0053] In yet another exemplary embodiment, the system (100) isemployed in an image classification scenario within the agricultural sector,wherein an agritech application involves one thousand (1,000) labeled images ofplant leaves for disease detection. The system (100) detects a limited hardwareresource of a 4 GB GPU and automatically generates a configuration including abatch size of eight (8), an image resolution of two hundred and twenty-four (224)pixels, a learning rate of 3x10-4, and ten (10) training epochs. The processor (102)provides a rationale indicating that a smaller batch size with extended trainingcycles improves model generalization under low-data conditions, whileaccommodating GPU memory constraints without compromising featureresolution. This configuration allows efficient execution of deep learningworkflows in resource-constrained or edge-compute environments, maximizingmodel performance despite hardware limitations and facilitating scalableagricultural AI deployment for field-level diagnostics.

[0054] In a further exemplary embodiment, the system (100) is deployedfor tabular classification in a financial fraud detection use case, wherein a fintechcompliance team inputs a dataset containing approximately two million(2,000,000) transaction records. The processor (102) generates an optimizedconfiguration comprising three (3) training epochs, a batch size of sixty-four (64),an AdamW optimizer, early stopping enabled, and a weight decay parameter of0.01. The domain-specific justification generated by the system (100) specifiesthat financial transaction data typically exhibits high noise levels and temporaldrift, necessitating appropriate regularization to prevent overfitting to anomalousor transient patterns. Early stopping criteria and weight decay thus improve modelgeneralization and robustness against market fluctuations. By automaticallygenerating such configuration parameters and their corresponding compliancedocumentation, the system (100) enables accelerated and explainable frauddetection model training, supporting faster regulatory audit readiness and reducingcompliance implementation timelines.

[0055] In another embodiment, the system (100) integrates real-timethermal feedback from GPU sensors, adjusting computational load or precisiondynamically to prevent thermal throttling. This enhances the longevity andreliability of GPU hardware, representing a tangible technical contribution todevice performance.

[0056] In yet another embodiment, the processor (102) implements cross-framework interoperability, wherein configurations generated for one MLframework (e.g., PyTorch) are automatically translated for compatibility withanother framework (e.g., TensorFlow or JAX) using rule-based configurationmapping. This provides hardware-independent optimisation while reducingredundant computation during multi-framework experimentation.

[0057] In a further embodiment, the system (100) employs incrementalself-calibration, wherein each successfully trained model contributes to updatingthe internal knowledge base with validated hyperparameter-hardware mappings,thereby enabling continuous improvement of configuration accuracy over time.

[0058] In one embodiment, the system (100) includes an adaptiveprofiling engine that continuously measures the actual versus predicted GPUmemory usage during live training and iteratively adjusts future configurationrecommendations based on discrepancy metrics, achieving autonomous self-learning optimisation.

[0059] FIG. 2 illustrates an exemplary flowchart representing a method forautomatically generating a configuration for training a machine learning model, inaccordance with an embodiment of the present disclosure.

[0060] In one embodiment, the present disclosure provides a method (200)for automatically generating a configuration for training a machine learningmodel. The method is executed by at least one processor of a computing devicecomprising one or more graphics processing units (GPUs), a memory, a userinterface, and a network interface. The disclosed method advantageously enablesdynamic adaptation of training parameters based on available hardware resourcesand dataset characteristics, thereby optimising model training efficiency withoutrequiring manual trial-and-error configuration.

[0061] At step (202), the processor (102) detects available hardwareresources by performing GPU-driver level calls through low-level APIs such asCUDA or ROCm. Through these API calls, the processor (102) identifies thememory capacity of each GPU, the number of available GPUs, GPU computecapability, current workload utilisation, and inter-GPU communicationbandwidth. This detection process may also include querying runtime GPUmemory utilisation in real-time using driver interfaces. The information obtainedforms a hardware profile representing the system's training capacity limits.

[0062] At step (204), the processor (102) receives, via either the userinterface or the network interface, dataset metadata including a task type (e.g.,image classification, language modelling, segmentation), a domain (e.g., vision,text, audio), and a dataset size (e.g., number of samples or data volume). Thedataset metadata provides contextual cues necessary for selecting an appropriatebase configuration from a knowledge base that stores known optimalconfigurations for various domains and model types.

[0063] At step (206), the processor (102) obtains a model identifierrepresenting the architecture intended for training (e.g., ResNet50, BERT, orGPT-like model). Using architectural parameters such as number of layers, hiddensize, and parameter count, the processor (102) estimates model memory usage,including contributions from parameter storage, optimiser state tensors, andforward-activation memory. The estimation may be computed analytically basedon floating-point precision type or empirically based on reference statistics storedin the knowledge base.

[0064] At step (208), the processor (102) calculates the maximum feasiblebatch size that can be executed within the available GPU memory capacity. Thiscalculation accounts for model size, temporary buffers, activation checkpointing,and mixed-precision memory reduction factors. Using the detected hardwareresources, dataset metadata, and estimated model memory usage, the processor(102) then proceeds to step (210), where it selects candidate hyperparametersincluding learning rate, batch size, and number of epochs. These selections areperformed by referencing a knowledge base stored in memory, which maps pastempirically verified configurations to similar model-dataset-hardware contexts.

[0065] In one embodiment, prior to validation, the processor computesadjusted hyperparameters. Specifically, the processor computes an adjustedlearning rate (LR_adjusted) according to the formula:LR_adjusted = LR_base x √(Batch_size_new / Batch_size_base),and determines gradient accumulation steps (GA_steps) using:GA_steps = Effective_batch_size / Maximum_hardware_batch_size.These computations ensure consistency in optimisation dynamics when the batchsize deviates from reference configurations.

[0066] At step (212), the processor performs validation by executing asimulated GPU memory allocation to detect potential memory overflowconditions. In one implementation, this involves performing a dummy forwardpass of the model with placeholder tensors to estimate peak memory consumptionduring actual training. This simulation allows early detection of out-of-memoryrisks before initiating a real training session.

[0067] In another embodiment, when the validation process indicates arisk condition such as insufficient GPU memory or overutilization, the processor(102) automatically triggers a fallback strategy. The fallback strategy includes oneor more of the following:increasing gradient accumulation steps to distribute the effective batch sizeacross multiple iterations;reducing sequence length or input size to lower per-sample memory cost;enabling mixed-precision execution (FP16 / BF16) for memory-efficientcomputation, performed only when tensor core support is detected in the GPU;reducing batch size proportionally; andrecommending distributed data-parallel training across multiple GPUs.This adaptive fallback ensures that a functional configuration is always produced,even under hardware-constrained environments.

[0068] At step (214), upon successful validation, the processor (102)automatically generates a configuration for model training. The generatedconfiguration includes parameters such as selected learning rate, batch size,number of epochs, optimiser type, mixed-precision setting, and gradientaccumulation schedule. The configuration is serialised in a machine-readableformat (e.g., JSON, YAML) for direct deployment into standard trainingframeworks such as PyTorch or TensorFlow.

[0069] In another embodiment, the processor (102) maintains a parameterjustification log that records each parameter's derivation based on hardwarelimitations, dataset scale, and architecture-specific decisions. This log is madeaccessible through the user interface for audit, explainability, and compliancedocumentation purposes.

[0070] Furthermore, the processor (102) may retrieve templateconfigurations corresponding to the detected dataset domain from the knowledgebase. When no exact domain match is available, the processor (102) appliesinterpolation techniques to infer near-optimal hyperparameters by referencingconfigurations from adjacent domains or similar model scales.TECHNICAL ADVANCEMENT:

[0071] The method and system achieve significant improvement in thefunctioning of computing hardware, particularly GPU-based machine learningenvironments. By detecting available hardware resources using GPU-driver levelcalls, the processor performs real-time interrogation of hardware components suchas memory capacity, compute capability, and communication bandwidth. Unlikeconventional ML configuration tools that operate at framework level (e.g.,PyTorch or TensorFlow), the disclosed system interacts directly with low-leveldriver APIs (NVML, ROCm-SMI) to obtain hardware telemetry data. This low-level hardware interaction enables the system to optimize GPU allocation andscheduling, resulting in enhanced control over GPU memory management,reduction of idle GPU states, and improved overall processing throughput. Thiscreates a direct improvement in hardware utilization efficiency, lowering kernellaunch latency and memory fragmentation effects. The technical effect is ahardware-aware execution environment that ensures efficient utilization ofcomputational resources and prevents the hardware from entering memory-overflow or under-utilization states.

[0072] The feature of obtaining model identifiers and estimating memoryrequirements provides a predictive estimation of computational load based on themodel's parameter storage, optimizer state, and activation footprint. This allowsthe processor to calculate a maximum feasible batch size without exceedingphysical GPU memory limits, producing the direct technical advantage ofpreventing system crashes, reducing out-of-memory failures, and achieving stable,uninterrupted training execution. Further, the processor performs proactive GPUmemory reservation prior to training initiation, dynamically reallocating sharedbuffers based on inter-GPU bandwidth monitoring, resulting in economical use ofmemory and enhancing hardware reliability, which are recognized technicalimprovements in computer system performance.

[0073] The computation of adjusted learning rate and automatic gradientaccumulation introduces an adaptive scaling mechanism that dynamically tunescomputational effort in proportion to available GPU capacity. This featureprovides improved speed of convergence, faster execution, and bettersynchronisation between software-level optimisation and hardware-level resourceconstraints. By implementing a hardware-feedback control loop, the systemautomatically recalibrates optimiser hyperparameters in real time based on GPUtelemetry (temperature, throttling, and memory usage). It eliminates the latencyintroduced by manual configuration and reduces the number of training iterationsrequired to achieve a desired accuracy level, thereby increasing processing speedand computational efficiency.

[0074] The inclusion of validation through simulated GPU memoryallocation provides an internal simulation-based exception handling mechanism.By predicting execution failures before actual training begins, the system avoidsexecution interruptions, ensuring improved system stability and self-healingcapability. This predictive validation forms a core technical control process thatdirectly affects hardware safety and operational consistency, thereby improvingthe robustness of GPU-intensive computing environments. Additionally, thesystem supports predictive rollback, wherein previously stable configurations arereinstated when the simulation detects repeated memory overflow patterns, furtherimproving fault tolerance.

[0075] The fallback strategy, which automatically adjusts configurationparameters upon detecting a risk condition, contributes to system adaptability andagility. The capability to autonomously enable mixed-precision execution(FP16 / BF16), distribute workloads across multiple GPUs, or adjust input sizeresults in real-time adaptability to changing computational states. This enablesefficient data processing and real-time control of execution parameters, which areconcrete technical benefits in dynamic computing environments. The system thusbehaves as a self-optimizing control loop, continuously aligning trainingexecution with hardware conditions.

[0076] The utilization of GPU bandwidth and compute capability metricsin the selection of hyperparameters enables hardware-aware optimization oftraining configuration, improving data throughput efficiency and reducingmemory access times. This results in higher processing speed, lower latency, andenhanced real-time performance of ML training tasks, which are measurableimprovements in system performance. The ability to intelligently distributeworkloads across multiple GPUs using data parallelism and gradientcheckpointing further enhances compute throughput and memory efficiency,offering a substantial performance gain in large-scale training scenarios.

[0077] The logging and justification of parameter selections contribute toenhanced transparency and interpretability of system decisions, providing astructured interface that improves human-machine interaction. This enhances theusability and reliability of the computing system while ensuring that decisionsremain traceable for compliance and audit purposes. The ability to operate locallywithout transferring raw datasets adds an additional layer of security and dataintegrity, preventing unauthorized access and improving the confidentiality oftraining data.

[0078] Collectively, these features enable a technically improvedcomputing infrastructure that goes beyond a mere software algorithm ormathematical method. The claimed system directly influences the operation ofGPUs and memory subsystems, improving hardware utilization efficiency,execution reliability, memory economy, and computational speed. The integrationof predictive validation, dynamic adaptation, and fallback correction mechanismsensures continuous, real-time optimization of training performance.

[0079] . In a further embodiment, the system integrates a hardwarefeedback loop with predictive telemetry analytics that forecasts GPU saturationtrends and pre-emptively redistributes workloads, achieving hardware-leveloptimization not possible with static configuration methods.

[0080] These features, in combination, demonstrate that the claimedinvention achieves enhanced control of hardware operations, reducedcomputational failures, and increased system efficiency, thereby constituting aconcrete technical solution to the practical problems associated with memorymanagement, performance instability, and resource inefficiencies in machinelearning training systems.

[0081] Best Mode of Working with Real-World Example:

[0082] In an exemplary embodiment, the system and method for automaticconfiguration generation for machine learning training are implemented in adistributed GPU-based computing environment, such as a multi-GPU workstationor a cloud instance provisioned with NVIDIA A100 or AMD MI300 GPUs.During operation, the processor executes a configuration initialization modulewhich first performs hardware interrogation through CUDA or ROCm driver-level API calls to obtain parameters such as available GPU memory, computecapability, current GPU workload utilization, and inter-GPU communicationbandwidth. For example, in a system with four A100 GPUs, each with 40 GB ofmemory and 600 GB / s bandwidth, the system automatically detects these metricswithout requiring manual entry.

[0083] Subsequently, dataset metadata-such as task type (e.g., imageclassification or language modeling), dataset size (e.g., 500 GB), and domain-isreceived either through the local user interface or via a remote network source,such as a cloud data repository. Based on this input, the processor retrieves modelarchitecture identifiers from a locally stored knowledge base, for instance, aResNet-152 or Transformer-based model, and estimates model-specific memoryrequirements by accounting for parameter tensors, optimizer state variables, andintermediate activation maps. The processor then calculates the maximum feasiblebatch size, such as 128 samples per GPU, ensuring that the projected GPUmemory footprint does not exceed 95% of available memory capacity.

[0084] In one practical implementation, the system automatically adjuststhe learning rate according to the batch size by applying the scaling relationLR_adjusted = LR_base x √(Batch_size_new / Batch_size_base).

[0085] For example, if the base learning rate is 1e-3 for a batch size of 32,and the hardware allows a batch size of 128, the system computes a learning rateof 2e-3 to maintain gradient stability. Concurrently, gradient accumulation stepsare derived from the relationshipGA_steps = Effective_batch_size / Maximum_hardware_batch_size,ensuring that global batch size consistency is preserved even across varying GPUcapacities.

[0086] During validation, a simulated forward-pass is executed withdummy data to evaluate GPU memory allocation. If the simulation predicts amemory overflow, the system applies a fallback strategy-for instance, enablingmixed-precision (FP16 / BF16) computation, automatically reducing inputsequence length, or distributing training across available GPUs using dataparallelism. In an example case, if an FP32 execution exceeds available memoryby 10%, the processor automatically transitions to mixed-precision mode, therebyreducing memory usage by approximately 40% and preventing runtime failure.

[0087] The system also exhibits adaptability in dynamic environmentssuch as edge-AI inference clusters, where available hardware resources fluctuate.When a GPU becomes temporarily unavailable, the configuration engineautomatically re-estimates available resources and recalibrates parameters, such asreducing batch size or adjusting data parallelism degree, ensuring uninterruptedoperation. This hardware-aware adaptability significantly improves trainingefficiency and throughput in heterogeneous GPU infrastructures.

[0088] Furthermore, the system maintains data security by performingconfiguration generation locally, thereby avoiding exposure of sensitive datasetsto cloud servers. For example, in a hospital deploying deep learning for medicalimage analysis, the invention enables fully local auto-configuration withouttransmitting patient data outside the local network-ensuring compliance withprivacy regulations while optimizing computational performance.

[0089] The user interface component further contributes to practicalusability by presenting justifications for each configuration decision-such as"learning rate increased due to higher available GPU memory"-therebyenhancing transparency and enabling informed operator control. This allowsresearchers or engineers to monitor and override parameters where necessary,leading to improved human-machine collaboration.

[0090] In real-world deployments, such as automated model trainingpipelines in AI-based autonomous driving or large-scale natural language modeltraining, the claimed invention substantially reduces manual configuration timefrom hours to seconds, minimizes training interruptions caused by memoryoverflow, and ensures maximal hardware utilization. Consequently, the inventionachieves faster model convergence, improved system reliability, and efficientresource utilization, demonstrating significant technical advancement in terms ofhardware-aware optimization, real-time adaptability, and fault-tolerant machinelearning execution.

Claims

1. A method (200) for automatically generating a configuration for training a machine learning model, the method executed by at least one processor of a computing device having one or more graphics processing units (GPUs), memory, a user interface (108) and a network interface, the method (200) comprising: detecting (202), by the processor (102) using GPU-driver level calls, available hardware resources including a memory (104) capacity of each GPU, a number of GPUs, GPU compute capability, current workload status and communication bandwidth; receiving (204), via the user interface (108) or over the network interface, dataset metadata including a task type, a domain and a dataset size; obtaining (206), by the processor (102), a model identifier and estimating model memory usage including parameter storage, optimizer state and forward-activation memory based on the model architecture; calculating (208), by the processor (102) using the GPU memory capacity, a maximum feasible batch size that can be executed without exceeding available GPU memory; selecting (210), by the processor (102) from a knowledge base stored in memory (104), candidate hyperparameters including learning rate, batch size and number of epochs based on the detected hardware resources, dataset metadata, and model memory estimates; validating (212), by the processor (102), a simulated GPU memory allocation for the candidate configuration to detect a memory (104) overflow condition; generating (214) automatically, by the processor (102), a configuration for training a machine learning model.

2. The method (200) as claimed in claim 1, wherein before validating, the method further comprises: computing, by the processor (102), (a) an adjusted learning rate based on: LR_adjusted = LR_base x √(Batch_size_new / Batch_size_base), and (b) gradient accumulation steps based on: GA_steps = Effective_batch_size / Maximum_hardware_batch_size.

3. The method (200) as claimed in claim 1, wherein after validating, the method further comprises: automatically applying a fallback strategy when validation indicates a risk condition, wherein the fallback strategy comprises at least one of: increasing gradient accumulation steps, reducing sequence length or input size, enabling mixed-precision execution (FP16 / BF16), reducing batch size, or recommending distributed training across the GPUs.

4. The method (200) as claimed in claim 1, wherein detecting hardware resources includes invoking CUDA or ROCm API calls for querying real-time GPU memory utilization.

5. The method (200) as claimed in claim 1, wherein simulating memory allocation comprises running a dummy forward-pass on the GPUs to estimate peak memory consumption.

6. The method (200) as claimed in claim 1, wherein enabling mixed-precision execution is performed only when the processor (102) identifies GPU tensor core support.

7. The method (200) as claimed in claim 1, wherein: the processor (102) logs parameter justification including hardware limitations, dataset scale and architecture-specific decisions for audit and compliance display on the user interface (108); and the processor (102) retrieves template configurations corresponding to the dataset domain and applies interpolation when no exact match is found.

8. A system (100) for automatically generating a configuration for training a machine learning model, the system (100) comprising: at least one processor (102); a memory (104) storing a knowledge base of domain-specific hyperparameter templates; one or more GPUs (106-1, 106-2, ..., 106-N) electrically connected with the processor (102) and providing hardware-accelerated model execution capability; a user interface (108) configured to receive user inputs and display recommended configurations; and a network interface to communicate with remote resources; wherein the at least one processor (102) is configured to: retrieve hardware resource information from the GPUs, the hardware resource information including free GPU memory, compute capability and bandwidth; obtain dataset metadata and a model identifier from the user interface (108) or the network interface; compute model memory requirements and maximum feasible batch size; select recommended hyperparameters based on the dataset metadata, model memory requirements and hardware resources using the knowledge base stored in the memory (104); simulate GPU memory usage to validate the configuration; and automatically implement fallback adjustments to produce a validated configuration executable on the available GPUs.

9. The system (100) as claimed in claim 8, wherein the processor (102) : estimates per-sample memory footprint by dividing total predicted activation memory by computed tensor shape; and detects a security constraint in the dataset metadata and automatically selects a configuration preserving data locality without transferring raw dataset through the network interface.

10. The system (100) as claimed in claim 8, wherein the processor (102) is configured to allocate workloads across multiple GPUs using data parallelism and gradient checkpointing to optimize memory consumption and compute throughput.