Dynamic optimization method, system and storage medium for text-to-graph model

Abstract

This application proposes a dynamic optimization method, system, and storage medium for a Wensheng graph model. The method parses the Wensheng graph model to identify sub-model components and determine their execution order, pre-calculates the static weights of each component loaded into GPU memory, measures the peak memory usage of each component through forward computation under multi-dimensional operating parameters, and generates a runtime memory lookup table. During inference, components are scheduled sequentially according to their execution order. After the current component completes its forward computation, the next component to be executed is pre-loaded or unloaded on demand based on the static weights and the runtime memory lookup table to update the set of loaded components in GPU memory. In principle, on-demand memory scheduling is achieved through the joint determination of static weights and measured peak values; in effect, it can reduce the peak GPU memory usage per inference cycle, reduce the risk of memory overflow, and improve deployment and operation capabilities on devices with limited GPU memory.

Description

Technical Field

[0001] This application relates to the field of generative artificial intelligence technology, and in particular to a dynamic optimization method, system, and storage medium for a text-based graph model. Background Technology

[0002] In recent years, text-based image processing technology based on diffusion models has developed rapidly. However, with the continuous expansion of model size (e.g., the SDXL model has billions of parameters), existing technologies simultaneously load massive models such as CLIP, UNET, and VAE into video memory, requiring extremely large video memory resources (typically exceeding 12GB, or even 20GB). This significantly raises the hardware barrier, making it impossible for most consumer-grade graphics cards to run such large models, severely limiting the technology's widespread adoption and application scenarios. This "full loading, continuous persistence" approach exposes serious technical flaws.

[0003] Although the three sub-models must be executed sequentially in the inference process, only the currently executing sub-model is active for most of the time, while the others remain idle. However, in current technologies, these idle models still occupy valuable GPU memory. This inefficient use of critical computing resources is essentially a huge waste of resources, making it impossible to deploy larger models or process more tasks simultaneously with limited hardware resources.

[0004] Therefore, there is an urgent need for a technology that can accurately characterize runtime memory requirements and schedule the loading and unloading of sub-models on demand during the inference phase, thereby reducing the risk of memory overflow and enabling efficient deployment and reliable operation of large text graph models with limited memory resources. Summary of the Invention

[0005] This application provides a dynamic optimization method for text graph models to solve the problems in the prior art where large text graph models suffer from excessively high peak memory usage, low memory utilization, and memory overflow risk during the inference phase due to the continuous resident of all sub-models in GPU memory. This makes it impossible to achieve efficient deployment and reliable operation of large text graph models with limited memory resources.

[0006] Firstly, this application provides a dynamic optimization method for a text-based graph model, the method comprising:

[0007] The structure of the Wensheng graph model is analyzed, multiple independent sub-model components are identified, the execution order of each sub-model component is determined, and the static weights required for each sub-model component to be loaded into the GPU memory are pre-calculated.

[0008] By actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table;

[0009] Upon receiving an inference request containing runtime parameters, each sub-model component is sequentially scheduled to perform forward computation according to the execution order described above.

[0010] Based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes the forward calculation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components. The set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory.

[0011] Optionally, the process of analyzing the structure of the text-based graph model, identifying multiple independent sub-model components, determining the execution order of each sub-model component, and pre-calculating the static weights required for each sub-model component to be loaded into GPU memory includes:

[0012] Read the storage file of the text graph model, parse the structure of the text graph model, and identify multiple independent sub-model components, including a text encoding model, a diffusion denoising model, and a latent encoding model;

[0013] Based on the total number of parameters of each sub-model component, the number of bytes corresponding to the data type of each parameter, and the estimated runtime overhead, the static weight of each sub-model component is calculated. The static weight is the amount of memory required for each sub-model component to be loaded into the GPU memory.

[0014] Optionally, the step of actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table includes:

[0015] Identify key configuration parameters that affect GPU memory usage, including prompt word sequence length, batch size, and generated image resolution;

[0016] Select multiple parameter combinations from the multidimensional parameter space composed of the key configuration parameters;

[0017] Each sub-model component is loaded into the GPU memory in sequence, and forward calculation is performed under the parameter combination to measure the peak memory usage.

[0018] The peak memory usage is monitored and recorded through the GPU performance monitoring interface, and the sub-model component is unloaded from the GPU memory.

[0019] The runtime memory lookup table is generated based on the correspondence between all recorded parameter combinations and peak memory usage.

[0020] Optionally, based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes its forward computation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components. The memory management operation includes a preloading strategy, which includes:

[0021] Based on the running parameters of the inference request, the runtime video memory lookup table is queried to obtain the peak video memory usage required by the next sub-model component to be executed;

[0022] Query the current amount of free GPU memory;

[0023] If the amount of free video memory is greater than or equal to the static weight of the next sub-model component to be executed, then the next sub-model component to be executed is preloaded from CPU memory to GPU video memory and added to the set of loaded sub-model components.

[0024] Optionally, the memory management operation further includes a dynamic unloading strategy. If the amount of free GPU memory currently available is less than the peak memory usage required by the next sub-model component to be executed, then the dynamic unloading strategy is executed. The dynamic unloading strategy includes:

[0025] The sub-model component that has recently completed forward computation is unloaded from GPU memory and removed from the set of loaded sub-model components until available memory meets the peak memory usage required by the next sub-model component to be executed.

[0026] Optionally, before receiving an inference request containing runtime parameters, the method further includes:

[0027] The identified text encoding model, diffusion denoising model, and latent encoding model are all preloaded into CPU memory and kept resident.

[0028] The memory management operation is performed based on a fixed execution order of the text encoding model, the diffusion denoising model, and the latent encoding model.

[0029] Optionally, the peak memory usage includes the memory occupied by the model weights, the memory allocated by the runtime memory pool of the computing framework, and the memory occupied by the intermediate activation feature maps generated during the forward computation process.

[0030] Secondly, this application provides a dynamic optimization system for a text-based graph model, the system comprising:

[0031] The parsing model module is used to parse the structure of the text graph model, identify multiple independent sub-model components, determine the execution order of each sub-model component, and pre-calculate the static weights required for each sub-model component to be loaded into the GPU memory.

[0032] The video memory lookup table generation module is used to measure the peak video memory usage of each sub-model component by actually running each sub-model component under different combinations of running parameters, map the running parameters to the peak video memory usage, and generate a runtime video memory lookup table.

[0033] The forward computation module is used to, upon receiving an inference request containing running parameters, sequentially schedule each sub-model component to perform forward computation according to the execution order.

[0034] The memory management execution module is used to perform memory management operations on the next sub-model component to be executed after the currently executed sub-model component has completed its forward calculation, based on the static weight size and the runtime memory lookup table, so as to update the set of loaded sub-model components, wherein the set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory.

[0035] Thirdly, this application provides a smart terminal, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0037] The advantages of this application compared to existing technologies are as follows: By pre-calculating the static weights of each sub-model component and combining this with a runtime memory lookup table for on-demand scheduling, sub-models are loaded into GPU memory only when needed, thus avoiding resource waste caused by the entire model remaining resident in GPU memory and significantly reducing peak memory usage per inference cycle. Accurate prediction of memory requirements based on a measured runtime memory lookup table makes GPU memory allocation more aligned with actual task load, improving the dynamic utilization efficiency of memory resources. A dynamic unloading strategy automatically unloads completed sub-model components when memory is insufficient, ensuring that subsequent components can be loaded and executed smoothly, reducing the risk of memory overflow. By adopting an asynchronous preloading strategy, the next sub-model component is transferred to GPU memory in parallel during the current sub-model computation phase, masking I / O latency and improving inference throughput performance. Through a preloading judgment and unloading decision coordination mechanism, memory allocation can be dynamically adjusted during inference, ensuring stable and continuous inference tasks. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0041] Figure 1 A flowchart illustrating a dynamic optimization method for a text-based graph model provided in this application embodiment;

[0042] Figure 2 This application provides a schematic diagram of the structure of a dynamic optimization system for a text-based graph model.

[0043] Figure 3 This is a schematic diagram of the structure of a smart terminal provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0046] like Figure 1 As shown, Figure 1 A flowchart illustrating a dynamic optimization method for a text-based graph model provided in this application embodiment is shown below. The method includes:

[0047] S100 analyzes the structure of the Wensheng graph model, identifies multiple independent sub-model components, determines the execution order of each sub-model component, and pre-calculates the static weights required for each sub-model component to be loaded into the GPU memory.

[0048] In this embodiment, the structure file of the Wensheng graph model is first read, its network topology and module dependencies are parsed, and sub-model components that can be loaded and executed independently are identified. A sub-model component refers to a model unit with an independent parameter file, clearly defined input / output interfaces, and the ability to perform forward computation independently on a computing device.

[0049] After parsing, based on the number of parameters, data types, and video memory storage format of each component, the static weight size required to load it into GPU video memory is calculated. The static weight size refers to the fixed capacity occupied by the model weight file in video memory after loading, which is used to represent the video memory requirement of the sub-model when it is not running. This step provides basic data for subsequent video memory allocation and dynamic scheduling.

[0050] S200: By actually running each of the sub-model components under different combinations of operating parameters, the peak video memory usage of each sub-model component is measured, the operating parameters are mapped to the peak video memory usage, and a runtime video memory lookup table is generated.

[0051] In this embodiment, to accurately describe the model's memory requirements under different operating parameters, the system performs runtime memory sampling on each sub-model component and constructs a runtime memory lookup table. The runtime memory lookup table is a data structure used to store the correspondence between operating parameters and peak memory usage, reflecting the model's memory usage characteristics under different inference conditions. Operating parameters refer to task configurations that affect the model's memory usage, such as prompt word sequence length, batch size, and generated image resolution.

[0052] In this step, the inference test of the sub-model is performed under multiple combinations of running parameters, its peak video memory usage is measured and the corresponding relationship is recorded, so as to obtain a runtime video memory lookup table that can be used for querying in the subsequent inference stage.

[0053] S300: Upon receiving an inference request containing running parameters, the system sequentially schedules each sub-model component to perform forward computation according to the execution order.

[0054] In this embodiment, upon receiving an inference request containing runtime parameters, the forward computation of each sub-model is executed sequentially according to a predetermined sub-model execution order. Forward computation refers to the unidirectional inference process from input to output in a neural network, without involving gradient backpropagation. During execution, the GPU memory occupancy status is determined in real time based on the aforementioned static weight values ​​and runtime memory lookup table information, and dynamic loading and unloading operations are performed between GPU memory and CPU memory to control the set of sub-model components currently residing in the GPU memory.

[0055] Among them, CPU memory and GPU memory correspond to the computer's main memory and the graphics card's video memory, respectively. The system exchanges data through a bus, thereby dynamically managing the model components without affecting the inference order.

[0056] S400, based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes the forward calculation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components, wherein the set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory.

[0057] In this embodiment, after the forward computation of the current sub-model component is completed, the system performs memory management operations based on the memory requirements of the next sub-model to be executed and the current free capacity of the GPU memory. Memory management includes preloading the next component from CPU memory when there is sufficient memory, or unloading components that have completed computation when there is insufficient memory.

[0058] The set of loaded sub-model components represents the status of all sub-model components currently stored in GPU memory. As the inference process progresses, the system updates this set in real time to reflect the latest GPU memory usage. Through this dynamic management mechanism, the present invention effectively reduces peak GPU memory usage while ensuring the continuity of the inference task, achieving efficient and stable inference execution of the Wensheng graph model on memory-constrained devices.

[0059] Furthermore, the structure of the parsed text-based graph model is analyzed, multiple independent sub-model components are identified, the execution order of each sub-model component is determined, and the static weights required for each sub-model component to be loaded into GPU memory are pre-calculated, including:

[0060] S101, Read the storage file of the text graph model, parse the structure of the text graph model, and identify multiple independent sub-model components, including a text encoding model, a diffusion noise reduction model, and a latent encoding model.

[0061] In this embodiment of the application, after the optimization system is started and before the formal processing of the generation request, the specified diffusion model (e.g., the SDXL model) is analyzed. This step is not dependent on the GPU and can be completed on the CPU.

[0062] Specifically, the system first reads the model's storage file (such as .safetensors or .ckpt files), parses its internal structure, and identifies and confirms the independent sub-model components contained within the model, including but not limited to CLIP, UNET, and VAE models. For each identified sub-model, the system pre-calculates the memory size required after loading it into the GPU memory. By reading the model's structure description file, the system parses the overall computational graph structure of the model, identifies functional units that can be independently loaded and executed, and defines them as sub-model components.

[0063] In this embodiment, the sub-model components include three types: a text encoding model, a diffusion denoising model, and a latent encoding model. The text encoding model encodes text into feature vectors that the text-to-image model can understand, controlling the direction of diffusion denoising to generate an image that conforms to the text description. The diffusion denoising model interprets the text encoding vectors and, guided by them, performs a cyclic denoising process in a specific direction to generate an image close to the text description. The latent encoding model transforms the image from pixel space to latent space, or vice versa, significantly reducing the computational load of the diffusion process. During the parsing process, the system establishes the calling order and data dependencies between the sub-model components, providing an execution sequence for subsequent inference scheduling.

[0064] S102, calculate the static weight of each sub-model component based on the total number of parameters of each sub-model component, the number of bytes corresponding to the data type of each parameter, and the estimated runtime overhead. The static weight is the amount of memory required for each sub-model component to be loaded into the GPU memory.

[0065] In this embodiment, the total number of model parameters in the sub-model component is first calculated, including the total number of all weights and biases. The storage space required for each parameter is calculated based on the type identifier in the model file, and the storage space for all parameters is summed to obtain the basic weight capacity of the model. That is, the number of bytes occupied by each parameter is determined according to its data type (e.g., FP32, FP16, BF16) (e.g., FP16 occupies 2 bytes).

[0066] The static weight size parameter directly reflects the memory consumption of the model weight file and is an important basis for dynamic memory scheduling. This value is obtained by statistically analyzing three parts of information: the total number of parameters for each sub-model component, determining the data type of the parameters, and evaluating runtime overhead. The peak memory required to load each sub-model is estimated using the formula: Memory Usage ≈ Total Number of Parameters × Number of Bytes Occupied by Each Parameter + Runtime Overhead (such as intermediate activation feature maps, working buffers, etc.).

[0067] Through the above analysis and calculation process, the system can obtain the structural information, execution order, and corresponding static weights of each sub-model component before the inference stage. This information provides basic data support for subsequent memory scheduling and dynamic loading strategies, enabling the system to load and unload memory on demand based on the actual memory capacity when executing inference tasks. This avoids overflow problems caused by excessively high memory peaks and improves the overall utilization efficiency of GPU memory.

[0068] Furthermore, the step of actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table includes:

[0069] S201, Identify key configuration parameters that affect GPU memory usage, including prompt word sequence length, batch size, and generated image resolution;

[0070] S202, Select multiple parameter combinations in the multidimensional parameter space composed of the key configuration parameters;

[0071] S203, load each sub-model component into the GPU memory in sequence, and perform forward calculation under the parameter combination to measure the peak memory usage;

[0072] S204, monitor and record the corresponding peak video memory usage through the GPU performance monitoring interface, and unload the sub-model component from the GPU video memory;

[0073] S205, Generate the runtime memory lookup table based on the correspondence between all recorded parameter combinations and peak memory usage.

[0074] In this embodiment, to accurately grasp the memory usage characteristics of each sub-model component under different operating conditions, the system performs memory sampling tests during the model initialization phase and constructs a runtime memory query table. The runtime memory query table is a data structure used to store the correspondence between combinations of operating parameters and peak memory usage. Its function is to provide a basis for memory scheduling decisions during the inference phase, enabling the system to dynamically predict memory requirements based on task configuration.

[0075] First, the system identifies key runtime parameters that affect memory usage. Runtime parameters refer to task configuration elements that directly influence memory usage during inference. Key parameters include, but are not limited to: Token Length: Directly affects the input size and activation tensor size of the CLIP text encoder. Batch Size: Memory usage for most sub-models is approximately linearly positively correlated with batch size. Image Resolution: Directly affects the input / output feature map size of the UNet model and VAE decoder, and is one of the main determinants of memory usage. Sampling Steps: While usually not affecting peak memory usage per inference iteration, it may affect the cumulative loop overhead of UNet (this can be fixed by controlling variables).

[0076] Within a multi-dimensional parameter space comprised of key operational parameters, several representative parameter combinations are selected for testing. Each parameter combination corresponds to a specific inference task scenario, such as a combination of short prompts, single-sample input, and low-resolution output, or a combination of long prompts, multi-sample batches, and high-resolution output. By running the model under these different combinations, the memory usage patterns of the model under varying workload conditions can be comprehensively obtained.

[0077] The system selects a series of representative parameter combinations from the above parameter space, such as (batch_size=1, resolution=512x512, token_length=77), (batch_size=2, resolution=1024x1024, token_length=77).

[0078] Subsequently, under controlled conditions (ensuring that the GPU memory is initially free), the following operations are performed automatically and sequentially: (1) clear the GPU cache; (2) load the target sub-model (such as UNet) onto the GPU; (3) run the forward inference of the sub-model once or multiple times using the currently selected parameter combination; (4) during the run, monitor and record the peak GPU memory consumed by the sub-model under this specific parameter combination in real time, which is usually achieved by calling the performance monitoring API provided by the GPU driver (such as NVIDIA's nvml library); (5) after the run is completed, unload the model from the GPU, clean up the memory, and prepare for the next measurement.

[0079] During testing, the system sequentially loads each sub-model component into GPU memory and performs forward computation. Forward computation refers to the process of data inference from the input layer to the output layer without backpropagation, typically used to generate model output or compute feature results. During this process, the system calls GPU performance monitoring interfaces (such as the memory management APIs provided by CUDA or ROCm) to record the dynamic usage of GPU memory in real time and detect peak memory usage, obtaining the peak GPU memory usage for each sub-model component under that parameter combination.

[0080] After completing a test, the system unloads the current sub-model component from the GPU memory, releasing memory resources to allow testing of the next sub-model component. By summarizing the test results of all parameter combinations and sub-model components, the system obtains the correspondence between multiple sets of operating parameters and peak memory usage, and stores these data entries in the form of tables or index structures, forming a runtime memory lookup table.

[0081] After traversing all preset parameter combinations, the system organizes and persists the collected parameter combination-peak memory usage mapping relationship into a runtime memory lookup table. This table can be a multidimensional array, a dictionary, or a small embedded database. Subsequently, in actual inference tasks, the scheduler does not need to perform time-consuming tests again; it can simply look up this table based on the specific parameters of the current request (e.g., batch_size=1, resolution=1024x1024) to obtain a highly accurate expected memory usage value.

[0082] The generated runtime memory lookup table not only records the peak memory usage of each sub-model under different operating parameters, but also includes auxiliary indicators such as computation time and memory bandwidth utilization, which are used for subsequent scheduling strategy optimization and expansion of multi-model shared scheduling. This lookup table is invoked during the inference phase. When the system receives an inference request, it can quickly look up the corresponding memory requirements based on the input parameters, and then combine this with static weight information to determine whether to perform preloading or unloading operations, thereby achieving accurate prediction and on-demand allocation of memory.

[0083] Through the runtime memory lookup table generation mechanism in this embodiment, the system can establish a quantitative model of memory requirements before inference, avoiding memory management from relying on experience estimation or fixed threshold settings, thereby significantly improving the accuracy and execution stability of dynamic scheduling and providing a reliable basis for subsequent preloading and unloading strategies.

[0084] Furthermore, based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes its forward computation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components. The memory management operation includes a preloading strategy, which includes:

[0085] A1. Based on the running parameters of the inference request, query the runtime memory lookup table to obtain the peak memory usage required by the next sub-model component to be executed.

[0086] A2, query the current amount of free GPU memory;

[0087] A3. If the amount of free video memory is greater than or equal to the static weight of the next sub-model component to be executed, then the next sub-model component to be executed is preloaded from CPU memory to GPU video memory and added to the set of loaded sub-model components.

[0088] In this embodiment, a preloading judgment process is triggered after each sub-model component completes its forward computation and releases its immediate activation. This process determines whether to preload the next sub-model component to be executed into the GPU memory. The preloading judgment process first performs a memory cleanup operation. This operation includes calling the memory reclamation / release interface provided by the computing framework or device driver to reclaim releasable cache and temporary buffers, and, where possible, triggering memory defragmentation or merging to obtain a larger contiguous block of available memory (hereinafter referred to as "available memory capacity"). "Available memory capacity" refers to the memory capacity that can be used for new data loading and meets the requirements of a single continuous allocation after memory cleanup is completed.

[0089] Subsequently, based on the runtime parameters (such as prompt sequence length, batch size, generated image resolution, etc.) in the received inference request, the system retrieves the peak memory requirement of the next sub-model component to be executed corresponding to the runtime memory lookup table, which is used to estimate the maximum memory usage of the next sub-model component during runtime. The system then reads the current contiguous available memory capacity of the GPU and compares this capacity with the static weight size of the next sub-model component to be executed, where the static weight size is the fixed amount of memory occupied by the weight itself after the sub-model component is loaded into the GPU memory.

[0090] When the contiguous available GPU memory capacity is greater than or equal to the static weight of the next sub-model component to be executed, the system initiates an asynchronous transfer operation to preload the next sub-model component from the CPU memory to the GPU memory. This asynchronous transfer uses a host-to-device non-blocking data channel (e.g., DMA-based bus transfer or an asynchronous copy interface provided by the framework), and allows subsequent computation of the current sub-model component to continue during the transfer, thereby achieving parallel transfer and computation and masking data migration delays. After preloading is complete, the system marks the sub-model component as loaded and updates the status information of the set of loaded sub-model components, facilitating quick querying and use by the subsequent scheduling module.

[0091] If the continuous available video memory capacity is less than the static weight size, the preloading judgment process will return a decision not to perform preloading, and will attempt to preload again after the scheduling module triggers the unloading decision process to release more video memory.

[0092] Based on this example, assume the inference task requires sequential execution of the sub-model component chain: ABC-D. All models have been pre-loaded into CPU memory, and their M_weight and M_peak information are known. The following steps are performed:

[0093] (1) The scheduler loads Model_current (e.g., the currently executing sub-model component A) from its CPU memory to GPU memory.

[0094] (2) The system runs forward inference computation of Model_current to complete its function in this round of tasks.

[0095] (3) After the inference computation is completed, Model_current_A remains in GPU memory. At this time, the scheduler immediately begins to prepare for loading the next model (Model_next_B, such as the next sub-model component model B to be executed):

[0096] a. Query VRAM requirements: Based on the specific parameters of the current inference task (such as batch size, resolution, etc.), query the pre-generated runtime VRAM query table to obtain the peak VRAM M_peak_next_B required for Model_next_B to run.

[0097] b. Evaluate the current VRAM: The scheduler calls relevant instructions such as torch.cuda.empty_cache() to perform VRAM cleaning and release fragmented caches generated by the memory pool mechanisms of frameworks such as PyTorch. Then, query the current free VRAM Free_mem of the GPU.

[0098] c. Decision-making and preloading: Compare M_weight_next_B (the weight size of Model B) with the current Free_mem.

[0099] If Free_mem >= M_weight_next_B, it means that the current free VRAM is sufficient to preload the weights of the next model. The scheduler immediately initiates an asynchronous transfer operation to preload Model_next_B from CPU memory to GPU VRAM. This operation is carried out in parallel with the subsequent steps to hide the I / O latency.

[0100] If Free_mem < M_weight_next: It means that the current VRAM is insufficient and preloading is not performed temporarily.

[0101] Furthermore, the memory management operation further includes a dynamic unloading strategy. If the free VRAM capacity of the current GPU VRAM is less than the peak VRAM occupancy required by the next sub-model component to be executed, the dynamic unloading strategy is executed. The dynamic unloading strategy includes:

[0102] Unload the sub-model component that has recently completed forward calculation from the GPU VRAM and delete it from the set of loaded sub-model components until the available VRAM meets the peak VRAM occupancy required by the next sub-model component to be executed.

[0103] In this embodiment, the dynamic unloading strategy is triggered when the preloading judgment process determines that the continuous available VRAM capacity of the GPU VRAM is insufficient to meet the peak VRAM requirements of the next sub-model component to be executed, in order to release sufficient VRAM space for subsequent component loading and execution. This strategy includes the following steps.

[0104] First, the system identifies candidate unloadable objects. Candidate objects are prioritized as sub-model components that have completed forward computation and are no longer needed in the current inference process; "no longer needed" means that the sub-model component will not be called in subsequent execution sequences or its function has been replaced by downstream results. The system can determine this attribute based on the sub-model execution sequence, the control flow information of the current inference request, or the scheduler's dependency analysis results. The selection of candidate unloadable objects can also consider their static weights and unloading overhead, prioritizing components that release the most GPU memory with minimal overhead.

[0105] Secondly, the system unloads the selected candidate components from the GPU memory back to the CPU memory in reverse order of the forward computation of the sub-model components (i.e., the most recently completed components are prioritized). The unloading operation includes: calling the memory copy / release interface provided by the deep learning framework or device driver to migrate the weights of the target sub-model components from the GPU memory back to the storage area of ​​the CPU memory, and releasing the corresponding GPU memory usage after the migration is completed. After each unloading, the system performs a memory cleanup operation to reclaim any remaining temporary buffers and merge scattered free areas as much as possible, thereby updating the continuous available GPU memory capacity.

[0106] Subsequently, the system re-detects the contiguous available GPU memory capacity and determines whether this capacity meets the peak memory requirements of the next sub-model component to be executed. If it does, the unloading process ends and preloading or direct loading of the next sub-model component is triggered; if it still does not meet the requirements, the system continues to select and unload the next completed candidate component in reverse order until the contiguous available GPU memory capacity is sufficient to meet the peak memory requirements or there are no more candidate components available for unloading. To avoid unnecessary overhead caused by frequent unloading / loading, the system can apply a threshold or cost function (e.g., considering the probability and cost of reloading after unloading) when selecting unloading objects to balance the unloading benefits with future overhead.

[0107] Based on this example, assume the inference task requires sequential execution of the sub-model component chain: ABC-D. All models have been pre-loaded into CPU memory, and their M_weight and M_peak information are known. The following steps are then performed:

[0108] (1) After the preloading operation (if it occurs), the scheduler needs to clear enough space for the run of Model_next. It needs to ensure that: Free_mem >= M_peak_next_B.

[0109] a. Check running conditions: The scheduler re-evaluates the current free video memory Free_mem (this value has been reduced if preloading has just occurred).

[0110] b. Condition satisfied: If Free_mem >= M_peak_next_B, it means that no model needs to be unloaded and Model_next_B is already ready to run.

[0111] c. Condition not satisfied: If Free_mem < M_peak_next_B, the scheduler starts the unloading process:

[0112] (2) Unload the completed models: The scheduler first unloads the models that have been executed and are no longer needed subsequently (such as Model_current, i.e., Model A) from the GPU memory and moves them back to the CPU memory. Then, torch.cuda.empty_cache() is executed again and Free_mem is checked.

[0113] (3) Recursive unloading: If the available video memory is still less than M_peak_next after unloading Model_current, the scheduler will continue to unload the previous models in reverse order of the execution sequence (i.e., starting from the model that completed farthest) until Free_mem >= M_peak_next is satisfied. (For example, when running Model D, it may be necessary to unload Model A and B).

[0114] (4) Set Model_next as the new Model_current. Repeat the steps of the preloading and dynamic unloading strategies until all sub-models are executed.

[0115] Furthermore, before receiving an inference request containing running parameters, the method further includes:

[0116] Preload all the identified text encoding model, diffusion denoising model, and latent encoding model into the CPU memory and keep them resident;

[0117] Perform the memory management operation based on the fixed execution order of the text encoding model, diffusion denoising model, and latent encoding model.

[0118] Before receiving an inference request containing running parameters, the system preloads the sub-model components such as the identified text encoding model, diffusion denoising model, and latent encoding model from persistent storage into the host memory (i.e., the CPU memory) and keeps their parameter copies resident in the CPU memory for quick transfer to the GPU memory when needed subsequently.

[0119] The term "maintaining resident status" refers to allocating contiguous or addressable storage areas in CPU memory for the weights and related metadata of the sub-model components and maintaining an effective mapping. This ensures that subsequent host-to-device (Host→Device) transfers only require copying these resident memory areas without further disk reads or decoding. This preloading operation can be completed during system startup or model loading initialization, and the base address and byte length of each sub-model component in CPU memory are recorded as input parameters for subsequent asynchronous transfers.

[0120] Furthermore, this embodiment implements memory management operations based on a fixed execution order of the text encoding model, diffusion denoising model, and latent encoding model: during inference, the scheduling module triggers the forward computation of each sub-model in a cyclical manner according to this fixed order; simultaneously, the memory management module performs on-demand preloading and unloading between CPU memory and GPU memory based on the static weight size and the runtime memory lookup table. The fixed execution order refers to the fact that the running order of each sub-model component in a single inference process is a determined sequence of the three types of components. This sequence is determined by the model structure parsing step and used for scheduling strategy decisions, thereby simplifying dependency analysis and reducing runtime control overhead. The preloading and unloading actions in this embodiment can employ asynchronous transmission and parallel scheduling to complete the transmission of the next component in parallel while the current component is executing, reducing inference latency and improving memory utilization efficiency.

[0121] Furthermore, the peak memory usage includes the memory occupied by the model weights, the memory allocated by the runtime memory pool of the computing framework, and the memory occupied by the intermediate activation feature maps generated during the forward computation process.

[0122] When constructing the runtime memory lookup table or performing memory measurement and scheduling calculations, the "peak memory usage" consists of the following parts, which are measured and recorded separately: First, the static storage amount occupied by the model weights of the sub-model components in the GPU memory (i.e., the aforementioned static weight size); Second, the memory allocated by the runtime memory pool of the computing framework, i.e., the memory allocated and maintained in the GPU memory by the deep learning framework (e.g., PyTorch, TensorFlow, etc.) for actions such as execution graph, memory pool management, working buffer, and temporary tensor allocation during forward computation; Third, the memory occupied by the intermediate activation feature maps generated during the forward computation process, including intermediate tensors output by convolution / attention layers, as well as activation copies that exist simultaneously in batch processing parallel or multi-branch structures.

[0123] When measuring and recording peak video memory usage, the system monitors and aggregates the three types of video memory consumption mentioned above: Before loading sub-model components and starting forward computation, the system records the baseline video memory already occupied (mainly model weights). During forward computation, the system samples the video memory usage curve in real time through the GPU performance monitoring interface. After computation, the system calculates the total video memory value at the peak point and its component increases, thereby obtaining component estimates for model weights, runtime memory pool usage, and activation feature map usage. This component measurement facilitates providing more granular video memory requirement information for each parameter combination in the runtime video memory lookup table, supporting the scheduling module in considering the controllability of different video memory consumption sources when making pre-loading judgments and unloading decisions (for example, if unloading weights alone cannot reduce the instantaneous peak caused by activation, a strategy of unloading more completed components or adjusting running parameters should be prioritized).

[0124] By modeling and recording peak video memory usage in a runtime video memory lookup table, this embodiment improves the accuracy of video memory demand prediction and provides an operable measurement basis for subsequent preloading, asynchronous copying and unloading strategies.

[0125] like Figure 2 As shown, Figure 2 This application provides a schematic diagram of the structure of a dynamic optimization system for a text-based graph model, the system comprising:

[0126] The parsing model module 510 is used to parse the structure of the text graph model, identify multiple independent sub-model components, determine the execution order of each sub-model component, and pre-calculate the static weight size required for each sub-model component to be loaded into the GPU memory.

[0127] The video memory lookup table generation module 520 is used to measure the peak video memory usage of each sub-model component by actually running each sub-model component under different combinations of running parameters, map the running parameters to the peak video memory usage, and generate a runtime video memory lookup table.

[0128] The forward calculation module 530 is used to, upon receiving an inference request containing running parameters, sequentially schedule each sub-model component to perform forward calculations according to the execution order.

[0129] The memory management execution module 540 is used to perform memory management operations on the next sub-model component to be executed after the currently executed sub-model component has completed forward calculation, based on the static weight size and the runtime memory lookup table, so as to update the set of loaded sub-model components, wherein the set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory.

[0130] like Figure 3 As shown, Figure 3This is a schematic diagram of the structure of a smart terminal provided in an embodiment of this application. The smart terminal 700 of this embodiment includes: a processor 710 ( Figure 3 The above describes the process of implementing the steps in the method embodiment. The memory 720 is shown as only one, and the processor 710 is a memory 720 and a computer program 721 stored in the memory 720 and executable on the memory 720. The memory 720 executes the computer program 721 to send a request to the processor 710, and the processor 710 returns a result to implement the steps in the above method embodiment.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0137] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the various method embodiments described above.

[0138] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A dynamic optimization method for a text-based graph model, characterized in that, The method includes: The structure of the Wensheng graph model is analyzed, multiple independent sub-model components are identified, the execution order of each sub-model component is determined, and the static weights required for each sub-model component to be loaded into the GPU memory are pre-calculated. By actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table; Upon receiving an inference request containing runtime parameters, each sub-model component is sequentially scheduled to perform forward computation according to the execution order described above. Based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes the forward calculation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components. The set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory. The process involves actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table, including: Identify key configuration parameters that affect GPU memory usage, including prompt word sequence length, batch size, and generated image resolution; Select multiple parameter combinations from the multidimensional parameter space composed of the key configuration parameters; Each sub-model component is loaded into the GPU memory in sequence, and forward calculation is performed under the parameter combination to measure the peak memory usage. The peak memory usage is monitored and recorded through the GPU performance monitoring interface, and the sub-model component is unloaded from the GPU memory. The runtime memory lookup table is generated based on the correspondence between all recorded parameter combinations and peak memory usage.

2. The method according to claim 1, characterized in that, The structure of the analytical text image model is analyzed, multiple independent sub-model components are identified, the execution order of each sub-model component is determined, and the static weights required for each sub-model component to be loaded into GPU memory are pre-calculated, including: Read the storage file of the text graph model, parse the structure of the text graph model, and identify multiple independent sub-model components, including a text encoding model, a diffusion denoising model, and a latent encoding model; Based on the total number of parameters of each sub-model component, the number of bytes corresponding to the data type of each parameter, and the estimated runtime overhead, the static weight of each sub-model component is calculated. The static weight is the amount of memory required for each sub-model component to be loaded into the GPU memory.

3. The method according to claim 1, characterized in that, Based on the static weight size and the runtime memory lookup table, after the currently executing sub-model component completes its forward computation, a memory management operation is performed on the next sub-model component to be executed to update the set of loaded sub-model components. The memory management operation includes a preloading strategy, which includes: Based on the running parameters of the inference request, the runtime video memory lookup table is queried to obtain the peak video memory usage required by the next sub-model component to be executed; Query the current amount of free GPU memory; If the amount of free video memory is greater than or equal to the static weight of the next sub-model component to be executed, then the next sub-model component to be executed is preloaded from CPU memory to GPU video memory and added to the set of loaded sub-model components.

4. The method according to claim 3, characterized in that, The memory management operation also includes a dynamic unloading strategy. If the amount of free GPU memory is less than the peak memory usage required by the next sub-model component to be executed, the dynamic unloading strategy is executed. The dynamic unloading strategy includes: The sub-model component that has recently completed forward computation is unloaded from GPU memory and removed from the set of loaded sub-model components until available memory meets the peak memory usage required by the next sub-model component to be executed.

5. The method according to claim 2, characterized in that, Before receiving an inference request containing runtime parameters, the method further includes: The identified text encoding model, diffusion denoising model, and latent encoding model are all preloaded into CPU memory and kept resident. The memory management operation is performed based on a fixed execution order of the text encoding model, the diffusion denoising model, and the latent encoding model.

6. The method according to claim 1, characterized in that, The peak memory usage includes the memory used by the model weights, the memory allocated by the runtime memory pool of the computing framework, and the memory used by the intermediate activation feature maps generated during the forward computation process.

7. A dynamic optimization system for a text-based graph model, characterized in that, The system includes: The parsing model module is used to parse the structure of the text graph model, identify multiple independent sub-model components, determine the execution order of each sub-model component, and pre-calculate the static weights required for each sub-model component to be loaded into the GPU memory. The video memory lookup table generation module is used to measure the peak video memory usage of each sub-model component by actually running each sub-model component under different combinations of running parameters, map the running parameters to the peak video memory usage, and generate a runtime video memory lookup table. The forward computation module is used to, upon receiving an inference request containing running parameters, sequentially schedule each sub-model component to perform forward computation according to the execution order. The memory management execution module is used to perform memory management operations on the next sub-model component to be executed after the currently executed sub-model component has completed the forward calculation, based on the static weight size and the runtime memory lookup table, so as to update the set of loaded sub-model components, wherein the set of loaded sub-model components refers to the set of sub-model components currently stored in the GPU memory. The process involves actually running each of the sub-model components under different combinations of operating parameters, measuring the peak video memory usage of each sub-model component, mapping the operating parameters to the peak video memory usage, and generating a runtime video memory lookup table, including: Identify key configuration parameters that affect GPU memory usage, including prompt word sequence length, batch size, and generated image resolution; Select multiple parameter combinations from the multidimensional parameter space composed of the key configuration parameters; Each sub-model component is loaded into the GPU memory in sequence, and forward calculation is performed under the parameter combination to measure the peak memory usage. The peak memory usage is monitored and recorded through the GPU performance monitoring interface, and the sub-model component is unloaded from the GPU memory. The runtime memory lookup table is generated based on the correspondence between all recorded parameter combinations and peak memory usage.

8. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

Images