Graphics memory actual occupation inference method and device based on system performance inflection point, and medium

Abstract

The present application relates to a kind of based on system performance inflection point actual memory occupation inference method, equipment and medium.Method includes obtaining actual user request;According to actual acquisition time, target monitoring time zone, each actual user request corresponding word element information and word element delay standard curve detect whether it satisfies overall over-standard condition;If it satisfies overall over-standard condition, according to the word element information of each actual user request determine whether there is lag effect, wherein, lag effect includes target percentage of actual user request exists lag effect on the same word element number word element;If there is lag effect, according to the average request concurrency of target monitoring time zone, model weight, single block size estimate system performance inflection point actual memory occupation, to determine whether current pre-allocated video memory is reasonable.

Description

Methods, equipment, and media for inferring actual video memory usage based on system performance inflection points Technical Field

[0001] This invention relates to the field of GPU resource management technology, and in particular to a method, device and medium for inferring actual memory usage based on system performance inflection points. Background Technology

[0002] A Graphics Processing Unit (GPU), also known as a display core, display chip, or video processor, is a coprocessor used for processing images and graphics calculations. It is widely used in personal computers, workstations, and some mobile devices (such as smartphones and tablets). GPU memory usage refers to the amount of memory used by the graphics card to store graphics data or intermediate computation results, directly affecting the efficiency of task execution.

[0003] Currently, none of the mainstream GPU manufacturers on the market offer practical tools to monitor actual GPU memory usage. Most simply collect total memory usage data, failing to accurately calculate the actual amount of memory used. For example, an 80GB Cambricon GPU, after deploying the Qianwen 2.5-7b large model, will initially occupy 80% or 90% of the specified memory, corresponding to 64GB or 72GB. However, since no one is accessing the large model at this time, the 64GB or 72GB used is pre-allocated, and the actual amount of memory used is unknown, raising questions about potential resource waste. To address this pain point, there is an urgent need in existing technologies for a method to infer actual GPU memory usage. Summary of the Invention

[0004] One of the objectives of this invention is to provide a method, device, and medium for inferring actual video memory usage based on system performance inflection points.

[0005] According to one aspect of this application, a method for inferring actual video memory usage based on system performance inflection points is provided, the method comprising:

[0006] Obtain actual user requests, including the actual time of obtaining the actual user requests;

[0007] The system detects whether the overall exceedance condition is met based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve. The lexical information includes multiple sequentially arranged lexical units and the time required to generate each lexical unit.

[0008] If the overall exceedance condition is met, determine whether there is a lag effect based on the lexical information of each actual user request. The lag effect includes the lag effect of the target percentage of actual user requests on the lexical with the same lexical sequence number.

[0009] If there is a lag effect, the actual memory usage at the system performance inflection point is estimated based on the average request concurrency of the target monitoring time zone, model weights, and single block size, thereby determining whether the current pre-allocated memory is reasonable.

[0010] According to another aspect of this application, a computer device for inferring actual video memory usage based on system performance inflection points is provided, including a memory and a processor, wherein a computer program capable of being loaded by the processor and executed as described above is stored in the memory.

[0011] According to another aspect of this application, a computer-readable storage medium is provided, storing a computer program that can be loaded by a processor and executed as described above.

[0012] Compared to existing technologies, this application obtains actual user requests and detects whether the overall exceeding conditions are met based on the actual acquisition time of the actual user requests, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical latency standard curve. If the overall exceeding conditions are met, it determines whether there is a lag effect based on the lexical information of each actual user request. If a lag effect exists, it estimates the actual memory usage at the system performance inflection point based on the average request concurrency in the target monitoring time zone, model weights, and single block size, thereby determining whether the current pre-allocated memory is reasonable. This achieves the goal of inferring the actual memory usage based on the system performance inflection point. Attached Figure Description

[0013] Figure 1 shows a flowchart of a method for inferring actual video memory usage based on system performance inflection points according to an embodiment of this application;

[0014] Figure 2 shows a schematic diagram of a device structure for a method of inferring actual video memory usage based on system performance inflection points according to an embodiment of this application;

[0015] Figure 3 illustrates an exemplary system that can be used to implement the various embodiments described in this application. Detailed Implementation

[0016] The present application will now be described in further detail with reference to the accompanying drawings.

[0017] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), input / output interfaces, network interfaces, and memory.

[0018] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.

[0019] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0020] The devices referred to in this application include, but are not limited to, terminals, network devices, or devices formed by integrating terminals and network devices through a network. The terminals include, but are not limited to, any mobile electronic product capable of human-computer interaction (e.g., via a touchpad), such as smartphones and tablets. These mobile electronic products can use any operating system, such as Android or iOS. The network devices include electronic devices capable of automatically performing numerical calculations and information processing according to pre-set or stored instructions. Their hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. The network devices include, but are not limited to, computers, network hosts, single network servers, multiple network server clusters, or clouds composed of multiple servers. Here, a cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, wireless ad hoc network, etc. Preferably, the device can also be a program running on the terminal, network device, or a device formed by integrating the terminal and network device, network device, touch terminal, or network device and touch terminal through a network.

[0021] Of course, those skilled in the art should understand that the above-described devices are merely examples, and other existing or future devices that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0022] In the description of this application, "multiple" means two or more, unless otherwise expressly and specifically defined.

[0023] Referring to Figure 1, this embodiment of the invention provides a flowchart of a method for inferring actual video memory usage based on system performance inflection points. The method includes steps S11, S12, S13, and S14. In step S11, actual user requests are acquired, including the actual acquisition time of each user request. In step S12, the overall over-limit condition is checked based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve. The lexical information includes multiple sequentially arranged lexicals and the time required to generate each lexical. In step S13, if the overall over-limit condition is met, a lag effect is determined based on the lexical information of each actual user request. The lag effect includes a target percentage of actual user requests exhibiting a lag effect on lexicals with the same lexical sequence number. In step S14, if a lag effect exists, the actual video memory usage at the system performance inflection point is estimated based on the average request concurrency, model weight, and single-block size of the target monitoring time zone, thereby determining whether the current pre-allocated video memory is reasonable. For example, a server is equipped with four GPUs, each with 80GB of video memory. The Qwen2-7B model is deployed using tensor parallel deployment. vLLM configuration: block-size=16 (each KV cache block holds 16 tokens). It should be noted that the method described in this embodiment is specifically applied to a scenario where "computing resources are not saturated, and video memory resources are limited" (Memory-Bound). Specifically, within the target monitoring time zone of this embodiment, the computation utilization of the graphics processing unit (GPU) is lower than a preset safety threshold (e.g., 70%~80%). At this time, the bottleneck of system performance is mainly concentrated on video memory capacity and memory scheduling. Under this premise, the "overall excessive token latency" and "lag effect" detected in steps S12 and S13 can be clearly attributed to the blocking of video memory resource allocation (e.g., increased time spent by the block manager in searching for free blocks, or triggering the swap-out mechanism), thereby eliminating interference from the decrease in generation speed caused by computation unit saturation.

[0024] Specifically, in step S11, actual user requests are acquired, including the actual acquisition time of the actual user request. In some embodiments, the vLLM server receives the user request. User requests include, but are not limited to, actual user requests and simulated user requests. In some embodiments, user requests during actual monitoring of the application are referred to as actual user requests, and user requests during simulated operation are referred to as simulated user requests. For example, the actual acquisition time of actual user request A is 11:30:31.

[0025] In step S12, the system checks whether the overall exceedance condition is met based on the actual acquisition time, the target monitoring time zone, the token information corresponding to each actual user request, and the token delay standard curve. The token information includes multiple sequentially arranged tokens and the time required to generate each token. In some embodiments, the target monitoring time zone includes, but is not limited to, time intervals, such as 11:30:30-11:31:30, 10:00:00-10:01:00, etc. In some embodiments, the target monitoring time zone can be randomly determined by the system, or it can be a time interval with a fixed time interval as the target monitoring time zone, or it can be monitored continuously with a certain time interval as the monitoring unit (e.g., the target monitoring time zone). In some embodiments, a token includes the basic unit of text processing. Those skilled in the art can determine that, in response to a received user request, multiple tokens are generated sequentially based on the user request, and the system records the generation time of each token and the time required to generate each token. In some embodiments, the token delay standard curve includes a standard curve with instantaneous standard concurrency as the x-axis and standard token delay as the y-axis; the overall exceeding condition includes, but is not limited to, the difference between the target standard token delay and the average token delay corresponding to the average instantaneous concurrency of the target monitoring time zone in the token delay standard curve being equal to or greater than a target threshold. For a detailed explanation of this part, please refer to the corresponding embodiments below, which will not be repeated here. For example, an actual user request A includes tokens arranged in sequence: TokenA1 (generation time required 25ms), TokenA2 (generation time required 25ms)... In some embodiments, token delay includes, but is not limited to, the time required to generate tokens.

[0026] In step S13, if the overall exceedance condition is met, it is determined whether a lag effect exists based on the lexical information of each actual user request. The lag effect includes the presence of a lag effect on the same lexical sequence number in the actual user requests of the target percentage. For example, if there are 100 actual user requests within the target monitoring time zone, and 95 of these requests have a difference between the time required to generate the 6th lexical and the time required to generate previous lexicals equal to or greater than the target difference, this indicates that 95% of the actual user requests have a significant lag effect on the same lexical. Here, the target percentage includes 80%, thus confirming that a lag effect is met.

[0027] In step S14, if a lag effect exists, the actual memory usage at the system performance inflection point is estimated based on the average request concurrency, model weight, and block size of the target monitoring time zone, thereby determining whether the current pre-allocated memory is reasonable. In some embodiments, estimating the actual memory usage at the system performance inflection point based on the average request concurrency, model weight, and block size of the target monitoring time zone includes: obtaining the average request concurrency of the target monitoring time zone and using the average request concurrency as the number of active blocks; determining the actual memory usage based on the number of active blocks, block size, and model weight. For example, actual memory usage = model weight + (average request concurrency × block size). For example, the total weight memory of the Qwen-7B model is 144GB, with 4 cards in tensor parallelism, each card having a model weight of 36GB and a block size of 5MB. The average request concurrency of the target monitoring time zone with a lag effect is 10, so the number of active blocks is 10. Therefore, the KV Cache usage per card = 36GB + 0.049GB = 36.049GB. In practical deployments (e.g., using inference frameworks like vLLM), the system typically divides the physical memory space into a "static reserved area" and a "dynamic key-value cache." The "static reserved area" is pre-allocated during system initialization and is used to store model weights and runtime temporary data (Runtime Buffer, including intermediate activation values ​​and NCCL buffers) under maximum concurrent load. The remaining memory space is allocated to the "dynamic key-value cache" to store key-value cache blocks. Based on this memory model, the "model weight" term in the above formula (actual memory usage = model weight + number of active blocks × block size) represents the baseline of static usage, while "(number of active blocks × block size)" accurately reflects the actual level of the dynamic area. Therefore, when a lag effect (i.e., a performance inflection point) is detected, it means that the "dynamic key-value cache" has reached saturation. The value calculated using the above formula at this point is the upper limit of the effective memory capacity of the system while ensuring performance. This calculation method ignores the minute dynamic fluctuations in the reserved area, thus achieving the goal of quickly inferring the rationality of memory usage with low-cost computation in engineering. In some embodiments, determining whether the current pre-allocated video memory is reasonable includes: if the difference between the actual video memory usage and the current pre-allocated video memory is equal to or greater than the target difference, determining that the current pre-allocated video memory is unreasonable.

[0028] In some embodiments, detecting whether the overall exceedance condition is met based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve includes: within the target monitoring time zone, determining the average instantaneous concurrency of actual user requests within the target monitoring time zone based on the instantaneous concurrency of actual user requests per second and the total duration of the target monitoring time zone; filtering target actual user requests from the actual user requests based on the actual processing time of each actual user request and the target monitoring time zone, wherein target actual user requests include actual user requests processed within the target monitoring time zone; determining the average lexical delay corresponding to the target monitoring time zone based on the total processing time of the target actual user requests and the total number of lexical units; and determining whether the overall exceedance condition is met based on the average instantaneous concurrency, the average lexical delay, and the lexical delay standard curve. In some embodiments, the instantaneous concurrency includes the number of actual user requests being processed. For example, if the target monitoring time zone includes 11:30:30-11:31:30, then the total duration of the target monitoring time zone is 60 seconds. At 11:30:30, there are no requests being processed, so the instantaneous concurrency is 0. At 11:30:31, there are requests being processed: actual user request A1, so the instantaneous concurrency is 1. At 11:30:32, there are requests being processed: actual user request A1, so the instantaneous concurrency is 1. At 11:30:33, there are requests being processed: actual user request A1 and actual user request A2, so the instantaneous concurrency is 2. At 11:30:34, there are requests being processed: actual user request A1, actual user request A2, and actual user request A3, so the instantaneous concurrency is 3. At 11:30:35, there are requests being processed: actual user request A2 and actual user request A3, so the instantaneous concurrency is 2. In some embodiments, the average instantaneous concurrency includes the ratio between all instantaneous concurrency and the total duration of the target monitoring time zone. For example, in the above embodiment, if 11:30:30-11:30:35 is taken as the target monitoring time zone, then the average instantaneous concurrency of the target monitoring time zone is (0+1+1+2+3+2) / 6=1.5. In some embodiments, the actual processing time includes the time required from receiving the actual user request to completing the processing of the actual user request. For example, if the target monitoring time zone includes 11:30:30-11:31:30, the actual reception time of actual user request A1 is 11:30:30, and the actual processing completion time is 11:31:00; then the actual user request A1 is the actual user request that is processed and completed within the target monitoring time zone, and this actual user request A1 is taken as the target actual user request.For example, if the actual reception time of actual user request A2 is 11:30:35 and the actual processing completion time is 11:31:37, then actual user request A2 is not a user request processed and completed within the target monitoring time zone, and therefore cannot be considered a target user request. Similarly, if the actual reception time of actual user request A3 is 11:30:29 and the actual processing completion time is 11:31:30, then actual user request A3 is not a user request processed and completed within the target monitoring time zone, and therefore cannot be considered a target user request. In some embodiments, the total processing time of target user requests includes the sum of the actual processing times of target user requests within the target monitoring time zone. For example, if target user requests A1, A4, and A5 exist within the target monitoring time zone from 11:30:30 to 11:31:30, where the actual processing time of A1 is 30 seconds, A2's is 40 seconds, and A3's is 50 seconds... In some embodiments, the total number of tokens includes the sum of the number of tokens corresponding to the actual user requests within the target monitoring time zone. For example, if the number of tokens generated in response to actual user request A1 is 2, the number of tokens generated in response to actual user request A4 is 5, and the number of tokens generated in response to actual user request A5 is 8, then the total number of tokens is 15. In some embodiments, the average token latency includes the ratio between the total processing time and the total number of tokens. For example, in the above embodiments, the average token latency is 8. Further, whether the overall exceedance condition is met is determined based on the average instantaneous concurrency, the average token latency, and the token latency standard curve. For a detailed explanation of this part, please refer to the corresponding embodiments below, which will not be repeated here.

[0029] In some embodiments, the lexical delay standard curve includes multiple standard instantaneous concurrency counts and the standard lexical delay corresponding to each standard instantaneous concurrency count. Determining whether the overall exceedance condition is met based on the average instantaneous concurrency count, the average lexical delay, and the lexical delay standard curve includes: querying the lexical delay standard curve for a target standard instantaneous concurrency count that matches the average instantaneous concurrency count; using the standard lexical delay corresponding to the target standard instantaneous concurrency count as the target standard lexical delay; if the difference between the target standard lexical delay and the average lexical delay is equal to or greater than a target threshold, it is determined that the overall exceedance condition is met. For example, the lexical delay standard curve includes a curve with the standard instantaneous concurrency count as the abscissa and the standard lexical delay as the ordinate. In some embodiments, the standard instantaneous concurrency count and the standard lexical delay corresponding to each standard instantaneous concurrency count are obtained by sending simulated user requests during system idle time. For a detailed explanation of this part, please refer to the corresponding embodiments below, which will not be repeated here. In some embodiments, the standard instantaneous concurrency that matches the average instantaneous concurrency includes, but is not limited to, a standard instantaneous concurrency that is equal to or has a difference between it and the average instantaneous concurrency that is equal to or less than a difference threshold. The target standard lexical delay is found in the lexical delay standard curve corresponding to the average instantaneous concurrency (e.g., the standard lexical delay corresponding to the target standard instantaneous concurrency is used as the target standard lexical delay). For example, if the difference between the target standard lexical delay and the average lexical delay is equal to or greater than a target threshold, it is determined that the overall exceeding condition is met.

[0030] In some embodiments, the lexical delay standard curve is determined by the following method: during system idle time, simulated user requests with different instantaneous concurrency are sent based on a concurrent request sequence; for each instantaneous concurrency, after all simulated user requests have been processed, the average lexical delay of the simulated user request is determined based on the simulated processing time of each simulated user request and the total number of lexical units; the standard lexical delay corresponding to each instantaneous concurrency is determined based on the average total number of lexical units and the instantaneous concurrency; and a delay standard curve is generated based on multiple instantaneous concurrency and the standard lexical delay corresponding to each instantaneous concurrency. In some embodiments, system idle time includes, but is not limited to, when there are no actual user requests. In some embodiments, the concurrent request sequence includes, but is not limited to, 1, 2, 4, 6… or 1, 2, 3, 4, 5… etc., where 1, 2, 3, etc. represent the number of concurrent simulated user requests. For example, simulated user requests with different concurrency are sent according to the concurrent request sequence 1, 2, 4, 6, 8… At 12:30:30, a simulated user request is sent to the receiving end, making the concurrency of simulated user requests at 12:30:30 1. At 12:30:31, another simulated user request is sent to the receiving end, making the concurrency of simulated user requests at 12:30:31 2… In some embodiments, the simulation processing time includes the time required from receiving the simulated user request to completing the processing of that simulated user request. For example, when the instantaneous concurrency is 1, if there is a simulated user request B1, the simulation processing time for simulated user request B1 is 40 seconds, and the total number of tokens is 20, then the standard token delay corresponding to the instantaneous concurrency of 1 is 40 / 20 = 2. When the instantaneous concurrency is 2, there are simulated user requests B1 and B2. The simulated processing time for user request B1 is 40 seconds, the total number of tokens is 20, and the average token latency is 40 / 20=2. The simulated processing time for user request B2 is 30 seconds, the total number of tokens is 15, the average token latency is 30 / 15=2, and the standard token latency is 2 / 2=1. Finally, a standard curve for token latency is generated with the instantaneous standard concurrency as the x-axis and the standard token latency as the y-axis.

[0031] In some embodiments, determining whether a lag effect exists based on the lexical information of each actual user request includes: determining the target lexical number corresponding to the actual user request based on the lexical information of each actual user request, wherein the difference between the generation time required for the target lexical number and the generation time required for the lexical numbers preceding the target lexical number is equal to or greater than a target difference; identifying actual user requests with the same target lexical number; and determining that a lag effect exists if the number of actual user requests with the same target lexical number reaches a target percentage of all actual user requests. For example, the lexical units generated for each actual user request are numbered according to the generation order so that each lexical unit corresponds to a lexical number. For each actual user request, it is detected whether a target lexical number exists among the lexical numbers of the actual user request, wherein the difference between the generation time required for the lexical unit corresponding to the target lexical number and the generation time required for the lexical units preceding the target lexical number is equal to or greater than a target difference. This indicates that a significant lag occurs at the target lexical number when processing actual user requests. If the actual user requests for the target percentage occur after the same target term index, it indicates an overall lag effect.

[0032] Figure 2 shows a schematic diagram of a device for inferring actual video memory usage based on a system performance inflection point according to an embodiment of this application. The device includes a first module, a second module, a third module, and a fourth module. The first module is used to acquire actual user requests, wherein the actual user requests include the actual acquisition time of the actual user requests. The second module is used to detect whether the overall over-standard condition is met based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve. The lexical information includes multiple sequentially arranged lexicals and the time required to generate each lexical. The third module is used to determine whether there is a lag effect based on the lexical information of each actual user request if the overall over-standard condition is met. The lag effect includes the lag effect of the target percentage of actual user requests on the lexical with the same lexical sequence number. The fourth module is used to estimate the actual video memory usage at the system performance inflection point based on the average request concurrency, model weight, and single block size of the target monitoring time zone if there is a lag effect, thereby determining whether the current pre-allocated video memory is reasonable.

[0033] Here, the specific implementation methods corresponding to Module 1, Module 2, Module 3, and Module 4 are the same as or similar to the specific embodiments of steps S11, S12, S13, and S14 above, and therefore will not be repeated here, but are included by reference.

[0034] In addition to the methods and devices described in the above embodiments, this application also provides a computer-readable storage medium storing computer code that, when executed, performs the method described in any of the preceding embodiments.

[0035] This application also provides a computer program product that, when executed by a computer device, performs the method described in any of the preceding claims.

[0036] This application also provides a computer device, the computer device comprising:

[0037] One or more processors;

[0038] Memory, used to store one or more computer programs;

[0039] When the one or more computer programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described in any of the preceding methods.

[0040] Figure 3 illustrates an exemplary system that can be used to implement the various embodiments described in this application;

[0041] As shown in FIG3, in some embodiments, system 300 can be any of the devices described in each of the embodiments. In some embodiments, system 300 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 320) and one or more processors (e.g., one or more processors 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this application.

[0042] In one embodiment, the system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of the processors 305 and / or any suitable device or component communicating with the system control module 310.

[0043] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0044] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. In one embodiment, system memory 315 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 315 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0045] In one embodiment, the system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device 320 and (one or more) communication interfaces 325.

[0046] For example, NVM / storage device 320 may be used to store data and / or instructions. NVM / storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0047] NVM / storage device 320 may include storage resources that are physically part of a device on which system 300 is mounted, or that can be accessed by the device without necessarily being part of it. For example, NVM / storage device 320 may be accessed via a network through one or more communication interfaces 325.

[0048] One or more communication interfaces 325 may provide the system 300 with an interface to communicate over one or more networks and / or with any other suitable device. The system 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0049] In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 (e.g., memory controller module 330). In one embodiment, at least one of the processors 305 may be logically packaged with one or more controllers of the system control module 310 to form a system-in-package (SiP). In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die. In one embodiment, at least one of the processors 305 may be integrated with the logic of one or more controllers of the system control module 310 on the same die to form a system-on-a-chip (SoC).

[0050] In various embodiments, system 300 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 300 may have more or fewer components and / or different architectures. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0051] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0052] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0053] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0054] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or hereafter developed capable of storing computer-readable information / data for use by a computer system.

[0055] Herein, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run a method and / or technical solution based on the foregoing embodiments of this application.

[0056] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, all embodiments should be considered exemplary and non-limiting. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

Claims

1. A method for inferring actual video memory usage based on system performance inflection points, characterized in that, The method includes: acquiring actual user requests, wherein the actual user requests include the actual acquisition time of the actual user requests; detecting whether the overall exceedance condition is met based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve, wherein the lexical information includes multiple sequentially arranged lexicals and the time required to generate each lexical, and further includes: determining the average instantaneous concurrency of actual user requests within the target monitoring time zone based on the instantaneous concurrency of actual user requests per second and the total duration of the target monitoring time zone; and determining the average instantaneous concurrency of actual user requests within the target monitoring time zone based on the actual processing time of each actual user request and the target monitoring time zone. The process involves filtering target actual user requests from the actual user requests, where the target actual user requests include actual user requests processed within the target monitoring time zone; determining the average token latency corresponding to the target monitoring time zone based on the total processing time and total number of tokens for the target actual user requests; and determining whether the overall exceedance condition is met based on the average instantaneous concurrency, average token latency, and token latency standard curve, where the token latency standard curve includes multiple standard instantaneous concurrency numbers and the standard token latency corresponding to each standard instantaneous concurrency number, and further includes: querying the token latency standard curve for a target standard instantaneous concurrency number that matches the average instantaneous concurrency number based on the average instantaneous concurrency number. The standard lexical delay corresponding to the target instantaneous concurrency is taken as the target standard lexical delay; if the difference between the target standard lexical delay and the average lexical delay is equal to or greater than the target threshold, the overall exceeding condition is determined; if the overall exceeding condition is met, it is determined whether there is a lag effect based on the lexical information of each actual user request, wherein the lag effect includes a lag effect on the lexical with the same lexical index for the target percentage of actual user requests, and also includes: determining the target lexical index corresponding to each actual user request based on the lexical information of each actual user request, wherein the generation time required for the target lexical index is compared with the lexical index before the target lexical index. The difference between the time required for generation and the target difference is equal to or greater than the target difference; actual user requests with the same target byte sequence number are identified; if the number of actual user requests with the same target byte sequence number reaches the target percentage of all actual user requests, the lag effect is determined; if the lag effect exists, the actual memory usage at the system performance inflection point is estimated based on the average request concurrency, model weight, and single block size of the target monitoring time zone, thereby determining whether the current pre-allocated memory is reasonable, and further includes: obtaining the average request concurrency of the target monitoring time zone and using the average request concurrency as the number of active blocks; determining the actual memory usage based on the number of active blocks, single block size, and model weight;The lexical delay standard curve is determined by the following method: During system idle time, simulated user requests with different instantaneous concurrency levels are sent based on a concurrent request sequence; for each instantaneous concurrency level, after all simulated user requests have been processed, the average lexical delay of that simulated user request is determined based on the simulated processing time and the total number of lexical units; the standard lexical delay corresponding to that instantaneous concurrency level is determined based on the average total number of lexical units and the instantaneous concurrency level; the delay standard curve is generated based on multiple instantaneous concurrency levels and the standard lexical delay corresponding to each instantaneous concurrency level.

2. The method according to claim 1, characterized in that, Determining whether the current pre-allocated video memory is reasonable includes: if the difference between the actual video memory usage and the current pre-allocated video memory is equal to or greater than the target difference, then the current pre-allocated video memory is determined to be unreasonable.

3. A device for inferring actual video memory usage based on system performance inflection points, characterized in that, The device includes: a first module for acquiring actual user requests, wherein the actual user requests include the actual acquisition time of the actual user requests; and a second module for detecting whether the overall exceedance condition is met based on the actual acquisition time, the target monitoring time zone, the lexical information corresponding to each actual user request, and the lexical delay standard curve, wherein the lexical information includes multiple sequentially arranged lexical units and the time required to generate each lexical unit, and further includes: within the target monitoring time zone, determining the average instantaneous concurrency of actual user requests within the target monitoring time zone based on the instantaneous concurrency of actual user requests per second and the total duration of the target monitoring time zone; and based on the actual processing time of each actual user request and the target... The monitoring time zone is used to filter target actual user requests from the actual user requests, wherein the target actual user requests include actual user requests processed within the target monitoring time zone; the average word latency corresponding to the target monitoring time zone is determined based on the total processing time and total number of words in the target actual user requests; whether the overall exceedance condition is met is determined based on the average instantaneous concurrency, average word latency, and word latency standard curve, wherein the word latency standard curve includes multiple standard instantaneous concurrency and the standard word latency corresponding to each standard instantaneous concurrency, and further includes: querying the word latency standard curve for a target standard instantaneous concurrency that matches the average instantaneous concurrency based on the average instantaneous concurrency. The standard lexical delay corresponding to the target instantaneous concurrency is taken as the target standard lexical delay; if the difference between the target standard lexical delay and the average lexical delay is equal to or greater than the target threshold, it is determined that the overall exceeding condition is met; Module 1 is used to determine whether there is a lag effect based on the lexical information of each actual user request if the overall exceeding condition is met, wherein the lag effect includes the lag effect of the target percentage of actual user requests on the lexical with the same lexical index, and further includes: determining the target lexical index corresponding to the actual user request based on the lexical information of each actual user request, wherein the generation time required for the target lexical index is compared with the lexical index before the target lexical index. The difference between the time required for generation and the target difference is equal to or greater than the target difference; the actual user requests with the same target byte sequence number are identified; if the number of actual user requests with the same target byte sequence number reaches the target percentage of all actual user requests, the lag effect is determined; a fourth module is used to estimate the actual memory usage at the system performance inflection point based on the average request concurrency, model weight, and single block size of the target monitoring time zone if the lag effect exists, thereby determining whether the current pre-allocated memory is reasonable, and further includes: obtaining the average request concurrency of the target monitoring time zone and using the average request concurrency as the number of active blocks; determining the actual memory usage based on the number of active blocks, single block size, and model weight;The lexical delay standard curve is determined by the following method: During system idle time, simulated user requests with different instantaneous concurrency levels are sent based on a concurrent request sequence; for each instantaneous concurrency level, after all simulated user requests have been processed, the average lexical delay of that simulated user request is determined based on the simulated processing time and the total number of lexical units; the standard lexical delay corresponding to that instantaneous concurrency level is determined based on the average total number of lexical units and the instantaneous concurrency level; the delay standard curve is generated based on multiple instantaneous concurrency levels and the standard lexical delay corresponding to each instantaneous concurrency level.

4. A computer device, characterized in that, It includes a memory and a processor, and the memory stores a method for inferring the actual video memory usage based on the system performance inflection point, which can be loaded by the processor and executed as described in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, The storage contains a method for inferring actual video memory usage based on system performance inflection points, which can be loaded by the processor and executed as described in any one of claims 1 to 2.

Images