CNN inference task time slice segmentation method and system based on divisible tuple

Abstract

The invention provides a CNN reasoning task time slice segmentation method and system based on a divisible tuple, and relates to the technical field of data processing. The CNN reasoning task is decomposed into sequentially executed BMO sequences, and continuous BMO is combined into tuples based on data dependence; based on the time estimation model of the tuple, segmenting the tuple according to the length of the target time slice to generate sub-tuples; wherein the segmentation is divided according to data dimensions and is rounded down to a minimum divisible unit; tuples or sub-tuples are selected from the tuple list to be combined into the time slice, the total execution time does not exceed the length of the target time slice, memory allocation is managed in a unified mode before the time slice is executed, and the memory is released in a unified mode after the time slice is executed. Performance modeling and time prediction are carried out according to CNN model operation characteristics, the task is divided into time slices to support time division multiplexing scheduling, and the real-time performance, the resource utilization rate and the hardware universality are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for time slice segmentation in CNN inference tasks based on separable tuples. Background Technology

[0002] Research on real-time inference in CNNs aims to solve the problem of inference efficiency of individual models by optimizing inference algorithms and model structures to reduce the time consumption of a single inference, thereby reducing system latency.

[0003] With the rapid development of artificial intelligence applications, complex applications need to run multiple CNN inference models simultaneously on a single device. Especially in embedded systems with high real-time requirements, it is essential to ensure that latency-sensitive critical tasks are completed in a timely manner.

[0004] However, improving GPU hardware capabilities increases cost and power consumption. Existing GPU scheduling methods mainly rely on specific hardware characteristics to predict execution time and provide preemption mechanisms, making them difficult to apply across different GPUs or underlying algorithms and unable to meet the demand for general, low-overhead, real-time scheduling in multi-model concurrent scenarios. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a time-slice segmentation method and system for CNN inference tasks based on separable tuples. By performing performance modeling and time prediction based on the operational characteristics of CNN models, the task is segmented into time slices to support time-division multiplexing scheduling, thereby improving real-time performance, resource utilization, and hardware versatility.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a time slice segmentation method for CNN inference tasks based on separable tuples, comprising: The CNN inference task is decomposed into a sequence of sequentially executed BMOs, and consecutive BMOs are combined into tuples based on data dependencies. The time estimation model based on tuples divides tuples according to the target time slice length to generate subtuples; the division is based on the data dimension and rounded down to the smallest divisible unit. Select tuples or subtuples from the tuple list to form a time slice, ensuring that the total execution time does not exceed the target time slice length. Memory allocation is managed uniformly before the time slice is executed, and memory is released uniformly after execution.

[0007] Secondly, the present invention provides a time-slice segmentation system for CNN inference tasks based on separable tuples, comprising: The performance modeling module is used to decompose the CNN inference task into a sequence of sequentially executed BMOs and combine consecutive BMOs into tuples based on data dependencies. The tuple segmentation module is used to segment tuples based on the target time slice length according to the time estimation model of tuples, generating sub-tuples; wherein, the segmentation is divided according to the data dimension and rounded down to the smallest divisible unit; The time slice segmentation module is used to select tuples or sub-tuples from the tuple list to form a time slice, so that the total execution time does not exceed the target time slice length. Memory allocation is managed uniformly before the time slice is executed, and memory is released uniformly after execution.

[0008] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the time slice segmentation method for a CNN inference task based on separable tuples described in the first aspect.

[0009] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the time slice segmentation method for a CNN inference task based on separable tuples described in the first aspect.

[0010] Fifthly, the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps in the time slice segmentation method for a CNN inference task based on separable tuples described in the first aspect.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention uses data-dependent combination BMO as tuples to avoid redundant intermediate data storage caused by separate splitting of BMO, thus reducing memory overhead; it splits tuples according to data dimensions and rounds them down to the smallest divisible unit, combined with a time estimation model to ensure that the execution time of sub-tuples is accurate and controllable, meeting real-time requirements; it uniformly manages time slice memory allocation and release, avoiding time uncertainty caused by dynamic memory operations during execution. This invention is implemented entirely in software, without relying on specific GPU hardware, drivers or compilers, and is compatible with multiple types of GPU devices and underlying algorithm libraries, improving the versatility and portability of the solution, effectively supporting time-division multiplexing scheduling of multiple CNN models on a single device (especially embedded systems), balancing resource utilization and real-time performance.

[0012] (2) This invention employs a pure software-level time estimation and segmentation method. Based on the computational characteristics of the CNN inference task itself, it performs performance modeling and time prediction for each basic mathematical operation in the model, and segments the task into pure software time slices with execution times close to a given value. It does not rely on specific GPU drivers or compiler support and can adapt to various types of GPU devices (such as desktop GPUs, computing GPUs, and embedded GPUs) and underlying algorithm library implementations (such as cuDNN and cuBLAS). It achieves an average time estimation error of less than 5% and an average time slice segmentation error of less than 10% in different hardware environments, improving the versatility and portability of the method and avoiding the scheduling incompatibility problem caused by hardware characteristic dependence in traditional methods.

[0013] (3) This invention introduces the concept of separable tuples, dividing the CNN inference task into units containing relevant memory information and operation sequences. By sampling and regressing the time estimation models of basic operations such as GEMM, Im2col, GEMV, and Pooling, refined time prediction is achieved. Time slice segmentation accuracy: On a randomly generated CNN model, this method can accurately segment the task into time slices, with the actual execution time clustered in 80% to 90% of the target length and an error rate of less than 5%. At the same time, the segmentation process considers data integrity and memory dependency, reduces intermediate result storage overhead, and the additional time consumption is only about 0.63%, improving the accuracy and efficiency of task segmentation.

[0014] (4) This invention can accept a target time slice length and pre-statically divide the entire CNN inference task into a series of time slices; it also supports dynamically dividing the task into subtasks of a specified time length according to the real-time resource allocation requirements of the scheduler at runtime, providing great scheduling flexibility. The granularity of the division is controllable and the error is low: it can divide the task into time slices of arbitrary target length with an error rate of less than 10%, and the actual execution time is closely distributed around the set value, so that the system can effectively balance real-time performance and throughput.

[0015] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0017] Figure 1 The main flowchart of a time slice segmentation method for CNN inference tasks based on separable tuples provided in this embodiment of the invention; Figure 2This is a comparative diagram of no segmentation, individual operation segmentation, and overall tuple segmentation provided in an embodiment of the present invention; wherein, (a) represents a schematic diagram of a CNN inference task without segmentation; (b) represents a schematic diagram of segmenting Im2col and GEMM as individual operations; and (c) represents a schematic diagram of segmenting Im2col and GEMM as a whole. Figure 3 This is a schematic diagram of splitting Im2col convolutional tuples according to the input channel provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating an example of residual block memory planning provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Explanation of technical terms 1. BMO: BMO (basic mathematical operation) is defined in this invention as a logical operation with the following characteristics: (1) It has a clear start and end point, and its operation process is to modify the limited data in memory according to a specific algorithm; (2) In a constant hardware and software environment, its time consumption will not change with the change of the start time, nor with the location of the data in memory; (3) Its theoretical time complexity is an inherent property of the operation itself and does not change with the implementation method.

[0020] 2. Tuple: This is a concept proposed in this invention that lies between basic mathematical operations and components of a CNN inference model. It refers to a group of basic operations that can be divided in a CNN inference task, consisting of one or more basic mathematical operations (such as convolution and matrix multiplication), facilitating time estimation and segmentation. This concept was proposed to address the issues of data integrity and intermediate result storage during time-slice partitioning.

[0021] 3. Time slice: A unit of execution with a fixed or dynamic length, consisting of multiple tuples, whose execution time is close to a given value, used for time-division multiplexing scheduling.

[0022] 4. GEMM: GEMM (Generic Matrix Multiplication) is a fundamental operation in linear algebra, widely used in scientific computing, machine learning, deep learning, image processing and other fields.

[0023] 5. Im2col: Im2col (image to column) is a technique that converts image data into matrix form, primarily used to accelerate convolution operations in convolutional neural networks. Its core idea is to transform convolution operations into matrix multiplication, thereby leveraging efficient matrix multiplication algorithms from linear algebra to improve computational speed.

[0024] Example 1 Convolutional Neural Networks (CNNs), as deep learning models focused on processing structured grid data (such as images and time series), use a hierarchical structure of convolutional layers, pooling layers, and fully connected layers to automatically extract spatial features (such as edges and textures) from input data. With the ability to capture local features and the weight sharing mechanism of convolutional operations, they are widely used in fields such as image recognition, object detection, and natural language processing, and have the advantages of both computational efficiency and generalization ability.

[0025] CNN inference is a complete process in which the model generates output results based on input data, covering the entire chain from data input and hierarchical computation to result output. Typical scenarios include mobile device image recognition, autonomous driving environmental perception, and biometric verification. This task is not a single computational step, but consists of a series of closely related basic mathematical operations, such as feature extraction by convolution, data dimensionality reduction by pooling, and feature fusion by matrix multiplication (GEMM). Each operation is executed in the logical order of the model to jointly complete a reasoning judgment.

[0026] In the basic operation sequence of CNN inference, some operations have strong data dependencies and execution correlations. A typical example is the convolution operation accelerated by Im2col: the image tensor must first be converted into a matrix format using the Im2col operation, and then the matrix multiplication is performed using the GEMM operation. Temporary intermediate data is generated between the two. If such correlated operations are separated and processed independently, the intermediate data needs to be stored separately, which not only increases memory overhead but also introduces time loss due to data reading and writing, thus reducing inference efficiency.

[0027] To address the aforementioned issues, this invention proposes a method, system, medium, and device for time slice segmentation of CNN inference tasks based on separable tuples. Applicable to CNN inference task scheduling under CPU-GPU architecture, it aims to segment a complete CNN inference task into multiple independent time slices with execution times close to a given value, supporting time-division multiplexing scheduling. By defining separable tuples and performing segmentation based on a time estimation model of the CNN inference model, it achieves pure software-level task segmentation without relying on specific hardware or driver support.

[0028] like Figure 1As shown, this embodiment discloses a time slice segmentation method for CNN inference tasks based on separable tuples, including the following steps: S1: Decompose the CNN inference task into a sequence of sequentially executed BMOs, and combine consecutive BMOs into tuples based on data dependencies; S2: A time estimation model based on tuples splits tuples according to the target time slice length to generate sub-tuples; wherein, the splitting is based on the data dimension and rounded down to the smallest divisible unit; S3: Select tuples or subtuples from the tuple list to form a time slice, ensuring that the total execution time does not exceed the target time slice length. Memory allocation is managed uniformly before the time slice is executed, and memory is released uniformly after execution.

[0029] Next, combined Figure 1 This embodiment provides a detailed description of a time-slice segmentation method for CNN inference tasks based on separable tuples.

[0030] Previous approaches relied on hardware-specific predictions or end-to-end models, making them difficult to generalize. This invention avoids hardware dependence through a purely software approach, addressing real-time performance issues by utilizing a BMO-tuple-task execution model. The model input is a computation graph of a CNN inference task. After preprocessing, the performance model estimates the time for each BMO, then constructs tuples, and finally divides the data into time slices.

[0031] In S1, firstly, a divisible tuple is constructed to decompose the CNN inference task into a set of BMOs that need to be executed sequentially.

[0032] For example, BMOs include GEMM, Im2col transformation, activation functions, etc. These BMOs model performance based on the computational characteristics of the CNN model itself, such as treating the theoretical time complexity of each operation as an inherent property that does not change with hardware implementation.

[0033] Based on data dependencies, consecutive BMOs are combined into multiple tuples. Specifically, a set of sequentially executed BMOs is combined into a splittable tuple.

[0034] The concept of separable tuples lies between basic mathematical operations and CNN model components, facilitating the resolution of data integrity and intermediate result storage issues during time-slice partitioning. For example, for a convolutional layer accelerated using the Im2col method, combining the Im2col operation and the GEMM operation into a tuple ensures that intermediate results do not need to be saved separately, thereby reducing memory overhead.

[0035] For example, when performing a 3×3, 128-channel convolution on a 64×64×128 input image, the tuple memory mode implicitly contains intermediate data, such as... Figure 2As shown, by avoiding saving 18MB of intermediate data during switching and retaining only 4.5MB of input / output data, the total video memory is reduced from 20.5MB to 4.5MB, and the time per sub-tuple is close to t / 2. This method is more refined compared to model-level or component-level estimation and more general compared to hardware-information-based estimation.

[0036] The memory area occupied by each tuple can be clearly divided into read-write data segment, read-only data segment (parameter segment), and workspace segment (cache segment). This allows us to segment tuples into operation sequences and memory segment categories (read-write data segment such as input / output data, read-only data segment such as convolution kernel weights, and workspace segment such as temporary cache).

[0037] CNN inference tasks typically involve data flow between multiple tuples or time slices. The read / write data segment involves using the output of one layer as the input to another. The parameter segment is generally not modified during task execution, and the workspace segment stores temporary data during time slice execution. Dividing the data into segments is to clearly define the memory areas required for data transfer between time slices, preventing data loss or corruption. It also allows for pre-allocation of all memory segments in system scheduling applications, eliminating runtime memory allocation uncertainty and meeting real-time scheduling requirements. For example, in the subsequent S3, the time slice processing steps, "in," "out," and "ws" correspond to the data segment and workspace segment respectively, visually demonstrating how memory segments are arranged to optimize data flow.

[0038] The size of the data segment is determined by the model structure information, and the scheduling system needs to reserve the required space for subsequent tuples in advance based on this information; while the size of the cache segment is determined by the needs of the internal operations of the tuple, and is released immediately after the current tuple is executed.

[0039] This embodiment proposes the concept of separable tuples, which aggregate consecutive basic operations with strong correlations in CNN inference that are prone to additional overhead when split into smaller units, into unified operation units. Each separable tuple needs to define three types of memory region attributes: first, a read-write data segment, used to store operation input and output data; second, a read-only data segment, used to store fixed model parameters that the operation depends on; and third, a workspace segment, used to hold temporary data during operation execution. Through this structured design, separable tuples not only ensure the continuous execution of related operations and avoid redundant storage of intermediate data, but also provide a clear unit basis for subsequent time-slice partitioning and memory management.

[0040] In S2, although the tuple as the aggregation unit of BMO has solved the problems of intermediate data redundancy and data integrity caused by splitting BMO separately, it still needs to be further divided because it needs to adapt to the real-time scheduling requirements and hardware resource characteristics of CNN inference tasks.

[0041] On the one hand, the overall execution time of a tuple may exceed the preset target time slice length. If it is not divided, the GPU will be occupied by a single tuple for a long time, which will not meet the real-time requirements of latency-sensitive tasks in scenarios such as embedded systems. On the other hand, different GPU hardware (such as desktop GPUs and embedded GPUs) have different performance, and the idle time of the GPU changes dynamically. Tuple partitioning can generate sub-tuples with precise and controllable execution time based on hardware capabilities and real-time resource conditions, avoiding resource waste or performance overload. At the same time, combined with the round-down strategy of the smallest divisible unit (such as the thread block size), it ensures that the execution time of the sub-tuple does not exceed the threshold, laying the foundation for subsequent time slice construction and time-division multiplexing scheduling.

[0042] Specifically, the partitioning of a tuple depends on the partitioning of each basic operation within it. Since the input and output dimensions of each basic operation within a tuple are interrelated, an estimation model can be established between the runtime of the entire tuple and its input dimensions. Solving this estimation model essentially involves summing the estimation models of each operation, thereby calculating the amount of data that each processing step should complete within the expected time, based on the expected total runtime of the tuple.

[0043] For simple one-dimensional tuples such as memory copying, vector addition, and activation functions, the partitioning position can be determined by directly solving the estimation function, since the processing of each element is independent. For high-dimensional operations such as matrix-vector multiplication, the length of the output vector can be determined using an estimation model, and this can be used as the basis for partitioning, since the computation of each output element is independent. However, when dealing with complex operations involving tensors such as convolution and pooling, different input images can be computed completely independently, and different channels within the same image can also be further partitioned.

[0044] Specifically, in the GEMM method, each thread block first loads two corresponding 64×64 blocks of matrices A and B into shared memory. Then, each thread loads a column of block A and a row of block B into a register, performs the corresponding multiplication, and then accumulates them onto the element handled by the single thread. The memory access location then moves one element in the direction of K (row direction of block A, column direction of block B), and this iteration is repeated until, after 64 iterations, the blocks loaded in shared memory have been completely traversed. The execution time of each thread in a thread block depends only on the size of the thread block and the value of K, and is linearly related to K. = 1 + 2; in, This represents the total time required for a thread to complete all its computations within a thread block. 1 and 2 represents the fitting coefficients in the linear model, which are the slope and intercept, respectively; specifically, 1 is approximately the average time taken to process one computation in each of the K directions; 2 is a fixed cost independent of K.

[0045] For example, a hardware-independent time estimation model is used to calculate the overall execution time of a tuple. For instance, the time function of the Im2col operation is η1 × (number of input elements) + η2, where η1 and η2 are calibrated through sampling and least squares regression. This model is based on time estimation at the fundamental operation level of CNNs and is hardware-independent. If the execution time of a tuple exceeds the given time slice length t, it is split according to the input data dimension, for example, preferentially splitting by channel (C dimension) or batch (N dimension) to ensure that the sub-tuples are computed independently. For simple one-dimensional tuples, such as memory copying, vector addition, activation functions, etc., the estimated model function can be solved directly. To calculate the division position, The number of input elements is rounded down to the smallest unit to improve memory access efficiency. For high-dimensional operations such as matrix-vector multiplication, the length of the output vector can be determined using an estimation model, and this can be used as a basis for partitioning, since the calculation of each output element is independent. This operation is responsible for computation. ,in To estimate the output vector, A is an M×N matrix, and x and y are vectors of length N and M, respectively. and The coefficients are scalars. Since the input vector x needs to be shared by all threads, most GPU implementations provide constant memory blocks to store read-only inter-thread shared data. The execution time of the GEMV operation should be the total time consumed by the threads plus the time consumed by copying the input vector x to constant memory, approximated by the GEMV time estimation function. gemv That is: ; Where M and N are the number of rows and columns of matrix A, a For the time factor of the computational load, b and c It is the cost coefficient in terms of vector length. d It is a fixed overhead that is independent of the matrix size.

[0046] When dealing with complex operations involving tensors, such as convolution and pooling, different input images can be calculated completely independently, and different channels in the same image can be further divided. The time estimation function can be derived from the above steps.

[0047] Because a tuple contains a minimum divisible unit, the actual execution time of a sub-tuple is not necessarily strictly equal to the specified time. For example, one-dimensional operations are typically aligned to thread block sizes (e.g., 512), while high-dimensional operations are aligned to two-dimensional block sizes (e.g., 16 or 32) to improve memory access efficiency. The system employs a round-down strategy to ensure that the length of each time slice does not exceed a preset threshold. Specifically, this includes: Alignment to thread block size: GPU thread blocks are typically set to 256 or 512. If the data volume is 513, the system will process 512, pushing the remaining one to the next slice, or padding it but not executing it. GPU parallel execution depends on thread blocks. Non-alignment will cause threads to be idle, wasting computing resources. At the same time, this alignment process improves memory access efficiency because alignment ensures that memory accesses are merged, that is, adjacent threads access contiguous memory, reducing memory latency.

[0048] Alignment to 2D blocks: When processing a 28×28 image, if the cut point is in the middle, it must be aligned to a multiple of 16. High-dimensional data involves more dimensions. Non-alignment leads to non-contiguous memory access, increasing global memory access time.

[0049] Rounding down: If the calculated split point corresponding to time t is greater than the maximum time slice length threshold T set by the specified scheduler, it is rounded down to the nearest smallest unit boundary so that the actual time is less than or equal to T.

[0050] If not aligned, the split can lead to performance degradation or wasted memory. A purely software approach solves the versatility problem through tuples and rounding down, making it adaptable to different GPUs.

[0051] For example, a 3×3 Im2col convolution tuple with 4 input channels is split into two sub-tuples: each sub-tuple contains its own Im2col and GEMM operations, responsible for computing convolutions across different channels. The Im2col operation in the parent tuple processes all 4 channels of the input tensor, while each sub-tuple only processes the channel assigned to it. Figure 3 As shown. Subtuples share memory, and the β parameter of GEMM is set to 1 to accumulate the convolution results of all channels.

[0052] Both sub-tuples point to the same memory address, and starting with the second sub-tuple, the β parameter in its GEMM operation is set to 1 to accumulate the convolution results across channels. Furthermore, while partitioning the Im2col tuples by output channel allows for independent computation of each output channel, it leads to repeated execution of the Im2col operation or the need for additional storage of intermediate results. Therefore, this partitioning method fails to fully leverage the advantages of tuples in saving intermediate memory. Thus, partitioning by input channel or batch is more reasonable.

[0053] In this embodiment, a tuple partitioning strategy based on a time estimation model and a minimum divisible unit alignment strategy are employed to achieve fine-grained scheduling and control of GPU computing resources. This allows long-running tuples to be split into sub-tuples with controllable execution times, preventing a single task from occupying the GPU for extended periods and significantly improving the system's responsiveness and task switching efficiency in real-time scenarios such as embedded systems. Simultaneously, by rounding down to the thread block or 2D block size, the continuity and alignment of memory access are ensured, reducing idle threads and memory latency, improving hardware utilization and energy efficiency, and providing a reliable time-slice basis for subsequent time-division multiplexing scheduling.

[0054] In S3, the list of tuples corresponding to the CNN inference model is divided into time slices based on the execution time.

[0055] From the execution process perspective, time slice partitioning refers to selecting several tuples from the beginning of the task tuple list, combining them into a set that can be executed sequentially, and ensuring that their total execution time is close to a preset value. If the last selected tuple cannot fit completely into the current time slice, a sub-tuple whose execution time does not exceed the remaining time is partitioned and added to the current time slice. The unexecuted portion is then reinserted at the beginning of the task tuple list. The algorithm describes this partitioning process from the tuple list to the time slice, specifically: 1. Initialization: Create an empty list of time slices and an empty slice of time slice, and set the initial remaining time t to the target time T.

[0056] 2. Traverse model layers: Iterate through each layer in model M.

[0057] 3. Check and segment large layers: If the estimated time of the current layer... If it is greater than the remaining time t, then: 1) Calculate the split point That is, the scale that can just consume time t is obtained by using the inverse function of the time model.

[0058] 2) Split the layer, taking the first x segments of data from the layer as a new sub-tuple div, and update the remaining layer.

[0059] 3) Add the div to the current slice, then push the complete slice into the list, reset the slice to empty, and reset t to T (start a new time slice).

[0060] 4. Add a small layer: If the time of the layer does not exceed t, then directly add the entire layer to the current slice and update the remaining time. .

[0061] 5. Time slice boundary check: If the updated t is less than the allowable error e multiplied by T (i.e. the remaining time is too small to be effectively utilized), or the current layer is the last layer of the model, then push the current slice into the list, reset the slice to empty, and reset t to T.

[0062] 6. End and Return: After traversing all layers, return a list containing all time slices.

[0063] Data dependencies need to be organized holistically, including data transfer between tuples within the same time slice and between different time slices. Since the system assumes that execution of time slices is uninterrupted, memory allocation or deallocation operations should be avoided within a time slice to prevent introducing uncertainty in execution time. The memory required for all tuples (or operations) within a time slice should be allocated and managed uniformly before execution. Similar to tuples, the memory associated with a time slice can also be divided into three categories: data segments (used for data transfer between time slices), parameter segments (read-only data), and working segments (released immediately after the time slice ends).

[0064] The working area size of a time slice is determined by the maximum memory required by all the tuples it contains. All data segments spanning time slices (including parameter segments) are assigned unique IDs for management. These memory segments are allocated before the time slice is executed and are released in a timely manner through mechanisms such as reference counting.

[0065] For example, tuples or sub-tuples are extracted from the beginning of the tuple list of the CNN task, and the estimated time is accumulated until it approaches the given time slice length t. If the last tuple cannot be completely inserted, then the tuple is split into subtuples, added to the current time slice, and the remaining part is reinserted at the beginning of the list. For example... Figure 4As shown, when a model with three convolutional layers is divided into two time slices, the memory segment allocation is as follows. Each rectangle in the diagram represents a contiguous memory segment, and "in", "out", and "ws" indicate that a tuple within this segment uses that memory as its input, output, and workspace, respectively. Each time slice corresponds to a workspace segment used to store data passed between tuples and the tuple's workspace. Data ("in" and "out" in the diagram) is arranged alternately at the beginning and end of the workspace segment. The workspace size of a time slice depends on the maximum memory required by all tuples within that slice. All data segments (including parameter segments) spanning different time slices are assigned a unique ID for management. These segments are allocated before the time slice is executed and released promptly using methods such as reference counting.

[0066] In this embodiment, by dynamically partitioning the tuple list into time slices and implementing unified memory management, fine-grained scheduling and efficient reuse of GPU computing resources are achieved. By grouping tuples with controllable execution times into time slices, task fragmentation and resource idleness are avoided, significantly improving hardware utilization and system throughput. Simultaneously, the memory management mechanism of unified allocation before execution and unified release after execution reduces memory fragmentation and redundant allocation overhead, enhancing the continuity and determinism of memory access. Furthermore, by flexibly splitting and rearranging incomplete tuple portions, the strict controllability of time slice boundaries is ensured, providing low-latency and highly deterministic execution guarantees for real-time inference tasks.

[0067] This specific embodiment proposes the concept of "segmentable tuples," packaging data-dependent basic operations in CNN inference into operation groups. This fundamentally solves the problem of redundant intermediate result storage caused by splitting, significantly reducing memory overhead compared to traditional inter-layer splitting strategies. By constructing a correlation model between tuple runtime and input size, accurate prediction of time slice length is achieved. Combined with differentiated partitioning methods for tuples of different structures and a rounding down strategy, the execution duration of time slices is ensured to be controllable. Its pure software implementation path does not require hardware modification, breaking through the hardware compatibility limitations of existing splitting technologies and effectively balancing execution efficiency and memory usage in time-sharing multiplexing scheduling.

[0068] Example 2 This embodiment provides a time-slice segmentation system for CNN inference tasks based on separable tuples, including: The performance modeling module is used to decompose the CNN inference task into a sequence of sequentially executed BMOs and combine consecutive BMOs into tuples based on data dependencies. The tuple segmentation module is used to segment tuples based on the target time slice length according to the time estimation model of tuples, generating sub-tuples; wherein, the segmentation is divided according to the data dimension and rounded down to the smallest divisible unit; The time slice segmentation module is used to select tuples or sub-tuples from the tuple list to form a time slice, so that the total execution time does not exceed the target time slice length. Memory allocation is managed uniformly before the time slice is executed, and memory is released uniformly after execution.

[0069] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the time slice segmentation method for a CNN inference task based on separable tuples as described in Embodiment 1 above.

[0070] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the time slice segmentation method for a CNN inference task based on separable tuples as described in Embodiment 1 above.

[0071] Example 5 This embodiment provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the time slice segmentation method for a CNN inference task based on separable tuples as described in Embodiment 1 above.

[0072] The steps or modules involved in Embodiments 2 to 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for slicing time slices of CNN inference tasks based on splittable tuples, characterized in that, The method comprises the following steps: decompose the CNN inference task into a BMO sequence executed sequentially, combine consecutive BMOs into a tuple based on data dependency; a tuple-based time estimation model is used to split the tuple according to the target time slice length to generate a sub-tuple; wherein the splitting is performed in the data dimension and is rounded down to the minimum divisible unit; select tuples or sub-tuples from the tuple list to combine into a time slice, so that the total execution time does not exceed the target time slice length, and the memory allocation is uniformly managed before the execution of the time slice and is uniformly released after the execution of the time slice.

2. The CNN inference task time slice splitting method based on splittable tuples according to claim 1, wherein, The BMO sequence is a logical operation with a determined start and end node, and the execution time is not affected by the start time and data storage location, and the time complexity is an inherent attribute feature.

3. The CNN inference task time slice splitting method based on splittable tuples according to claim 1, wherein, The data dependency refers to the input-output correlation relationship between consecutive BMOs: the output data of the previous BMO is used as the input data of the next BMO, and the two BMOs need to be executed based on the correlation relationship and cannot be independently split.

4. The CNN inference task time slice splitting method based on splittable tuples according to claim 1, wherein, The tuple-associated memory segment includes a read-write data segment, a read-only data segment, and a workspace segment.

5. The CNN inference task time slice splitting method based on splittable tuples according to claim 1, wherein, The tuple-based time estimation model is used to split the tuple according to the target time slice length to generate a sub-tuple; wherein the splitting is performed in the data dimension and is rounded down to the minimum divisible unit, and specifically comprises: establish an estimation model of the running time and the input size for each tuple, and the estimation model is obtained by accumulating the time estimation models of the BMOs in the tuple; predict the execution time of the tuple according to the estimation model, if the execution time of the tuple exceeds the preset target time slice length, split the tuple into sub-tuples according to the BMO input-output size correlation relationship, and use the downward rounding strategy in combination with the minimum divisible unit of the BMO to ensure that the execution time of the sub-tuple does not exceed the remaining time.

6. The CNN inference task time slice splitting method based on splittable tuples according to claim 1, wherein, The tuple or sub-tuple is selected from the start position of the tuple list of the CNN inference task, and the estimated execution time is accumulated to not exceed the preset target time slice length to form a time slice; if the last tuple cannot be completely placed in the current time slice, the tuple is split into sub-tuples, the sub-tuple meeting the time length requirement is added to the current time slice, and the unexecuted part is reinserted into the beginning of the tuple list; coordinate the data dependency within and across the time slices, uniformly allocate the required memory before the execution of the time slice, and release the workspace segment immediately after the execution of the time slice is completed. The method comprises the following steps:

7. A time-slice segmentation system for CNN inference tasks based on separable tuples, characterized in that, a performance modeling module is used to decompose the CNN inference task into a BMO sequence executed sequentially, and combine consecutive BMOs into a tuple based on data dependency; a tuple splitting module is used to split the tuple according to the target time slice length to generate a sub-tuple based on a tuple-based time estimation model; wherein the splitting is performed in the data dimension and is rounded down to the minimum divisible unit; a time slice splitting module is used to select tuples or sub-tuples from the tuple list to combine into a time slice, so that the total execution time does not exceed the target time slice length, and the memory allocation is uniformly managed before the execution of the time slice and is uniformly released after the execution of the time slice. ​ 8. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the CNN inference task time slice splitting method based on splittable tuples in any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the CNN inference task time slice splitting method based on splittable tuples in any one of claims 1-6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the CNN inference task time slice splitting method based on splittable tuples in any one of claims 1-6.

Images