Timing methods and devices, electronic devices, storage media, and program products

Abstract

This disclosure provides a timing arrangement method and apparatus, electronic device, storage medium, and program product. The method includes: determining at least one candidate time window for a second type of operation associated with each of the first type of operations in a memory operation sequence, based on the first time window occupied by each of the first type of operations in the time sequence; determining a first estimated duration for each time window in the time sequence based on the first time window corresponding to the first type of operation; determining a second estimated duration for each time window based on the at least one candidate time window for each second type of operation; determining a second time window corresponding to each second type of operation based on the first estimated duration and the second estimated duration of each time window; and determining the execution timing information of the memory operation sequence based on the first time window of each first type of operation and the second time window of the second type of operation. This disclosure can reduce the total execution latency of the memory operation sequence.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a timing arrangement method and apparatus, electronic equipment, storage medium, and program product. Background Technology

[0002] With the continuous development of computer technology, there are many software engines or hardware engines in various types and scales of software and hardware systems that can run independently and asynchronously and synchronize when needed. These asynchronous engines can cooperate with each other to complete certain processing tasks.

[0003] However, these asynchronous software or hardware engines each have specific task types and workloads, and therefore have fixed execution times. These fixed execution times cause asynchronous engines to increase the total execution latency when cooperating to complete certain processing tasks, such as executing the same memory operation sequence, which further affects the running speed of the corresponding software and hardware systems. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of this disclosure propose a timing arrangement method and apparatus, electronic device, storage medium, and program product. The timing arrangement method proposed in this disclosure can reduce the total execution latency of executing memory operation sequences.

[0005] According to a first aspect of the present disclosure, a timing arrangement method is provided, the method comprising:

[0006] Based on the first time window occupied by each first type of operation to be executed in the memory operation sequence in each time window of the time sequence, at least one candidate time window of the second type of operation associated with each first type of operation is determined from each time window of the time sequence; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation.

[0007] Based on the first time window corresponding to the first type of operation, determine the first estimated duration of each time window in the time series; the first estimated duration of a time window is the duration required for the first type of operation to be executed by the first engine within the time window.

[0008] Based on at least one candidate time window for each type of operation, a second estimated duration is determined for each time window in the time series; the second estimated duration of a time window is the duration required for the type of operation within the time window to be executed by the second engine.

[0009] Based on the first estimated duration and the second estimated duration of each time window, a second time window corresponding to each second type of operation is determined from at least one candidate time window for each second type of operation; wherein, when the second type of operation is executed by the second engine within the corresponding second time window, the total execution latency corresponding to the memory operation sequence is less than when the second type of operation is executed by the second engine within any candidate time window other than the corresponding second time window; the total execution latency corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window within the time sequence;

[0010] The execution timing information of the memory operation sequence is determined based on the first time window of each of the first type of operations and the second time window of the second type of operations associated with the first type of operations.

[0011] According to a second aspect of the present disclosure, a timing arrangement apparatus is provided, the apparatus comprising:

[0012] The first determining module is configured to determine at least one candidate time window of a second type of operation associated with each first type of operation in the time sequence based on the first time window occupied by each first type of operation to be executed in the memory operation sequence in each time window of the time sequence; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation.

[0013] The first estimation module is configured to determine the first estimated duration of each time window in the time series based on the first time window corresponding to the first type of operation; the first estimated duration of a time window is the duration required for the first type of operation to be executed by the first engine within the time window;

[0014] The second estimation module is configured to determine the second estimated duration of each time window in the time series based on at least one candidate time window for each second type of operation; the second estimated duration of a time window is the duration required for the second type of operation within the time window to be executed by the second engine.

[0015] The second determining module is configured to determine a second time window corresponding to each second type of operation from at least one candidate time window for each second type of operation based on the first estimated duration and the second estimated duration of each time window; wherein, when the second type of operation is executed by the second engine within the corresponding second time window, the total execution latency corresponding to the memory operation sequence is less than when the second type of operation is executed by the second engine within any candidate time window other than the corresponding second time window; the total execution latency corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window within the time sequence;

[0016] The timing determination module is configured to determine the execution timing information of the memory operation sequence based on the first time window of each of the first type of operations and the second time window of the second type of operations associated with the first type of operations.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] processor;

[0019] Memory used to store computer programs or instructions;

[0020] The processor executes the computer program or instructions to implement the steps of the method proposed in the first aspect above.

[0021] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method proposed in the first aspect.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method proposed in the first aspect.

[0023] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0024] In the timing arrangement method proposed in this embodiment, the second time window corresponding to each second type of operation is determined when the total execution delay of the memory operation sequence is minimized, based on the time required for each first type of operation to be executed by the first engine, the time required for each second type of operation associated with the first type of operation to be executed by the second engine, the first time window corresponding to each first type of operation, and at least one candidate time window that each second type of operation can select. Based on the first time window corresponding to each first type of operation and the second time window corresponding to each second type of operation, the execution timing information for timing arrangement of the memory operation sequence is obtained.

[0025] Thus, when determining the second time window for the second type of operation associated with the first type of operation, compared to selecting a time window as the second time window for the second type of operation based on the first time window of the first type of operation and the timing relationship between the first and second types of operations, this embodiment first determines multiple candidate time windows for the second type of operation from multiple time windows in the time series, aiming to minimize the total execution latency of the memory operation sequence, and then determines the second time window from multiple candidate time windows. On the one hand, this makes the timing arrangement of the second type of operation more flexible; on the other hand, because the determination of the second time window takes into account the total execution latency of the memory operation sequence when the second type of operation is executed in different candidate time windows, it can minimize the total execution latency of the memory operation sequence and improve the speed and efficiency of the computer system in executing the memory operation sequence.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0028] Figure 1 This is a flowchart illustrating a timing arrangement method according to an exemplary embodiment.

[0029] Figure 2 This is a schematic diagram illustrating the asynchronous engine execution principle according to an exemplary embodiment.

[0030] Figure 3 This is a schematic diagram illustrating the execution principle of a memory operation sequence according to an exemplary embodiment.

[0031] Figure 4 This is a temporal distribution of a memory operation sequence according to an exemplary embodiment. Figure 1 .

[0032] Figure 5 This is a schematic diagram illustrating the configuration of a first variable and a second variable for a sequence of memory operations, according to an exemplary embodiment.

[0033] Figure 6 This is a schematic diagram illustrating the setting of a second constraint condition according to an exemplary embodiment.

[0034] Figure 7 This is a schematic diagram illustrating the determination of a first estimated duration and a second estimated duration according to an exemplary embodiment.

[0035] Figure 8 This is a topological illustration of a memory operation sequence according to an exemplary embodiment. Figure 1 .

[0036] Figure 9 This is a schematic diagram illustrating the setting of a third variable, a first variable of a first target sub-operation, and a second variable of a second target sub-operation, according to an exemplary embodiment.

[0037] Figure 10 This is a topological illustration of a memory operation sequence according to an exemplary embodiment. Figure 2 .

[0038] Figure 11 This is a schematic diagram illustrating a memory management principle according to an exemplary embodiment.

[0039] Figure 12 This is a memory resource distribution of the first cache area according to an exemplary embodiment. Figure 1 .

[0040] Figure 13 This is a memory resource distribution of the first cache area according to an exemplary embodiment. Figure 2 .

[0041] Figure 14 This is a schematic diagram of a resident storage area according to an exemplary embodiment. Figure 1 .

[0042] Figure 15 This is a schematic diagram of a resident storage area according to an exemplary embodiment. Figure 2 .

[0043] Figure 16 This is a timing diagram illustrating the storage of fourth target data into a first kernel and a second kernel according to an exemplary embodiment.

[0044] Figure 17 This is a schematic diagram illustrating time window fusion according to an exemplary embodiment.

[0045] Figure 18This is a temporal distribution of a memory operation sequence according to an exemplary embodiment. Figure 2 .

[0046] Figure 19 This is a schematic diagram of a timing arrangement device according to an exemplary embodiment.

[0047] Figure 20 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] This disclosure provides a timing arrangement method that can effectively reduce the total execution time of memory operation sequences.

[0050] See Figure 1 , Figure 1 This is a flowchart illustrating a timing arrangement method according to an exemplary embodiment. The timing arrangement method in this disclosure can be implemented by the following steps:

[0051] Step 101: Based on the first time window occupied by each first type of operation to be executed in the memory operation sequence in each time window of the time series, determine at least one candidate time window of the second type of operation associated with each first type of operation from each time window of the time series; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation.

[0052] Step 102: Based on the first time window corresponding to the first type of operation, determine the first estimated duration of each time window in the time series; the first estimated duration of a time window is the duration required for the first type of operation within the time window to be executed by the first engine.

[0053] Step 103: Based on at least one candidate time window for each type of operation, determine the second estimated duration of each time window in the time series; the second estimated duration of a time window is the duration required for the type of operation within the time window to be executed by the second engine.

[0054] Step 104: Based on the first and second estimated durations of each time window, determine the second time window corresponding to each second type of operation from at least one candidate time window of each second type of operation; wherein, when the second type of operation is executed by the second engine within the corresponding second time window, the total execution latency corresponding to the memory operation sequence is less than when the second type of operation is executed by the second engine within any candidate time window other than the corresponding second time window; the total execution latency corresponding to the memory operation sequence is determined by the difference between the first and second estimated durations of each time window within the time series;

[0055] Step 105: Determine the execution timing information of the memory operation sequence based on the first time window of each first type of operation and the second time window of the second type of operation associated with the first type of operation.

[0056] The timing arrangement method proposed in this disclosure can be applied to various types and sizes of software and hardware systems, and implemented in specific application scenarios involving the execution of memory operation sequences in these systems. These software and hardware systems are located within electronic devices, which can be portable electronic devices, computer devices, computer clusters, etc., and this disclosure does not impose any limitations on this.

[0057] It should be noted that the memory operation sequence proposed in this embodiment includes memory operations that are sequentially continuous. These memory operations can be the calculation operations of each network node in the network model on the input data, or the instruction operations executed by the electronic device in response to user instructions, etc., and are not limited to these.

[0058] For example, when the above memory operation sequence is applied to an artificial intelligence (AI) network model, these memory operations are represented as network operations executed by the network layers on each network node, and the corresponding memory operation sequence is represented as a directed acyclic graph of the AI ​​model; where the execution flow from one network node to the next network node represents the execution order (timing) of two memory operations, and the execution flows continuously branch and merge to form the directed acyclic graph of the AI ​​model.

[0059] In this embodiment of the disclosure, multiple different memory operations within a memory operation sequence can be executed by at least two engines. Although the at least two engines can run independently and asynchronously, the execution time required for each engine to execute its corresponding memory operation is fixed and cannot be compressed. Furthermore, there are synchronization points between the memory operations executed by the at least two engines, requiring them to wait for each other at these points until both engines reach the synchronization point before proceeding with subsequent operations. For example, before one engine can execute a memory operation, it must wait for the other engine to complete the data import operation associated with that memory operation. Therefore, orchestrating the execution order of each memory operation in the memory operation sequence to reduce the overall execution latency of the entire memory operation sequence becomes increasingly important.

[0060] The timing arrangement method corresponding to steps 101 to 105 proposed in this embodiment enables the execution time of at least two engines to be hidden from each other, thereby reducing the total execution latency of the entire memory operation sequence.

[0061] In step 101, the present embodiment first determines the first time window of each first type of operation to be executed in the memory operation sequence within multiple time windows of the time series; wherein, the first type of operations to be executed have a predetermined execution order (timing), one first time window corresponds to one first type of operation, and the corresponding first type of operation can be executed by the first engine within each first time window.

[0062] Here, the first type of operation corresponds to each memory operation in the memory operation sequence. When these memory operations are executed by the first engine, they need to utilize the memory resources in the hardware and software system. It should be noted that, due to the large capacity of the memory area in the system and the slow speed of data access (reading and writing), the memory resources of a designated cache area are usually used to execute the above memory operations. The designated cache area can be an on-chip cache, such as the first cache area (L1 cache) proposed below in this disclosure. The distance between the first cache area and the processor (CPU) in the system is closer than the distance between the memory area and the CPU. The memory capacity of the first cache area is smaller than the capacity of the memory area in the system. Furthermore, the speed at which the CPU accesses data in the first cache area is faster than the speed at which it accesses data in the memory area in the system.

[0063] For example, in an AI model, the first type of operation can be convolution, pooling, nonlinear mapping, etc., corresponding to each network node, and the first engine corresponding to the execution of the first type of operation can be a computation engine or a forward and backward inference engine, etc.

[0064] In some examples, the computing engine can perform convolution processing on the input data within the corresponding first time window. For example, when the input data is an image, the convolution processing can yield a dimension-reduced image. The computing engine can also perform pooling processing, nonlinear mapping processing, etc. on the input data within the corresponding time window, which will not be elaborated here in the embodiments of this disclosure.

[0065] In other examples, the forward and backward inference engines can perform forward computation on memory operations based on the input data within the corresponding first time window to obtain the result data for the next memory operation in the time sequence, or perform backward computation on memory operations to obtain the result data for the previous memory operation in the time sequence.

[0066] In this embodiment of the disclosure, the first type of operation in the memory operation sequence is usually associated with the second type of operation. The complete execution of the memory operation sequence depends on the execution of the second type of operation associated with the first type of operation by the second engine. Continuing the above example, if the first engine is a computing engine, then the second engine can be a data in / out (also known as direct memory access, DMA) engine; correspondingly, the second type of operation can be using the DMA engine to move the input data required for the execution of the first type of operation from the memory area into the first buffer area in advance (hereinafter referred to as data in / out operation), or to move the result data after the execution of the first type of operation from the first buffer area to the memory area (hereinafter referred to as data out / out operation); if the first engine is a forward / backward inference engine, then the second engine can be a parameter synchronization communication engine between nodes, and correspondingly, the second type of operation can be an update operation of the model parameters of the network layer for each network node.

[0067] See Figure 2 , Figure 2This is a schematic diagram illustrating the asynchronous engine execution principle according to an exemplary embodiment. The on-chip cache (first cache area) is a one-dimensional linear physical memory space, typically with a capacity of 256KB. The memory area is Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM). The DMA engine can perform input data loading for the first type of operation, i.e., loading input data from DDR into the first cache area. After loading, the computation engine uses the input data in the first cache area to perform calculations, and then outputs the result data to the first cache area for caching. Furthermore, the DMA engine loads the calculation result from the first cache area back to DDR. Of course, if the result data of the first type of operation is the input data for the next first type of operation to be executed, the result data can also reside in the first cache area. After loading other input data for the next first type of operation into the allocated location in the first cache area, the next first type of operation can be executed directly.

[0068] For the second type of operation, if the second engine needs to occupy a separate time window for each second type of operation associated with the first type of operation, it will increase the overall execution time. Since the second engine and the first engine can execute asynchronously, for a first type of operation, the associated second type of operation can be executed in a time window before (or after) the first type of operation.

[0069] See Figure 3 and Figure 4 , Figure 3 This is a schematic diagram illustrating the execution principle of a memory operation sequence according to an exemplary embodiment; Figure 4 This is a temporal distribution of a memory operation sequence according to an exemplary embodiment. Figure 1 .

[0070] here, Figure 3 The first type of operations that the computational engine can execute include operations 1 to 4; the input data required for the execution of operations 1 to 4 includes computational data and weight data; the output data includes result data; wherein, the computational data includes the objects acted upon by the network layers of each network node of the AI ​​model, and the weight data includes the model parameters of the network layers of the network nodes; it should be noted that in some scenarios, intermediate outputs between adjacent operations can be directly transferred through the cache of the first buffer, for example... Figure 3 Operation 2 in the middle. Figure 4In this context, the time sequence consists of the start time window (time window 0), each first time window (time windows 1 to 4), and the end time window (time window 5). In some embodiments, a related second-type operation (such as a data import operation, i.e., importing data for operation 1, weighting data for operation 2, and weighting data for operation 3) can be executed in the preceding time window of each first-type operation, and another related second-type operation (such as a data export operation, i.e., exporting the result data for operation 4) can be executed in the following time window. While this timing arrangement reduces execution latency to some extent, it lacks flexibility and may cause memory pressure on the first cache at different time windows, leading to the failure of the second engine.

[0071] In view of this, the embodiments of this disclosure can, for each first type of operation, use a first time window corresponding to another first type of operation that is different from the first type of operation, or a start time window in a time sequence used to start a memory operation sequence, or an end time window used to end the execution of a memory operation sequence, as a window in which the second type of operation associated with the first type of operation can be selected for execution, that is, determine at least one corresponding candidate time window; wherein, since the associated first type of operation and the second type of operation cannot be time compressed if they are executed in the same time window, it is set that the associated first type of operation and the second type of operation are not executed in the same time window, that is, the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation.

[0072] In step 102, based on the test results or historical execution results of each memory operation sequence, the time required for the first type of operation in each memory operation sequence to be executed by the first engine is determined; according to the first time window corresponding to each first type of operation and the execution time required, the first estimated time corresponding to each time window on the time sequence is obtained.

[0073] The execution time required for the first type of operation is represented by the number of clock cycles required for the first engine to execute the first type of operation using the memory resources of the first cache. If the time window is the first time window, the value of the first estimated duration of the time window is represented by the estimated value of the execution time required for the first type of operation within the time window. If the time window is not the first time window, the first type of operation is not executed within the time window, and the value of the first estimated duration of the time window is 0.

[0074] In step 103, based on the test results or historical execution results of each memory operation sequence, the time required for the second type of operation associated with the first type of operation in each memory operation sequence to be executed by the second engine is determined; according to at least one candidate time window corresponding to each first type of operation and the execution time required, the second estimated time corresponding to each time window on the time sequence is obtained.

[0075] The execution time required for the second type of operation is represented by the number of clock cycles required for the second engine to execute the second type of operation using the memory resources of the first cache. If the time window is a candidate time window corresponding to at least one second type of operation, the value of the second estimated duration of the time window is represented by the sum of the execution times required for all second type of operations that may be executed within the time window. If the time window is not a candidate time window corresponding to any second type of operation, no second type of operation is executed within the time window, and the value of the second estimated duration of the time window is 0.

[0076] It should be noted that the first and second estimated durations of each time window in the time series statistics in steps 102 and 103 are estimated from the execution process of a single memory operation sequence.

[0077] In step 104, since the second type of operation for the same first type of operation does not need to be repeated, that is, in actual implementation, each second type of operation is executed by the second engine only in one time window of at least one corresponding candidate time window; therefore, when each second type of operation selects different candidate time windows for execution, the present embodiment can obtain the total execution delay of the memory operation sequence under different execution conditions based on the first estimated duration and the second estimated duration of each time window in the time sequence; wherein, the total execution delay of the memory operation sequence is obtained by summing the delays corresponding to each time window when the memory operation sequence is fully executed.

[0078] The delay of each time window is the absolute value of the difference between the first estimated duration and the second estimated duration of each time window. It represents the overlapping part of the execution time when performing the first type of operation and the possible second type of operation within the corresponding time window. These overlapping parts are the parts that cannot be hidden within the time window, that is, the unavoidable delays generated when the asynchronous engine executes within the same corresponding time window. Therefore, the sum of the delays of each time window can represent the total execution delay of the memory operation sequence.

[0079] In this embodiment of the disclosure, since the total execution latency of the memory operation sequence varies when different candidate time windows are selected for execution of each second type of operation, if the total execution latency is minimized, it can be understood that the execution time of the asynchronous engine has achieved maximum hiding and compression in the entire time sequence; therefore, in this embodiment of the disclosure, the candidate time window corresponding to each second type of operation when the total latency is minimized is taken as the second time window of each second type of operation.

[0080] It should be noted that the embodiments of this disclosure can use mathematical models such as data analysis and linear programming to directly analyze and evaluate the total execution delay of the memory operation sequence, thereby determining the second time window corresponding to each second type of operation; the embodiments of this disclosure can also use the hypothesis-proof method to solve the total execution delay of various cases separately, and verify the accuracy of the candidate time window selected for each second type of operation being the second time window.

[0081] In step 105, based on the first time window of each first type of operation in the memory operation sequence and the second time window determined for each second type of operation in step 104, the execution timing information of the memory operation sequence executed using the asynchronous first engine and second engine can be determined; wherein, the execution timing information is a timing arrangement result used to indicate the first type of operation and / or the second type of operation that needs to be executed in each time window of the time sequence.

[0082] In the timing arrangement method proposed in this embodiment, the second time window corresponding to each second type of operation is determined when the total execution delay of the memory operation sequence is minimized, based on the time required for each first type of operation to be executed by the first engine, the time required for each second type of operation associated with the first type of operation to be executed by the second engine, the first time window corresponding to each first type of operation, and at least one candidate time window that each second type of operation can select. Based on the first time window corresponding to each first type of operation and the second time window corresponding to each second type of operation, the execution timing information for timing arrangement of the memory operation sequence is obtained. Thus, when determining the second time window for the second type of operation associated with the first type of operation, compared to selecting a time window as the second time window for the second type of operation based on the first time window of the first type of operation and the timing relationship between the first and second types of operations, this embodiment first determines multiple candidate time windows for the second type of operation from multiple time windows in the time series, aiming to minimize the total execution latency of the memory operation sequence, and then determines the second time window from multiple candidate time windows. On the one hand, this makes the timing arrangement of the second type of operation more flexible; on the other hand, because the determination of the second time window takes into account the total execution latency of the memory operation sequence when the second type of operation is executed in different candidate time windows, it can minimize the total execution latency of the memory operation sequence and improve the speed and efficiency of the computer system in executing the memory operation sequence.

[0083] In some embodiments, determining the second estimated duration of each time window in the time series based on at least one candidate time window for each second type of operation includes:

[0084] For each type of operation, at least one candidate time window is set, and multiple decision variables are set; each decision variable corresponds to a candidate time window of a type of operation; different values ​​of the decision variables are used to indicate whether the type of operation is executed by the second engine within the corresponding candidate time window.

[0085] Based on the expected execution of at least one type II operation within each time window in the time series, the decision variables corresponding to the type II operation, and the execution time required to execute the type II operation, determine the second estimated duration of each time window;

[0086] The above-mentioned method, based on the first and second estimated durations of each time window, determines the second time window corresponding to each second type of operation from at least one candidate time window, including:

[0087] The total execution latency corresponding to the memory operation sequence is determined based on the first and second estimated durations of each time window in the time series.

[0088] The objective function is to minimize the total execution delay corresponding to the sequence of memory operations.

[0089] Based on the constraints and objective function of multiple decision variables, the target values ​​of multiple decision variables are determined; wherein, the target values ​​of the decision variables are used to determine the second time window of the second type of operation corresponding to the decision variables.

[0090] Here, for each candidate time window that can be selected for execution of the second type of operation, a corresponding decision variable is set; these decision variables have two different selectable values; for example, the value of the decision variable corresponding to each second type of operation can be 1 or 0; where, when the value of the decision variable is 1, it indicates that the second engine needs to execute the second type of operation corresponding to the decision variable within the candidate time window; when the value of the decision variable is 0, it indicates that the second type of operation corresponding to the decision variable does not need to be executed within the corresponding candidate time window.

[0091] After setting the decision variables, based on the expected execution of at least one second-type operation in each time window and the execution time required for each second-type operation, the product of each decision variable and the execution time required for each second-type operation in each time window is first obtained, and the product is summed to obtain the summed result. The summed results of each second-type operation within the time window are summed again to obtain the second estimated duration of each time window. The delay between the first estimated duration and the second estimated duration in each time window of the time series is summed to obtain the total execution delay of the memory operation sequence.

[0092] Understandably, since the second estimated duration is determined by the decision variables, and the total execution latency of the memory operation sequence is determined by the second estimated duration of each time window, the total execution latency varies depending on the different values ​​of the decision variables. Thus, this embodiment aims to minimize the total execution latency, determines the objective function, and uses constraints on multiple decision variables as limitations to constrain the objective function into a linear problem, obtaining a linear programming model. Furthermore, it uses OR tools (OR-Tools) and other related technologies to perform linear analysis on this linear programming model, determining the target value of each decision variable when the objective function is minimized.

[0093] Here, in each second type of operation, the candidate time window corresponding to the objective value of 1 in at least one decision variable is used as the second time window of that second type of operation.

[0094] It should be noted that the constraints on multiple decision variables can be time constraints, memory resource constraints, or environmental constraints on the execution of the first type of operation and the associated second type of operation, etc. For example, time constraints can be restrictions on the execution time of the second type of operation. For instance, some second type of operations must be executed within a certain candidate time window, so the value of the decision variable for that candidate time window is set to 1, and the value of other existing candidate time windows is set to 0. Memory resource constraints can be constraints on whether the second type of operation can be executed within a candidate time window based on whether memory resources are sufficient, which is represented as constraints on the value of the variable corresponding to the candidate time window. Environmental constraints can be restrictions on the number / type of operations that can be executed within a time window, combined with the resource consumption or system power consumption of other running processes / threads in the computer system, which is represented as constraints on the value of the variable corresponding to the candidate time window. Of course, the specific solutions for the above-mentioned time constraints, memory resource constraints, and environmental constraints are not limited to the examples described above. In actual scenarios, there are naturally other actual constraints, which will not be further elaborated here.

[0095] Thus, this embodiment sets multiple decision variables for each second type of operation, which can transform the analysis of the total execution latency of the memory operation sequence into a linear problem of solving the objective function. This can improve the rationality and accuracy of determining the second time window for the second type of operation. Moreover, compared with the scheme of calculating the minimum value of the total execution latency under different conditions to determine each second time window, it saves calculation time, simplifies the calculation logic, and thus effectively improves the efficiency of timing arrangement for the memory operation sequence.

[0096] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0097] The first constraint is determined based on the constraint relationship between the decision variables corresponding to at least one candidate time window for each second type of operation;

[0098] For each time window of the time series, determine the first estimated memory amount of the first type of operation corresponding to the first engine execution time window in the first cache.

[0099] Based on the amount of memory occupied by the operation data associated with each second type of operation, and at least one decision variable corresponding to the second type of operation, determine the second estimated amount of memory occupied by the operation data associated with at least one second type of operation expected to be executed within each time window in the first cache.

[0100] Based on the first and second estimated memory amounts within each time window, and the memory amount of the first cache, a second constraint is determined; the second constraint restricts the sum of the first and second estimated memory amounts within each time window to be less than the memory amount of the first cache.

[0101] The above method, based on the constraints and objective function of multiple decision variables, determines the target values ​​of these variables, including:

[0102] Based on the objective function and the first and second constraints, the target values ​​of multiple decision variables are determined.

[0103] Here, by considering the temporal relationship between the second type of operation and the associated first type of operation, as well as the execution requirements of the second type of operation, a first constraint condition can be set for the values ​​of the multiple decision variables of each second type of operation. The first constraint condition can remove some free values ​​of the multiple decision variables and constrain the value of one decision variable or multiple decision values.

[0104] In some examples, the first constraint can constrain the value of a decision variable; for example, if second-type operations are not allowed to be performed in certain specific time windows, the decision variable for any second-type operation that may exist in this time window is set to 0; as another example, if the first-type operation is the first memory operation in a memory operation sequence, and the associated second-type operation is the input data loading operation of the first memory operation, then the decision variable for the data loading operation in the start time window is set to 1; as yet another example, if the first-type operation is the last memory operation in a memory operation sequence, and the associated second-type operation is the result data unloading operation of the last memory operation, then the decision variable for the data unloading operation in the end time window is set to 1.

[0105] In other examples, the first constraint can also constrain multiple decision variables of a second type of operation simultaneously; for example, when there are multiple candidate time windows for a second type of operation, the values ​​of the decision variables of the first preset number of candidate time windows corresponding to the first time window of the far-from-related first type of operation can all be set to 0; wherein, the first preset number can be 1 / 2 or 2 / 3 of the total number of candidate time windows corresponding to the second type of operation, and of course, the first preset number can also be other preset values, which are not limited in this disclosure.

[0106] In this embodiment of the disclosure, in addition to the first constraint condition described above, the second constraint condition for the values ​​of multiple decision variables is also determined based on the amount of memory in the first cache area.

[0107] Since both the first engine executing the first type of operation and the second engine executing the second type of operation require the use of on-chip cache memory resources, the amount of memory occupied by executing the first type of operation and / or the second type of operation in each time window can be obtained through historical execution and testing; and the amount of memory occupied by executing the first type of operation and / or the second type of operation in each time window cannot exceed the total memory of the first cache in that time window. In this way, by constraining the amount of memory, the second type of operation that cannot be executed in some time windows can be constrained, that is, the value of the decision variable can be constrained.

[0108] Specifically, in this embodiment of the present disclosure, the first estimated memory amount for each time window of the time series is estimated based on the memory amount of the first type of operation executed by the first engine in the first cache; then, the second estimated memory amount for each memory operation is determined based on the memory amount occupied by the operation data associated with each second type of operation, the type of the second type of operation, and at least one associated decision variable.

[0109] It should be noted that the associated operation data differs depending on the type of the second type of operation. For example, if the second type of operation is a data import operation relative to the first type of operation, the associated operation data is the input data required to execute the first type of operation; if the second type of operation is a data export operation relative to the first type of operation, the associated operation data is the result data after the first type of operation is executed. The time window for the data import operation determines from which the input data begins to occupy memory, while the time window for the data export operation determines the memory occupation of the result data within the time window between the first time window and the first time window corresponding to the first type of operation. Therefore, for the second estimated memory amount within each time window, when the type of the second type of operation to be executed within that time window is different, it is jointly determined by the decision variable corresponding to that time window and the decision variables and the memory occupied by the operation data of the candidate time windows before (or after) the same second type of operation.

[0110] Thus, by using the first and second estimated memory amounts in each time window, the total memory usage generated in each time window is determined, and the memory amount in each time window where the sum of the first and second estimated memory amounts is less than the first cache amount is taken as the second constraint condition. This second constraint condition can statistically constrain the possible memory amounts in each time window, thereby limiting the values ​​of each decision variable.

[0111] Therefore, through the first and second constraints, the embodiments of this disclosure can impose linear constraints on the values ​​of each decision variable in the objective function, thereby determining more reasonable and realistic target values ​​for each decision variable.

[0112] In some embodiments, determining at least one candidate time window for a second type of operation associated with each first type of operation from the time windows of the time series based on the first time window occupied by each first type of operation to be executed in the memory operation sequence includes:

[0113] Based on the first time window of each first type of operation to be executed in the memory operation sequence, at least one time window in the time sequence that is earlier than the first time window is determined as at least one candidate time window of the first sub-operation associated with the first type of operation; wherein, the execution time of the first sub-operation is earlier than the execution time of the first type of operation associated with the first sub-operation;

[0114] Based on the first time window of each first type of operation, at least one time window in the time series that is later than the first time window is determined as at least one candidate time window of the second sub-operation associated with the first type of operation; the execution time of the second sub-operation is later than the execution time of the first type of operation associated with the second sub-operation.

[0115] The decision variables include: the first variable corresponding to the first sub-operation, and the second variable corresponding to the second sub-operation.

[0116] Here, the second type of operation associated with the first type of operation includes the first sub-operation and the second sub-operation.

[0117] In one example of a scenario disclosed herein, when the memory operation sequence is applied to an AI model training scenario, the first engine is a forward and backward inference engine, and the second engine is a parameter synchronization communication engine between nodes. In this case, the first type of operation includes computation operations in two propagation directions (forward and backward). For the forward computation operation, the first sub-operation corresponding to the second type of operation can be a weight synchronization operation of the communication engine on the weight data corresponding to the network nodes. Since the forward computation of the network nodes can only be performed after the weight update synchronization, this embodiment can determine at least one time window whose timing is earlier than the first time window corresponding to the forward computation operation as at least one candidate time window for the aforementioned weight synchronization operation. Furthermore, for the backward computation operation (i.e., the backward gradient momentum parameter update operation), the second sub-operation corresponding to the second type of operation can be a gradient synchronization operation of the gradient momentum parameter by the communication engine. Since the gradient momentum parameter can only be updated after calculation, this embodiment can determine at least one time window whose timing is later than the first time window of the backward computation operation as at least one candidate time window for the aforementioned gradient synchronization operation.

[0118] Correspondingly, the decision variables set in the embodiments of this disclosure may include a first variable corresponding to the first sub-operation (e.g., the first variable corresponding to the weight synchronization operation mentioned above) and a second variable corresponding to the second sub-operation (e.g., the second variable corresponding to the gradient synchronization operation mentioned above).

[0119] In another example of this disclosure, when the memory operation sequence is applied to an AI model application scenario, the first engine is a computation engine and the second engine is a DMA engine. Continuing from the previous example, the first type of operation can be a convolution operation, a pooling operation, a non-linear mapping operation, etc.; the first sub-operation of the second type of operation is a data loading operation for the input data of the first type of operation, and the second sub-operation of the second type of operation is a data loading operation for the result data of the first type of operation.

[0120] Here, because the first type of operation can only be executed by the first engine after the input data is loaded, this embodiment of the disclosure determines at least one time window whose timing is earlier than the first time window of the first type of operation as at least one candidate time window for the data loading operation; in addition, because the result data is generated only after the first type of operation is executed, this embodiment of the disclosure determines at least one time window whose timing is later than the first time window of the first type of operation as at least one candidate time window for the data unloading operation.

[0121] Correspondingly, the decision variables set in the embodiments of this disclosure may include a first variable corresponding to the first sub-operation (e.g., the import variable corresponding to the data import operation mentioned above) and a second variable corresponding to the second sub-operation (e.g., the export variable corresponding to the data export operation mentioned above).

[0122] It should be noted that the embodiments of this disclosure, including the first sub-operation, the second sub-operation, the decision variable, and the steps of setting the constraints of the variable and solving the objective function, can be adapted to the examples of the two scenarios mentioned above. This disclosure will use the example of the second scenario (an example where the asynchronous engine is a computing engine and a DMA engine, the first sub-operation is a data import operation, and the second sub-operation is a data export operation) for introduction and explanation. The steps and processes of the first example are also applicable, and this disclosure will not repeat them in detail below.

[0123] This embodiment of the disclosure considers the temporal correlation between different sub-operations of the second type of operation and the first type of operation, sets the candidate time window corresponding to the first sub-operation to be earlier than the first time window of the associated first type of operation, and sets the candidate time window corresponding to the second sub-operation to be later than the first time window of the associated first type of operation, thereby making the design of decision variables more reasonable and improving the scientificity and accuracy of the temporal arrangement.

[0124] In some embodiments, determining the first constraint condition based on the constraint relationship between decision variables corresponding to at least one candidate time window for each second type of operation includes:

[0125] Determine the first sum value among the first variables corresponding to each candidate time window of each first sub-operation;

[0126] The first sum of the first variables in each first sub-operation is set to equal the first value, thus obtaining the first constraint relationship; the first constraint relationship is used to restrict the second engine to execute the first sub-operation in one of the candidate time windows of at least one candidate time window of the first sub-operation;

[0127] Determine the second sum value among the second variables corresponding to each candidate time window of each second sub-operation;

[0128] The second sum value between the second variables of each second sub-operation is set to equal the first value, thus obtaining the second constraint relationship; the second constraint relationship is used to restrict the second engine to execute the second sub-operation in one of the candidate time windows of at least one candidate time window of a second sub-operation;

[0129] Determine the first constraint condition based on the first and second constraint relationships.

[0130] In the example of the second scenario described above, the first sub-operation is a data import operation, and the second sub-operation is a data export operation. The first value can be 1. Thus, by setting the first sum of the first variables corresponding to each candidate time window of each data import operation to equal the first value, this embodiment of the present disclosure can restrict that in a data import operation corresponding to a first type of operation, only one first variable has a value of 1, while the other first variables have values ​​of 0. That is, a data import operation of a first type of operation is executed within only one time window. Similarly, by setting the second sum of the second variables corresponding to each candidate time window of each data export operation to equal the first value, this disclosure can restrict that in a data export operation corresponding to a first type of operation, only one first variable has a value of 1, while the other first variables have values ​​of 0. That is, a data export operation of a first type of operation is executed within only one time window.

[0131] It should be noted that a data loading operation corresponding to a first type of operation can be one or more, loading different input data (such as calculation data and weight data) required for the execution of the first type of operation into the first buffer. Thus, in this embodiment of the present disclosure, one or more first variables can be set for each first sub-operation (each corresponding to one or more data loading operations), and the first constraint relationship of at least one first variable for the same first sub-operation is the same as above, which will not be further elaborated here.

[0132] By providing the aforementioned first and second constraint relationships, the execution of the first and second sub-operations can be constrained, thereby improving the rationality and accuracy of the timing arrangement proposed in this embodiment.

[0133] In some embodiments, determining the second estimated memory amount occupied by the operation data associated with at least one second-type operation to be executed within each time window in the first cache, based on the amount of memory occupied by the operation data associated with each second-type operation and at least one decision variable corresponding to the second-type operation, includes:

[0134] From the multiple first variables associated with the first sub-operation expected to be executed in the i-th time window, determine at least one first target variable of the first sub-operation in the i-th time window; the temporal sequence of the candidate time window corresponding to at least one first target variable is less than or equal to i; i is greater than or equal to 1 and less than or equal to N, where N is the number of time windows contained in the time series;

[0135] From the multiple second variables associated with the second sub-operation expected to be executed in the i-th time window, determine at least one second target variable for the second sub-operation within the i-th time window; the temporal sequence of the candidate time window corresponding to at least one second target variable is greater than or equal to i.

[0136] Based on the amount of memory occupied by at least one first target variable and operation data associated with the first sub-operation in the i-th time window, and the amount of memory occupied by at least one second target variable and operation data associated with the second sub-operation in the i-th time window, determine the second estimated amount of memory occupied by operation data associated with at least one second type of operation expected to be executed in the i-th time window in the first cache.

[0137] Referring to the example of the second scenario above, the first sub-operation is a data import operation, and the second sub-operation is a data export operation. Since the time window for the data import operation determines when the input data begins to occupy memory, if there is a desired data import operation in the i-th time window, then among the first variables corresponding to that data import operation, the first variable corresponding to the i-th time window and the first variables corresponding to the candidate time windows of the same data import operation with earlier timing sequences are all used as the first target variable. Thus, in this embodiment, the sum of at least one first target variable corresponding to each data import operation in the i-th time window can be multiplied by the amount of memory occupied by the operation data associated with that data import operation to obtain a product result; and the sum of this product result corresponding to all possible data import operations in the i-th time window is used as the first sub-estimated memory amount for the data import operations within the i-th time window.

[0138] Similarly, since the time window for selecting the execution of the data fetching operation determines the memory usage of the result data within the time window between that time window and the first time window corresponding to the first type of operation, if there is a data fetching operation to be executed in the i-th time window, then among the second variables corresponding to the data fetching operation, the second variable corresponding to the i-th time window, and the second variables corresponding to each candidate time window of the same data fetching operation in a later sequence, are used as the second target variable. Thus, in this embodiment, the sum of at least one second target variable corresponding to each output fetching operation in the i-th time window can be multiplied by the amount of memory occupied by the operation data associated with the data fetching operation to obtain the product result. The sum of the product results corresponding to all possible data fetching operations in the i-th time window is used as the second sub-estimated memory amount for the data fetching operation within the i-th time window.

[0139] Furthermore, based on the sum of the first and second sub-estimated memory amounts in the i-th time window, the second estimated memory amount occupied by the operation data associated with at least one second-type operation expected to be executed within the i-th time window is determined in the first cache.

[0140] Here, the embodiments of this disclosure improve the comprehensiveness and accuracy of determining the second estimated memory amount by analyzing the first sub-operation and the second sub-operation that may be executed within each time window.

[0141] Below, using the example of the second scenario described above, we will provide an exemplary description of configuring the first variable and the second variable, setting the first constraint and the second constraint, and configuring the objective function in this embodiment of the disclosure.

[0142] See Figure 5 , Figure 5 This is a schematic diagram illustrating the configuration of a first variable and a second variable for a memory operation sequence according to an exemplary embodiment; wherein each first type of operation in the memory operation sequence includes operation 1 to operation 3; time window 0 is the start time window, time windows 1 to 3 are the first time windows corresponding to operation 1 to 3 respectively, and time window 4 is the end time window. The first variable configured in this embodiment includes Figure 5 The second variables configured include x5 to x8, as shown in x1 to x4.

[0143] In this context, x1 corresponds to time window 0 and is the first variable for the data import operation corresponding to the calculated data of operation 1. Time window 0 is a candidate time window for x1. x2 corresponds to time window 0, x3 corresponds to time window 1, and x4 corresponds to time window 2. x2, x3, and x4 are the three first variables for the data import operation corresponding to the weighted data of operation 3. Time windows 0, 1, and 2 are the three candidate time windows for the data import operation corresponding to the weighted data of operation 3. x5 corresponds to time window 2, x6 corresponds to time window 3, and x7 corresponds to time window 4. x5, x6, and x7 are the three second variables for the data export operation corresponding to the result data of operation 1. Time windows 2, 3, and 4 are the three candidate time windows for the data export operation corresponding to the result data of operation 1. x8 corresponds to time window 4 and is the second variable for the data export operation corresponding to the result data of operation 3. Time window 4 is a candidate time window for x8.

[0144] It should be noted that, because the input data required for the first type of operation is different for different network nodes, for example, operation 1 does not require weight data, operation 2 can use the cache of the first buffer to pass the input data (or result data) stored in operation 1 as input data, and the result data of the execution can also be used as input data for operation 3. Therefore, the variables configured for the loading operation of operation 1 only involve the loading of computational data, and operation 2 does not configure the first variable and the second variable. In actual implementation, this embodiment of the disclosure can also default to configuring the first variable for computational data and weight data for both operation 1 and operation 2, configuring the second variable for operation 2, and then setting the value of these first variables / second variables to 0 by adding other constraint relationships to the first constraint condition.

[0145] It should also be noted that the above-mentioned first variable and second variable can be optionally set in the embodiments of this disclosure; for example, since no first type of operation is executed in time window 0 and time window 4, the data import operation cannot achieve time hiding. Therefore, x2 and x7 can be omitted in the embodiments of this disclosure to reduce the number of variables and simplify the constraints and objective function.

[0146] Of course, the above Figure 5 This is just a simple example. Other cases can be designed according to the execution needs of each type I operation in the actual AI model. The embodiments disclosed herein will not be described in detail.

[0147] Regarding the aforementioned first and second variables, this embodiment of the disclosure further sets first and second constraints. The first constraint includes a first constraint relationship and a second constraint relationship. The first constraint relationship can be set as: x2+x3+x4==1, indicating that the weight data for operation 3 will definitely be imported in one of the three time windows (0 to 2). The second constraint relationship can be set as: x5+x6+x7==1, indicating that the result data for operation 1 will definitely be exported in one of the three time windows (2 to 4). Of course, the first constraint can also include other constraints, such as: x1==1, indicating that the calculation data for operation 1 can only be imported in time window 0; or, for example: x8==1, indicating that the result data for operation 3 can only be exported in time window 4.

[0148] For configuring the second constraint, see here. Figure 6 , Figure 6 This is a schematic diagram illustrating the setting of a second constraint condition according to an exemplary embodiment; wherein... Figure 6 Each line represents the amount of memory used by operations within a time window of the time series. Figure 6 The first column represents the first estimated memory amount A. Figure 6 The second and third columns together represent the second estimated memory amount, where the second column represents the second sub-estimated memory amount B for the corresponding time window, and the third column represents the first sub-estimated memory amount C for the corresponding time window. Specifically, op1_out represents the memory amount occupied by the operation data associated with the data out operation corresponding to operation 1, which is also the memory amount occupied by the result data of operation 1; op3_weight represents the memory amount occupied by the operation data associated with the data in operation 3, which is also the memory amount occupied by the weight data in the input data of operation 3.

[0149] Here, the result data of the first type of operation will generate memory usage B in the first cache area because it is moved out in a subsequent time window; Figure 6 Each B shown represents the sum of the second target variables for the data outgoing operation within the corresponding time window, multiplied by op1_out. It can be understood that the second target variables corresponding to time window 2 are x5, x6, and x7. Regardless of which of these variables takes a value of 1, op1_out of operation 1 will occupy memory in time window 2. Therefore, determining each second target variable and summing them can express the temporal continuity of op1_out's memory usage in the first buffer. Similarly, the weight data for the first type of operation will be moved in during earlier time windows, thus generating memory usage C in the first buffer earlier. Figure 6Each C in the equation represents the sum of the first target variables of the data import operation within the corresponding time window, and the product of op3_weight. It can be understood that the first target variables corresponding to time window 2 are x2, x3, and x4. Regardless of which of the first variables x2, x3, and x4 takes a value of 1, op3_weight of operation 3 will have memory usage in time window 2. Therefore, determining each first target variable and summing them can express the temporal continuity of the memory usage of op3_weight in the first buffer.

[0150] against Figure 6 In this example, the condition that A+B+C in each row is less than the memory size of the first cache (e.g., 256KB) is used as the second constraint mentioned above.

[0151] See Figure 7 , Figure 7 This is a schematic diagram illustrating the principle of determining a first estimated duration and a second estimated duration according to an exemplary embodiment; wherein Figure 7 Each row in the table represents the execution time required for each operation within a time window of the time series. Figure 7 The first column represents the first estimated duration M. Figure 7 The second column represents the second estimated duration T; the third column represents the absolute value of the difference between the first estimated duration M and the second estimated duration T, which represents the delay |MT| of the corresponding time window.

[0152] Here, the first estimated duration M is the number of clock cycles required to execute the first type of operation within the corresponding time window, such as... Figure 7 In the first column, time window 0 has no first-type operation, so the first estimated duration is 0. The first estimated duration of time window 1 is 1200 of operation 1. The first estimated duration of time window 2 is 2300 of operation 2. The first estimated duration of time window 3 is 500 of operation 3. Time window 4 has no first-type operation, so the first estimated duration is 0.

[0153] For each time window, the data import and data export operations may differ, but the memory usage of the data for each data import and data export operation can be estimated and set. The second estimated duration T is the sum of the clock cycles of each type of second operation within the corresponding time window and the product of the corresponding first variable, representing the sum of the clock cycles of each type of second operation that may occur within that time window; where, Figure 7 In the example, the clock cycles for the calculation data of operation 1 are 1300, the clock cycles for the weight data of operation 2 are 1200, the clock cycles for the result data of operation 1 are 2400, and the clock cycles for the result data of operation 3 are 1600.

[0154] here, Figure 7 The third column can represent the instantaneous delay |MT| of each time window in the part where the timing cannot be hidden from each other. In this embodiment, the |MT| of each time window is summed to obtain the total execution delay of the memory operation sequence. The objective function is obtained with the goal of minimizing the total execution delay. In this way, the execution time of the asynchronous computing engine and the DMA engine can be hidden from each other as much as possible, so that the overall delay of the execution memory operation sequence is minimized.

[0155] Here, in this embodiment of the present disclosure, linear programming tools such as OR-Tools are used to perform linear analysis on the objective function in conjunction with the above-mentioned first and second constraints, so as to obtain the target values ​​of each first and second variable when the objective function obtains the minimum function value.

[0156] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0157] In the case where there are at least two non-adjacent target operations among multiple first-type operations in a memory operation sequence, and the result data of the target operation with a earlier time sequence is used as the input data of the target operation with a later time sequence, a candidate time window for the third-type operation associated with the target operation is determined based on at least one time window between the first time windows occupied by each of the at least two target operations in the time sequence.

[0158] A third variable is set based on the candidate time window of the third type of operation associated with the target operation; each third variable corresponds to a candidate time window of a third type of operation; different values ​​of the third variable are used to indicate whether the result data of the target operation that is earlier in the time sequence within the candidate time window of the third type of operation is resident in the first buffer.

[0159] Here, the result data of the target operation that occurs earlier in time among at least two target operations is used as the input data of the target operation that occurs later in time. This can be understood as the target operation that occurs earlier in time acting as the producer and the target operation that occurs later in time acting as the remote consumer. This relationship between target operations is often applied in AI models with residual connection network segments.

[0160] For example, see Figure 8 , Figure 8 This is a topological illustration of a memory operation sequence according to an exemplary embodiment. Figure 1 In AI models with residual connection network segments, some network nodes' memory operations exhibit a producer-consumer relationship. Specifically, the producer outputs an intermediate activation into the first buffer, and subsequent consumer operations directly use this activation, eliminating the need for data fetching and reloading. For example... Figure 8As shown, operations 1 to 7 are the first type of operations in the memory operation sequence. After the output of operation 1 is used by the adjacent consumer operation 2, it also needs to be used by the remote consumer operation 7. Here, operation 1 and operation 7 can be used as two target operations that are not adjacent in time in this embodiment of the present disclosure.

[0161] For target operations that are not sequentially adjacent, after operation 2 is executed with the result data of operation 1 as input data, the result data needs to be released. However, moving the result data out of the first buffer and then moving it back in before operation 7 is executed will prolong the execution time and generate unnecessary power consumption. Therefore, it is possible to choose to keep the result data in the first buffer until it is released after operation 7 uses it. However, if the first buffer is short of memory, then the retention of the result data will put great memory pressure on the first buffer during this process.

[0162] Based on this, embodiments of this disclosure propose a third type of operation, determine candidate time windows for the third type of operation, and set a corresponding third variable for each candidate time window of the third type of operation associated with the target operation; wherein, the third type of operation indicates that the result data of the producer operation is stored in the first buffer, the candidate time windows of the third type of operation are the time windows between the first time window of the target operation that is earlier in time and the first time window of the target operation that is later in time, and the value of the third variable can be a first value (such as 1) or a third value (such as 0). Thus, if the value is 1, the third variable can indicate that the result data of the target operation that is earlier in time is stored in the first buffer within the candidate time window of the third type of operation, and if the value is 0, the third variable can indicate that the result data of the target operation that is earlier in time is not stored in the first buffer within the candidate time window of the third type of operation.

[0163] It should be noted that the same third variable is set in each candidate time window of the third type of operation associated with the same target operation. The value of the same third variable is consistent. Either they are all the first value, such as all of them being 1, which means that the result data always resides in the first buffer until the target operation in the later time sequence is executed; or the value of the same third variable is either all of them being the third value, such as all of them being 0, which means that the result data does not reside in the first buffer.

[0164] This disclosure defines whether the result data of non-adjacent target operations resides in the first buffer by setting a third variable. This allows the residency problem to be analyzed as a variable value, and through subsequent linear analysis, a more optimized solution can be selected between the solution of residing the result data and the solution of moving the result data out and then moving it back in.

[0165] In some embodiments, determining the first constraint condition based on the constraint relationship between decision variables corresponding to at least one candidate time window for each second type of operation includes:

[0166] The first constraint condition is determined based on the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation; wherein, the second target sub-operation is the second sub-operation associated with the target operation that comes first in time; and the first target sub-operation is the first sub-operation associated with the target operation that comes later in time.

[0167] Here, the second target sub-operation is the second sub-operation associated with the target operation that comes first in time, and the first target sub-operation is the first sub-operation associated with the target operation that comes later in time. Continuing with the example of the second scenario above, the second target sub-operation is a data out operation for the result data of the target operation that comes first in time, and the first target sub-operation is a data in operation for the result data.

[0168] Thus, embodiments of this disclosure can determine candidate time windows for data fetching operations associated with a target operation that occurs earlier in the time sequence, and candidate time windows for data fetching operations associated with a target operation that occurs later in the time sequence.

[0169] In some examples, embodiments of this disclosure may use the first time window between the first time window of the target operation that occurs earlier and the first time window of the target operation that occurs later as each candidate time window for the aforementioned data import operation and each candidate time window for the aforementioned data export operation; and determine the constraint relationship for the values ​​of the relevant variables based on the producer-consumer relationship, the logical relationship between the export result data and the import and resident result data, the first variable for each candidate time window of the aforementioned data import operation and the second variable for each candidate time window of the aforementioned data export operation.

[0170] For example, combining Figure 8Operation 2 is the consumer immediately following Operation 1, so Operation 2 needs to use the result data of Operation 1 as calculation data. Therefore, the aforementioned data fetching and data fetching operations should not occur in time window 1. Thus, in this embodiment, the values ​​of the first variable and the second variable corresponding to this time window can both be set to 0 as a constraint. In addition, since the result data can only be used after it is fetched into the cache, in this embodiment, the product of the first variable for the data fetching operation and the timing value of the corresponding candidate time window can be greater than the product of the second variable for the data fetching operation and the timing value of the corresponding candidate window as a constraint. Furthermore, since the third type of operation and the second type of operation will not both be executed between non-adjacent target operations, a constraint can be determined based on this logic. Thus, by limiting the above constraint relationships, the first constraint condition can be obtained, thereby restricting the values ​​of each first variable and the second variable through the first constraint condition.

[0171] In other examples, embodiments of this disclosure may also select candidate time windows in each time window between the first time window of the target operation that occurs earlier and the first time window of the target operation that occurs later, and determine the corresponding first and second variables, based on the producer-consumer relationship; for example, the first time window in these time windows may not be used as a candidate time window for the aforementioned data import operation; and the last time window in these time windows may not be used as a candidate time window for the aforementioned data export operation; thus, embodiments of this disclosure may reduce the setting of decision variables and corresponding constraints, thereby simplifying the obtained linear programming model.

[0172] Here, in this embodiment of the disclosure, the first and second variables of the second type of operation associated with the above-mentioned non-adjacent target operation can be constrained based on the third variable and the first constraint condition, so that a more optimized scheme can be selected from the scheme of retaining the result data and the scheme of re-importing the result data from the output data through subsequent linear analysis.

[0173] In some embodiments, determining the first constraint condition based on the constraint relationship between the second variables corresponding to each candidate time window of the second target sub-operation, the first variables corresponding to each candidate time window of the first target sub-operation, and the third variables of the third type of operation associated with the target operation includes:

[0174] Determine the third sum value between the second variables corresponding to each candidate time window of the second target sub-operation, and the fourth sum value between the first variables corresponding to each candidate time window of the first target sub-operation;

[0175] The third sum of the second variables corresponding to each candidate time window of the second target sub-operation is set to be equal to the fourth sum of the first variables corresponding to each candidate time window of the first target sub-operation, thus obtaining the third constraint relationship. The third constraint relationship is used to restrict that if the second target sub-operation is executed in any candidate time window of at least one candidate time window of the first target sub-operation, the second engine needs to execute the first target sub-operation in any candidate time window of at least one candidate time window of the first target sub-operation.

[0176] The sum of the second and third variables corresponding to each candidate time window of the second target sub-operation is set to equal the first value, thus obtaining the fourth constraint relationship. The fourth constraint relationship is used to restrict that the third type of operation associated with the target operation with a preceding time sequence does not need to be executed when the second target sub-operation is executed in any candidate time window of at least one corresponding candidate time window, or the second target sub-operation does not need to be executed when the third type of operation associated with the target operation with a preceding time sequence is executed.

[0177] The first parameter is determined based on the product between the timing value corresponding to each candidate time window of the first target sub-operation and the first variable corresponding to the candidate time window, and the product between the timing value corresponding to each candidate time window of the second target sub-operation and the second variable corresponding to the candidate time window; the first parameter is used to indicate the execution time difference between the first target sub-operation and the second target sub-operation when the second engine executes the first target sub-operation and the second target sub-operation.

[0178] The fifth constraint relationship is obtained by setting the sum of the first and second parameters to be greater than or equal to the second value; the second parameter is the product of the second value and the third variable; the fifth constraint relationship is used to restrict the execution time difference between the first target sub-operation and the second target sub-operation from being greater than the second value.

[0179] The first constraint condition is determined based on the third, fourth, and fifth constraint relationships.

[0180] Here, for the target operation that occurs later in time, if the second target sub-operation (such as the aforementioned data fetching operation for the result data) is executed in any time window between the first time window of the target operation that occurs earlier in time and the first time window of the target operation that occurs later in time, then the first target sub-operation (the aforementioned data fetching operation) must also be executed in one of its corresponding candidate time windows; only in this way can the result data of the target operation that occurs earlier in time be used by the target operation that occurs later in time. Therefore, in this embodiment of the present disclosure, the third sum value between the second variables corresponding to each candidate time window of the aforementioned fetching operation is set to be equal to the fourth sum value between the first variables corresponding to each candidate time window of the aforementioned data fetching operation, as a third constraint relationship, used to constrain the value relationship between the second variables and the first variables.

[0181] Continuing from the example of the second scenario above, see [link to example]. Figure 9 , Figure 9 This is a schematic diagram illustrating the setting of a third variable, a first variable of a first target sub-operation, and a second variable of a second target sub-operation, according to an exemplary embodiment; wherein, in conjunction with the above... Figure 8 and Figure 9 In the example, operations 1 to 7 correspond to each of the first type of operations, and time windows 0 to 6 correspond to the first time windows of operations 1 to 7, respectively. Here, operation 1 is a target operation with a earlier time sequence, and operation 2 is a target operation with a later time window. In this disclosure, the second variable (out0_store2 and out0_store3) of the aforementioned data transfer operation is set in time windows 2 and 3, and the second variable (out0_load3, out0_load4 and out0_load5) of the aforementioned data transfer operation is set in time windows 3, 4 and 5. The third variable out0_reside is set in time windows 2 to 5.

[0182] In out0_store(n), n is the time series value of the candidate time window corresponding to the variable. For example, out0_store2 is the second variable related to the aforementioned data loading operation for the result data corresponding to time window 2. Similarly, in out0_load(m), m is also the time series value of the candidate time window corresponding to the variable. For example, out0_store4 is the first variable related to the aforementioned data loading operation for the result data corresponding to time window 4. The interpretation of other variables is similar, and will not be elaborated here.

[0183] according to Figure 8 and Figure 9In this embodiment of the present disclosure, out0_store2+out0_store3 is used as the third sum, out0_load3+out0_load4+out0_load5 is used as the fourth sum, and out0_store2+out0_store3==out0_load3+out0_load4+out0_load5 is set as the third constraint relationship.

[0184] Furthermore, for the aforementioned non-adjacent target operations, since the target parameter with a later timing sequence must use the result data, if a second target sub-operation (such as the aforementioned data fetching operation for the result data) is executed within the time window between the first time window of the target operation with a earlier timing sequence and the first time window of the target operation with a later timing sequence, then the third type of operation does not need to be executed; if the third type of operation is executed, then the aforementioned data fetching operation does not need to be executed. That is, the result data either resides in the first buffer or is fetched from the first buffer by the second engine and then re-fetched. Thus, in this embodiment of the present disclosure, the sum of each second variable and a third variable corresponding to each candidate time window of the aforementioned fetching operation is set to a first value, which serves as a third constraint relationship to constrain the value relationship between the second variable and the first variable. Here, in conjunction with the above description of the present disclosure, the first value can be 1.

[0185] according to Figure 8 and Figure 9 In this embodiment of the disclosure, out0_reside+out0_store2+out0_store3==1 is used as the fourth constraint relationship mentioned above.

[0186] Meanwhile, since the data loading and unloading operations occur in two adjacent time windows, the memory resource shortage in the first cache cannot be alleviated. Considering the memory constraints of the first cache, this embodiment of the present disclosure may further specify that the time window for the aforementioned data loading operation of the result data is at least a second preset number of time windows later than the time window after the aforementioned data loading operation. The value of the second preset number is a second value, which is greater than or equal to 2.

[0187] Thus, in this embodiment, the product of the timing value corresponding to each candidate time window of the first target sub-operation (such as the aforementioned data import operation for the result data) and the first variable corresponding to the candidate time window is subtracted from the product of the timing value corresponding to each candidate time window of the second target sub-operation (such as the aforementioned data export operation) and the second variable corresponding to the candidate time window to obtain the first parameter. In this way, the first parameter can reflect the number of time windows between the second engine executing the aforementioned data export operation and the aforementioned data import operation, that is, reflect the execution time difference. Since only one of the third type of operation and the aforementioned data export operation occurs, this embodiment uses the product of the second value and the third variable as the second parameter, and sets the fifth sum of the first parameter and the second parameter to be greater than or equal to the second value to obtain the fifth constraint relationship. This can reflect that the aforementioned data import operation and the aforementioned data export operation for the result data are at least two time windows apart, or that the third type of operation, i.e., the residence of the result data, occurs within these time windows.

[0188] according to Figure 8 and Figure 9 In this embodiment of the disclosure, (5*out0_load5+4*out0_load4+3*out0_load3)-(2*out0_store2+3*out0_store3) is used as the first parameter; 2*out0_reside is used as the second parameter; and (5*out0_load5+4*out0_load4+3*out0_load3)-(2*out0_store2+3*out0_store3)+2*out0_reside>=2 is used as the fifth constraint relationship.

[0189] Thus, the embodiments of this disclosure can determine the first constraint condition through the third, fourth, and fifth constraint relationships, and obtain the target values ​​of the first and second variables through the linear programming model obtained by the first constraint condition and the objective function. Correspondingly, the value of the third variable can also be determined based on the value of the first variable of the first objective operation and the value of the second variable of the second objective operation. In this way, a scheme with a smaller total execution latency of the memory operation sequence can be obtained from the scheme of residing the result data and the scheme of moving out the result data and then moving it back in.

[0190] It should be noted that if the value of the third variable is the first value (such as 1), then the first target sub-operation and the second target sub-operation of the target operation will not occur, and the execution time of the second engine will not be generated. Therefore, the third type of operation itself does not affect the execution delay. Therefore, the embodiments of this disclosure still define the objective function through the first estimated time and the second estimated time, and analyze the objective function in combination with the first constraint condition determined by the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship, so as to obtain the values ​​of the first variable and the second variable.

[0191] In some embodiments, determining the second constraint based on the first estimated memory amount and the second estimated memory amount within each time window, as well as the memory amount of the first cache, includes:

[0192] Based on the product of the memory occupied by the result data of the target operation and the third variable, determine the third estimated memory occupied by the result data in the first cache within each candidate time window of the third type of operation associated with the target operation that comes first in time.

[0193] The second constraint is determined based on the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount within each time window, as well as the memory amount of the first cache.

[0194] The second constraint also limits the sum of the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount in each time window to be less than the memory amount of the first cache.

[0195] Here, because the third type of operation represents the residence of the result data of the target operation that precedes the time sequence in the first cache, the resident result data has a memory amount and will generate memory occupation in the first cache; thus, in this embodiment of the disclosure, the product of the memory occupied by the result data of the target operation and the third variable will be used as the third estimated memory amount occupied by the result data of the third type of operation associated with the target operation that precedes the time sequence in the first cache within each candidate time window.

[0196] For example, according to Figure 8 and Figure 9 For example, we set `op1_out * out0_size` as the third estimated memory amount `D` corresponding to each candidate time window of the third type of operation; here, `op1_out` is the memory amount occupied by the result data of operation 1, and `out0_size` is the third variable. Combined with... Figure 6For example, in this embodiment of the disclosure, the sum of the first estimated memory amount A, the second estimated memory amount (the sum of the first sub-estimated memory amount C and the second sub-estimated memory amount B) and the third estimated memory amount D in each time window can be less than the memory amount of the first cache (e.g., 256KB), as a second constraint condition; wherein, if no corresponding third variable is set in the time window, the value of the third estimated memory amount is 0.

[0197] Here, this embodiment of the disclosure improves the second constraint in the scenario by considering the continuous occupancy of the result data of the target operation in the first buffer due to the third type of operation, thereby enhancing the rationality of the obtained linear programming model and making the determined decision variables and the third variable more accurate.

[0198] For the aforementioned non-adjacent at least two target operations, in actual implementation, there may be cases where the result data of the target operation that occurs earlier can serve as input data for the at least two target operations that occur later. For example, see... Figure 10 , Figure 10 This is a topological illustration of a memory operation sequence according to an exemplary embodiment. Figure 2 ;in, Figure 10 Operations 1 through 14 are all first-class operations in the memory operation sequence. The operation number indicates the execution order of the operations, and the arrows represent producer-consumer relationships, meaning that the result data of the operation pointed to by the arrow can be used as the input data of the operation pointed to by the arrow. Here, the result data of operation 1 can be used as the input data of operations 2, 6, and 10. Therefore, operations 1 and 6, and operations 1 and 10 can all constitute the aforementioned non-adjacent target operations. Similarly, the result data of operation 5 can be used as the input data of operation 14. Therefore, operations 5 and 14 can constitute the aforementioned non-adjacent target operations. The result data of operation 9 can be used as the input data of operation 14. Therefore, operations 9 and 14 can constitute the aforementioned non-adjacent target operations.

[0199] Here, if the target operation with a preceding time sequence has a target operation with a subsequent time sequence, such as operations 5 and 14 mentioned above, then this embodiment of the disclosure can set corresponding third variables, corresponding first constraints, and second constraints for operations 5 and 14. This allows for the minimization of the objective function to obtain the second variable for the data transfer operation of operation 5, the first variable for the data transfer operation of operation 14, and the results of the corresponding third variables. This enables a more optimized choice between the data retention scheme for the result data of operation 5 and the scheme of transferring data out first and then transferring it in. Similarly, operations 9 and 14 can be set as described above, which will not be repeated here.

[0200] Furthermore, if a target operation with a preceding time sequence has at least two target operations with a following time sequence (e.g., operation 1 and operation 6, operation 10), then this embodiment of the disclosure can set two variable groups. One variable group includes a second variable for the data transfer operation of operation 1, a first variable for the data transfer operation of operation 6, and a third variable within the time window between the time window corresponding to operation 2 and the time window corresponding to operation 5. The other variable group includes a second variable for the data transfer operation of operation 1, a first variable for the data transfer operation of operation 10, and a third variable within the time window between the time window corresponding to operation 7 and the time window corresponding to operation 9. Here, the variables within each variable group are subject to a first constraint condition formed by the aforementioned third, fourth, and fifth constraint relationships, and a second constraint condition determined by the aforementioned third, second, and first estimated variables.

[0201] It should be noted that, in actual implementation, the embodiments of this disclosure can also set constraints between the above-mentioned different variable groups. For example, the sum of the sums of the first variables in each variable group is the first value, and the first value is 1. This limits that the result data of the producer operation is only fetched out once between the target operation (producer) in the earlier time sequence and the target operation (farthest consumer) in the last time sequence. For another example, if the sum of the second variables in the previous variable group is 1, then before the first consumer uses it, a data fetching operation of the result data has occurred. In order to reduce memory usage, the third variables in the subsequent variable groups are all set to 0. In this way, when the subsequent consumer uses it, the result data can be fetched back in using only the second engine.

[0202] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0203] Based on the execution timing information of the memory operation sequence, determine the first type of operation and / or the second type of operation to be executed in each time window of the time sequence;

[0204] Within each time window, allocate memory resources of the first cache area for the first type of operation and / or the second type of operation to be executed within the time window;

[0205] After performing a first type of operation and / or a second type of operation based on the allocated memory resources, at least a portion of the memory resources allocated to the first type of operation and / or the second type of operation are released.

[0206] Here, after obtaining the execution timing information, it is possible to determine the first type of operation and / or the second type of operation that can be performed within each time window of the time series. Thus, in this embodiment of the present disclosure, when it is determined that a first type of operation needs to be performed within each time window, memory resources are allocated for the first type of operation, allowing the first engine to execute the first type of operation based on the memory resources; and when a second type of operation needs to be performed, memory resources are allocated for the second type of operation, allowing the second engine to execute the second type of operation based on the memory resources.

[0207] It should be noted that the amount of memory resources allocated for the first type of operation is the same as the amount of memory occupied in the first cache when executing the first type of operation; the amount of memory resources allocated for the second type of operation is the same as the amount of memory occupied in the first cache for the operation data of the second type of operation.

[0208] In order to ensure that the first buffer has sufficient resources for subsequent second and first operations when performing the first and / or second operations, the embodiments of this disclosure may release the memory resources allocated to the currently executed first and / or second operations in the first buffer, except for the data that needs to reside in the first buffer for sharing, and then execute the first operation in the next time sequence after the release is completed.

[0209] Continuing from the example of the second scenario above, see [link to example]. Figure 11 , Figure 11 This is a schematic diagram illustrating a memory management principle according to an exemplary embodiment; here, Figure 11 yes Figure 5 The diagram shows one possible value for each of the first and second variables; where the first variables x1 and x3 are 1, and the other first variables are 0; the second variables x6 and x8 are 1, and the other second variables are 0; here, a first variable value of 1 indicates that the DMA engine needs to move the calculated data or weight data of operation 1 into the first buffer, and a second variable value of 1 indicates that the DMA engine needs to move the result data out of the first buffer. Figure 11 The diagram shows the memory resource allocation, memory usage, and resource release of the first buffer during time windows 0 to 4, where the first variables x1 and x3 are 1 and the other first variables are 0, and the second variables x6 and x8 are 1 and the other second variables are 0.

[0210] It should be noted that if memory resources are not required to reside in memory, they should be released within the time window after use. For example, in time window 1, the data calculated for operation 1 ( Figure 11The first buffer (op1 input) is released immediately after use; if the memory resource needs to reside, it will not be released in the current time window. For example, the result data of operation 2 (op2 output) is not released in the second buffer. It should also be noted that the first buffer can also allocate temporary buffer resources (op3 buffer) for some special operations. For example, in time window 3, temporary buffer resources are allocated for operation 3, and they are released in time window 3 after operation 3 is calculated.

[0211] In this embodiment of the disclosure, by adjusting the allocation and release of resources in the first buffer, the safe and effective execution of the memory operation sequence can be ensured.

[0212] In this embodiment of the disclosure, due to the limitations of the second constraint condition mentioned above, the obtained execution timing information of memory operations can determine the amount of memory resources allocated to the first type of operation and / or the second type of operation in each time window, and ensure that the amount of memory resources allocated does not exceed the total memory amount of the first cache in that time window. However, in actual implementation, see... Figure 12 , Figure 12 This is a memory resource distribution of the first cache area according to an exemplary embodiment. Figure 1 In this context, the shaded areas 3000 and 1000 represent the memory allocated for operational data, while the other areas are free memory. Figure 12 The storage location corresponding to the operation data with a memory size of 1000 is shown to be separated from the memory resource operation data with a memory size of 3000. That is, the operation data with a memory size of 1000 is treated as memory fragment. Therefore, if the memory operation to be executed within the time window requires the allocation of memory resources with a memory size of 4000, the total amount of free memory in the first cache (3000 and 1000 in the blank area) is sufficient, but there is still a phenomenon that the memory resources cannot be allocated successfully.

[0213] Therefore, in order to solve the problem of memory resource allocation failure caused by the above-mentioned memory fragmentation, it is necessary to reasonably eliminate or reduce the generation of memory fragmentation.

[0214] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0215] If memory resource allocation for a first-type operation and / or a second-type operation to be executed within a third time window in a time series fails, a fourth time window is inserted into the time series; the fourth time window is the time window preceding the third time window.

[0216] Determine the first storage location of the first target data in the first cache; wherein, the first target data is the operation data stored in the first cache, and the first storage location of the first target data in the first cache is not adjacent to the second storage location of the operation data other than the first target data in the first cache;

[0217] Within the fourth time window, the first target data is moved from the first storage location of the first cache to the third storage location of the first cache; the third storage location is a free storage location adjacent to the second storage location among multiple free storage locations in the first cache;

[0218] After the first target data has been moved, memory resources are reallocated for the first type of operation and / or the second type of operation to be executed in the third time window.

[0219] Here, for a third time window in the time series containing first-type operations and / or second-type operations, if there is no whole block of free memory in the first buffer, then to allocate memory resources for the first-type operations and / or second-type operations, it is necessary to first eliminate the memory fragments existing in the first buffer. At this time, a time window is inserted between the third time window and the adjacent previous time window as the fourth time window, and the storage location of the operation data stored in the first buffer is adjusted within the fourth time window.

[0220] First, the first target data existing in the first cache is determined, and then within the inserted third time window, the first target data is moved from the first storage location to the third storage location. Here, the first target data is the operation data stored in the first cache, and the first storage location of the first target data in the first cache is not adjacent to the second storage location of the operation data other than the first target data in the first cache; that is, the first target data is considered a memory fragment in the first cache (e.g., ...). Figure 12 (The operation data with a memory size of 1000); because the third storage location is a free storage location adjacent to the second storage location among multiple free storage locations in the first cache, the embodiments of this disclosure concentrate memory fragments into non-memory fragments stored in the first cache. In this way, after shifting at least one target operation in the first cache, memory fragments in the first cache can be eliminated, so that the memory resources allocated in the first cache (i.e., the occupied operation data) are concentrated at the head of the first cache or at the tail of the first cache.

[0221] Thus, after the first target data is moved within the fourth time window, memory resources can be reallocated for the first type of operation and / or the second type of operation to be executed within the third time window.

[0222] For example, see Figure 13 , Figure 13This is a memory resource distribution of the first cache area according to an exemplary embodiment. Figure 2 Here, by Figure 12 Moving data from memory location 1000 to memory location 3000 will yield the desired result. Figure 13 The memory resource distribution shown indicates that a memory resource of size 4000 can be allocated in the first cache for use in the next time window.

[0223] Thus, by adjusting the storage location of the first target data in the first cache area, the embodiments of this disclosure can improve the success rate of memory resource allocation and further improve the efficiency of memory operation sequence execution.

[0224] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0225] Based on the first type of operation and / or the second type of operation to be executed in each time window of the time series, determine the memory usage and remaining memory of the first buffer in each time window;

[0226] The target remaining memory amount is determined based on the remaining memory amount corresponding to multiple time windows in the time series; the target remaining memory amount is the minimum value among the remaining memory amounts corresponding to multiple time windows.

[0227] A resident storage area is set up in the first cache area; the amount of memory in the resident storage area is determined by the target remaining memory amount, and the storage location of the resident storage area is located at the end of the first cache area;

[0228] The second target data is determined from the operation data associated with each of the first type of operations to be executed in the memory operation sequence; the second target data is the weight data of the first type of operations.

[0229] Based on the amount of memory required by the second target data, at least one third target data is determined from at least one second target data; the sum of the memory amounts of at least one third target data is less than or equal to the memory amount of the resident storage area, and the memory amount of each third target data is less than the memory amount of the second target data other than the third target data in at least one second target data.

[0230] Store at least one third target data in the resident storage area.

[0231] Here, based on the execution timing information of the memory operation sequence, after determining the first type of operation and / or the second type of operation to be executed in each time window, the memory usage of the first buffer in each time window can be estimated; thus, the remaining memory in each time window can be determined based on the difference between the memory amount in the first buffer in each time window and the aforementioned memory usage.

[0232] See Figure 14 and Figure 15 , Figure 14 This is a schematic diagram of a resident storage area according to an exemplary embodiment. Figure 1 , Figure 15 This is a schematic diagram of a resident storage area according to an exemplary embodiment. Figure 2 ;in, Figure 14 The length of each line represents the amount of memory in the first cache. Each line indicates the amount of memory occupied by the first engine executing the first type of operation (A'), the amount of memory occupied by the second engine executing the first sub-operation (B'), the amount of memory occupied by the second engine executing the second sub-operation (C'), and the remaining memory (E, blank area) for each time window in each time window. Figure 14 It is known that the remaining memory amount E corresponding to multiple time windows in the time series can be used to determine the minimum remaining memory amount, which is taken as the target remaining memory amount. The target remaining memory amount represents the capacity of the space area in the first cache that will not be used when the memory operation sequence is executed according to the execution timing information. Thus, the embodiments of this disclosure can set the target remaining memory amount as follows: Figure 14 and Figure 15 The resident storage area X shown is used to store some weighted data with a small amount of memory in the resident storage area X for use by subsequent Type I operations.

[0233] It should be noted that, in this embodiment of the disclosure, the resident storage area can be set at the end of the first cache area, so that the resident storage area will not form memory fragmentation. In actual implementation, resources can also be allocated for the operation of each time window sequentially from the head to the tail of the first cache area. It should also be noted that after the operation of each time window is completed, the non-resident memory resources need to be released to reduce the generation of memory fragmentation.

[0234] In the case of multiple identical first-type operations in a memory operation sequence, although the data processed by the first engine when executing multiple first-type operations may be different, the weight data of the first-type operations used by the first engine when executing the first-type operations is fixed. To reduce the number of times the second engine performs second-type operations on the weight data of the first-type operations, embodiments of this disclosure can fully utilize the space of the resident storage area to store the weight data of the first-type operations in the resident storage area. This eliminates the need for the second engine to frequently perform second-type operations on the weight data of multiple identical first-type operations, thereby effectively reducing both overall execution latency and system power consumption.

[0235] Considering the limited memory space in the resident storage area, the weight data required by each type I operation in the memory operation sequence can be filtered to determine the weight data that can be stored in the resident storage area.

[0236] Specifically, in this embodiment of the disclosure, the second target data is determined from the operation data associated with each of the first type of operations to be executed in the memory operation sequence; the second target data is the aforementioned weight data. Of course, in the first scenario described above, the weight data may also be model parameters, gradient data, etc. In addition, the second target data may also be operation data that can be shared by at least one first type of operation, such as label data in a supervised learning model.

[0237] After determining all the second target data, this embodiment of the disclosure can sort the second target data from smallest to largest according to the amount of memory required, obtaining an ascending sequence. For the ascending sequence, the amount of memory occupied by each second target data in the sequence is added to the amount of memory occupied by all the preceding second target data in the sequence. If the sum of the sums of the preceding second target data in the sequence is less than or equal to the amount of memory in the resident storage area, and the sum of the sums of the following second target data is greater than the amount of memory in the resident storage area, then the preceding second target data and all the second target data before it are considered as third target data. Thus, because the sum of the memory amounts of all third target data is less than or equal to the amount of memory in the resident storage area, and because of the ascending sequence, the amount of memory occupied by each third target data will be less than the amount of memory occupied by all the second target data except for the third target data, all third target data can be resident in the aforementioned resident storage area.

[0238] In this embodiment of the disclosure, the aforementioned determined third target data can be pre-loaded into the first cache for storage. Here, in this embodiment of the disclosure, the third target data can be stored in the resident cache in descending order of memory size. Furthermore, this third target data can be released from the first cache after the memory operation sequence is completed, or immediately released when each third target data no longer needs to be used by subsequent first-type operation data.

[0239] Understandably, by setting up a resident cache to store the second target data with a small memory footprint, this disclosure can reduce memory fragmentation caused by the failure to release this data in a timely manner. Furthermore, this disclosure sorts the second target data from smallest to largest according to the amount of memory required, and determines the second target data that ranks first as the third target data. This allows more weight data required for the execution of the first type of operation to be stored in the resident cache, thereby reducing the latency caused by moving this data in and out, and reducing unnecessary power consumption generated by the operation of moving this data in and out, compared to storing the second target data with a larger memory footprint.

[0240] In some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0241] If there is at least one identical first-type operation in the memory operation sequence of the first kernel and the memory operation sequence of the second kernel, at least one identical fourth target data is determined from the first operation data and the second operation data based on the at least one identical first-type operation; wherein, the first operation data is the operation data of the first sub-operation associated with at least one identical first-type operation in the first kernel; and the second operation data is the operation data of the first sub-operation associated with at least one identical first-type operation in the second kernel.

[0242] Based on the second time window of the first sub-operation associated with the fourth target data in the first kernel and the second time window of the first sub-operation associated with the fourth target data in the second kernel, a first target time window and a second target time window are determined; the timing of the first target time window is earlier than the second time window of the first kernel and any second time window of the second kernel; the timing of the second target time window is later than the second time window of the first kernel and any second time window of the second kernel.

[0243] Within the first target time window, the fourth target data is stored in the second cache; wherein the fourth target data stored in the second cache can be shared by the first kernel and the second kernel.

[0244] And within the second target time window, release the memory resources occupied by the fourth target data in the second cache area.

[0245] Here, the memory operation sequence proposed in this embodiment can be executed by a first core and a second core in the processor that can be executed in parallel; wherein, the first core and the second core can execute the same memory operation sequence in parallel, or they can execute different memory sequences in parallel, and this embodiment does not limit this.

[0246] In actual implementation, if the memory operation sequences executed by the first kernel and the second kernel each contain the same first type of operation, it can be determined whether the same first type of operation requires the same input data. In this case, the operation data of the first sub-operation associated with the first type of operation in the first kernel (such as the data loading operation in the example above) is determined as the first operation data, and the operation data of the first sub-operation associated with the same first type of operation in the second kernel (such as the data loading operation) is determined as the second operation data. Then, one or more identical fourth target data are determined from the first operation data and the second operation data.

[0247] Here, the fourth target data can be the weight data required for the execution of each network node in the AI ​​model or some shared input data such as label data. The weight data may include bias data.

[0248] Since these fourth target data are needed by both the first and second kernels, but if they both need to read the fourth target data from the memory area, the access speed is slow and the power consumption is high; therefore, in this embodiment of the present disclosure, another on-chip cache with a smaller memory size and a higher access speed than the memory area can be set up, such as a second cache (L2 cache). The data stored in the L2 cache can be accessed and moved by both the first and second kernels; in this way, in this embodiment of the present disclosure, the fourth target data can be stored in the second cache as needed, so that the first and second kernels can directly access the fourth target data from the second cache, improving the access speed and reducing unnecessary power consumption.

[0249] Specifically, in this embodiment, the second time window of the first sub-operation associated with the fourth target data in the first kernel and the second time window of the first sub-operation associated with the fourth target data in the second kernel are first determined. In terms of time sequence, a time window whose time sequence is earlier than the second time window of the first kernel and the second time window of the second kernel are taken as the first target time window. In actual implementation, the fourth target data is stored in the second cache within the first target time window.

[0250] In this embodiment of the present disclosure, the method for determining the first target time window may be: comparing the timing of the second time window of the first kernel and the second time window of the second kernel determined above, and taking the time window before the second time window with the earlier timing as the first target time window.

[0251] Furthermore, to reduce the memory resource consumption of the second cache, this embodiment of the present disclosure can release the fourth target data from the second cache after the execution of the first sub-operation corresponding to the first kernel and the second kernel, that is, after accessing the fourth target data. Specifically, this embodiment of the present disclosure can, in terms of time sequence, use a time window whose timing is later than the second time window of the first kernel and the second time window of the second kernel determined above as the second target time window; and in actual implementation, release the fourth target data from the second cache within the second target time window.

[0252] In this embodiment of the present disclosure, the method for determining the second target time window may be: comparing the timing of the second time window of the first kernel and the second time window of the second kernel determined above, and taking the later time window of the second time window with the later timing as the first target time window.

[0253] For example, see Figure 16 , Figure 16This is a timing diagram illustrating the storage of fourth target data into a first kernel and a second kernel according to an exemplary embodiment; wherein the first kernel core1 and the second kernel core2 both have the same first type of operation: operation 3; each time window of the first kernel core1 and the second kernel core2 corresponds to each time window of the time sequence; wherein operation 3 in both kernels requires the same weight data (weight3); through the execution timing information determined by core1 and the execution timing information determined by core2, it is determined that core1 can move weight3 into time window 1, and core1 can move weight3 into time window 2; thus, in this embodiment of the present disclosure, time window 0, whose timing is earlier than time window 1 and time window 2, is taken as the first target time window, and weight3 is moved from the memory area (DDR) to the second cache area (L2 cache) for storage in time window 0 using the DMA engine; subsequently, in time window 2, the first kernel uses the DMA engine to move weight3 from the second cache area to the first cache area; in time window 3, the second kernel uses the DMA engine to move weight3 from the second cache area to the first cache area. In addition, in this embodiment of the present disclosure, time window 3, which is later than time window 1 and time window 2, is used as the second target time window, and weight 3 is released from the second buffer in time window 3.

[0254] The embodiments disclosed herein improve the speed at which parallel kernels acquire the fourth target data and reduce the power consumption of the system executing memory operation sequences by setting up a second cache for parallel kernels and moving the same fourth target data required by all parallel kernels from the memory area in advance.

[0255] In this embodiment of the disclosure, for some memory operation sequences, there are cases where the execution time required for multiple consecutive first-type operations is very short, but the execution time required for the associated second-type operations is very long. In this case, the first time window corresponding to these first-type operations cannot be hidden from the second-type operations that occur within the time window, that is, the delay generated within the corresponding time window is large.

[0256] In view of this, in some embodiments, the timing arrangement method proposed in this disclosure further includes:

[0257] Based on the second estimated duration of each time window in the time series, at least two temporally adjacent third target time windows are determined from multiple time windows in the time series; the second estimated duration of each third target time window is less than a preset duration threshold.

[0258] Based on the earliest fourth target time window among at least two temporally adjacent third target time windows, multiple third target variables are determined from the second variables corresponding to the at least two temporally adjacent third target time windows; the third target variables are the second variables corresponding to the third target time windows other than the fourth target time window among the at least two temporally adjacent third target time windows.

[0259] Based on the latest fifth target time window among at least two temporally adjacent third target time windows, multiple fourth target variables are determined from the first variables corresponding to the at least two temporally adjacent third target time windows; the fourth target variables are the first variables corresponding to the third target time windows other than the fifth target time window among the at least two temporally adjacent third target time windows.

[0260] Multiple third objective variables and multiple fourth objective variables are set to have values ​​equal to the third value, resulting in the third constraint condition; wherein, the third value of the third objective variable is used to indicate that the second sub-operation is not executed within the candidate time window corresponding to the third objective variable; the third value of the fourth objective variable is used to indicate that the first sub-operation is not executed within the candidate time window corresponding to the fourth objective variable;

[0261] The above-mentioned determination of the target values ​​of multiple decision variables based on the objective function and the first and second constraints includes:

[0262] The objective value is determined based on the objective function and the first, second, and third constraints.

[0263] In this embodiment of the disclosure, in order to determine the case where the execution time required for the second type of operation corresponding to at least two consecutive time windows is very long, the disclosure determines at least two temporally adjacent third target time windows in the multiple time windows of the time series based on the second estimated duration of each time window in the time series. Here, the second estimated duration of each third target time window is less than a preset duration threshold. The preset duration threshold is used to reflect that the total time required to execute the second type of operation in the corresponding third time window is much greater than the time required to execute the first type of operation in the time window, thereby seriously increasing the total execution latency of the memory operation sequence.

[0264] Here, each time window can have a unique corresponding preset duration threshold, which is determined by the first estimated duration of the corresponding time window; the preset duration thresholds corresponding to each first time window can also be the same. The preset duration threshold can be an empirical value determined by evaluating the execution duration of memory operations in each time window during the historical execution or testing of the same memory operation sequence. This disclosure embodiment does not limit this.

[0265] After determining the sequentially consecutive third target time windows, embodiments of this disclosure can merge these sequentially consecutive third target time windows into a single time window, thereby re-timing the operations that can be performed in each time window.

[0266] Here, taking the example of the second scenario mentioned above, the steps to merge these temporally consecutive third target time windows into a single time window are as follows: First, determine the earliest fourth target time window (the first third time window) among at least two temporally adjacent third target time windows. Then, remove the second variable corresponding to the data outgoing operations of the fourth target time window from the second variable corresponding to the data outgoing operations of the fourth target time window, resulting in multiple third target variables. Simultaneously, determine the latest fifth target time window (the last third time window) among at least two temporally adjacent third target time windows. Then, remove the first variable corresponding to the data incoming operations of the fifth target time window from the first variable corresponding to the data incoming operations of the fifth target time window, resulting in multiple fourth target variables. After determining the multiple fourth target variables and multiple fifth target variables, set them all to their third values ​​as the third constraint conditions.

[0267] The third value can be 0, indicating that no data outgoing operation is performed within the candidate time window corresponding to the third target variable, and no data incoming operation is performed within the candidate time window corresponding to the fourth target variable. Thus, by setting all first and second variables to 0 in at least two consecutive third time windows, except for the second variable for data outgoing operations in the first time window and the first variable for data incoming operations in the last time window (which are unrestricted), these third time windows are merged into a single time window.

[0268] For example, see Figure 17 , Figure 17 This is a schematic diagram illustrating time window fusion according to an exemplary embodiment; wherein, through the aforementioned third constraint, time window 2, time window 3 and time window 4 can be merged to form a new time window.

[0269] In this embodiment of the disclosure, by adding a third constraint, an updated linear programming model can be formed using the first constraint, the second constraint, the third constraint, and the objective function. This makes the values ​​of the decision variables obtained by the linear programming model more reasonable and more practical.

[0270] In this embodiment of the disclosure, the timing arrangement method for memory operation sequences can be applied to AI models, combined with... Figure 2For operations in different network nodes of an AI model that require asynchronous execution by asynchronous engines, the DMA engine can move input and weight data from the memory area to the first buffer. After the computation engine calculates each type of operation, it may need to move the result data from the first buffer to the memory area. The most basic constraint for arranging the execution order of these two asynchronous engines is to satisfy data dependencies. For example, it requires that the DMA engine move its input data into the first buffer before each memory operation is calculated; if the result data of a memory operation needs to be moved out, the DMA engine should only be scheduled to perform the data removal operation after the memory operation has been completed. Additionally, for example... Figure 3 In this sequence, the weight data for operation 3 needs to be moved into the first buffer in advance. Meanwhile, operation 2 might be executing, requiring the first buffer to allocate memory space for its input and result data. Furthermore, the result data for operation 1 might also need to be moved out of memory at this time. This means that while operation 2 is executing, the space occupied by the result of operation 1 in the first buffer must also be guaranteed. This implies that the input or result data of multiple adjacent memory operations share the limited memory of the first buffer with the input or result data of the current operation. Therefore, each time window of the memory operation sequence requires sufficient memory allocation in the first buffer to accommodate these tensors. Simultaneously, since both the computation engine and the DMA engine have execution time, minimizing the total execution latency of the memory operation sequence depends on the ability to hide the execution times of the parallel computation engines and DMA engines within each time window from each other, overall.

[0271] In related technologies, timing arrangements are hard-coded, but can be programmed using complex conditional loops and other control flows. For example, a basic model might be: sorting the memory operations in the memory operation sequence corresponding to the AI ​​model according to depth-first topological order, determining a time window for each memory operation, and combining the start and end time windows to obtain... Figure 4 The six time windows shown; and for each memory operation, it can be as follows: Figure 4 The approach involves either loading input data into a time window before the time window corresponding to the first type of operation, or loading output data into a time window after it. However, in practice, there are also execution methods with lower memory usage and lower overall latency; see [link to relevant documentation]. Figure 18 , Figure 18 This is a temporal distribution of a memory operation sequence according to an exemplary embodiment. Figure 2 ; Figure 18 and Figure 4 This describes the timing arrangement method for operations within the same memory operation sequence, based on actual testing. Figure 18 The overall latency is lower, and the memory usage of the first cache is smaller.

[0272] Therefore, if we need to consider all possible time windows for moving in weighted data, computational data, and result data, these possibilities increase the complexity of hard coding. Furthermore, all tensors may have similar multiple options, resulting in a large number of overall decision points. It's impossible to handle each decision individually in the coding process; therefore, it can only be done as follows: Figure 4 As shown, a fixed time window is used to advance or delay the current time, but such a fixed arrangement will certainly not yield the optimal solution.

[0273] In view of this, this disclosure proposes a timing arrangement method, the specific process of which is described above. The timing arrangement scheme of this disclosure takes each operation in the memory operation sequence of the AI ​​model that needs to be executed asynchronously by the asynchronous engine as an optimization decision point, and abstracts the optimization decision point as a decision variable. In this way, by setting constraints and objective functions, a linear programming model that is effective for all optimization decision points is obtained by overall planning. Through the linear programming model, the decision results of each optimization decision point are obtained, thereby reducing the overall latency of the execution of the memory operation sequence and improving the memory shortage problem. It can also be applied to scenarios such as multi-core data sharing and the resident sharing of small memory weight data.

[0274] Here, when the aforementioned AI model is a ResNet50 model, this embodiment of the disclosure utilizes the above-described timing arrangement method to perform timing arrangement processing on the ResNet50 training process, and trains ResNet50 based on the obtained execution timing information. Through testing, the overall execution time obtained by performing forward training on ResNet50 using the execution timing information is 8409.05us, which is a 15% improvement compared to the overall time of 9893.63us without timing arrangement. In addition, the overall execution time obtained by performing backward training on ResNet50 using the execution timing information is 299120.60us, which is a 36% improvement compared to the overall time of 473676.13us without timing arrangement.

[0275] This disclosure also proposes a timing arrangement device. See also Figure 19 , Figure 19 This is a schematic diagram illustrating the structure of a timing arrangement device according to an exemplary embodiment. The timing arrangement device 1900 mainly includes:

[0276] The first determining module 1901 is configured to determine at least one candidate time window of a second type of operation associated with each first type of operation in the time sequence based on the first time window occupied by each first type of operation to be executed in the memory operation sequence in each time window of the time sequence; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation.

[0277] The first estimation module 1902 is configured to determine the first estimated duration of each time window in the time series based on the first time window corresponding to the first type of operation; the first estimated duration of a time window is the duration required for the first type of operation within the time window to be executed by the first engine.

[0278] The second prediction module 1903 is configured to determine the second estimated duration of each time window in the time series based on at least one candidate time window for each second type of operation; the second estimated duration of a time window is the duration required for the second type of operation within the time window to be executed by the second engine.

[0279] The second determining module 1904 is configured to determine a second time window corresponding to each second type of operation from at least one candidate time window for each second type of operation based on a first estimated duration and a second estimated duration for each time window; wherein, when the second type of operation is executed by the second engine within the corresponding second time window, the total execution latency corresponding to the memory operation sequence is less than when the second type of operation is executed by the second engine within any candidate time window other than the corresponding second time window; the total execution latency corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window within the time sequence;

[0280] The timing determination module 1905 is configured to determine the execution timing information of the memory operation sequence based on the first time window of each first type of operation and the second time window of the second type of operation associated with the first type of operation.

[0281] In some embodiments, the second prediction module 1903 is further configured to set multiple decision variables based on at least one candidate time window for each second type of operation; wherein each decision variable corresponds to a candidate time window for a second type of operation; different values ​​of the decision variables are used to indicate whether the second type of operation is executed by the second engine within the corresponding candidate time window; and the second predicted duration of each time window is determined based on at least one second type of operation expected to be executed within each time window in the time series, the decision variables corresponding to the second type of operation, and the execution duration required to execute the second type of operation.

[0282] The second determining module 1904 is further configured to determine the total execution delay corresponding to the memory operation sequence based on the first estimated duration and the second estimated duration of each time window in the time series; to determine the minimization of the total execution delay corresponding to the memory operation sequence as the objective function; and to determine the target values ​​of multiple decision variables based on the constraints of multiple decision variables and the objective function; wherein the target values ​​of the decision variables are used to determine the second time window of the second type of operation corresponding to the decision variables.

[0283] In some embodiments, the second determining module 1904 is further configured to: determine a first constraint condition based on the constraint relationship between decision variables corresponding to at least one candidate time window of each second type of operation; for each time window of the time series, determine a first estimated memory amount of the first type of operation corresponding to the first engine execution time window in the first cache; determine a second estimated memory amount of the operation data associated with at least one second type of operation to be executed in each time window in the first cache based on the memory amount occupied by the operation data associated with each second type of operation and at least one decision variable corresponding to the second type of operation; determine a second constraint condition based on the first estimated memory amount and the second estimated memory amount in each time window and the memory amount of the first cache; the second constraint condition restricts the sum of the first estimated memory amount and the second estimated memory amount in each time window to be less than the memory amount of the first cache; and determine the target values ​​of multiple decision variables based on the objective function and the first and second constraint conditions.

[0284] In some embodiments, the first determining module 1901 is further configured to determine at least one time window in the time sequence that is earlier than the first time window as at least one candidate time window for a first sub-operation associated with the first type of operation, based on the first time window of each first type of operation to be executed in the memory operation sequence; wherein the execution time of the first sub-operation is earlier than the execution time of the first type of operation associated with the first sub-operation; and to determine at least one time window in the time sequence that is later than the first time window as at least one candidate time window for a second sub-operation associated with the first type of operation, based on the first time window of each first type of operation; wherein the execution time of the second sub-operation is later than the execution time of the first type of operation associated with the second sub-operation; wherein the decision variables include: a first variable corresponding to the first sub-operation and a second variable corresponding to the second sub-operation.

[0285] In some embodiments, the second determining module 1904 is further configured to: determine a first sum value among the first variables corresponding to each candidate time window of each first sub-operation; set the first sum value among the first variables of each first sub-operation equal to the first value to obtain a first constraint relationship; the first constraint relationship is used to restrict the second engine from executing the first sub-operation in one candidate time window of at least one candidate time window of the first sub-operation; determine a second sum value among the second variables corresponding to each candidate time window of each second sub-operation; set the second sum value among the second variables of each second sub-operation equal to the first value to obtain a second constraint relationship; the second constraint relationship is used to restrict the second engine from executing the second sub-operation in one candidate time window of at least one candidate time window of the second sub-operation; and determine a first constraint condition based on the first constraint relationship and the second constraint relationship.

[0286] In some embodiments, the second determining module 1904 is further configured to: determine at least one first target variable of the first sub-operation within the i-th time window from among a plurality of first variables associated with the first sub-operation expected to be executed in the i-th time window; the timing of the candidate time window corresponding to the at least one first target variable is less than or equal to i; i is greater than or equal to 1 and less than or equal to N, where N is the number of time windows contained in the time series; determine at least one second target variable of the second sub-operation within the i-th time window from among a plurality of second variables associated with the second sub-operation expected to be executed in the i-th time window; the timing of the candidate time window corresponding to the at least one second target variable is greater than or equal to i; and determine a second estimated memory amount occupied by the operation data associated with at least one second type of operation expected to be executed in the i-th time window in the first cache based on the memory amount occupied by the at least one first target variable of the first sub-operation and the operation data associated with the first sub-operation, and the memory amount occupied by the at least one second target variable of the second sub-operation and the operation data associated with the second sub-operation in the i-th time window.

[0287] In some embodiments, the timing arrangement device further includes: a setting module;

[0288] The configuration module is configured to, in the case that there are at least two non-adjacent target operations among multiple first-type operations in a memory operation sequence, and the result data of the target operation with a earlier time sequence is used as the input data of the target operation with a later time sequence, determine the candidate time window of the third-type operation associated with the target operation based on at least one time window between the first time windows occupied by the at least two target operations in the time sequence; and set a third variable based on the candidate time window of the third-type operation associated with the target operation; wherein each third variable corresponds to a candidate time window of a third-type operation; and different values ​​of the third variable are used to indicate whether the result data of the target operation with a earlier time sequence is resident in the first buffer within the candidate time window of the third-type operation.

[0289] In some embodiments, the second determining module 1904 is further configured to determine a first constraint condition based on the constraint relationship between the second variables corresponding to each candidate time window of the second target sub-operation, the first variables corresponding to each candidate time window of the first target sub-operation, and the third variables of the third type of operation associated with the target operation; wherein the second target sub-operation is the second sub-operation associated with the target operation that comes first in time; and the first target sub-operation is the first sub-operation associated with the target operation that comes later in time.

[0290] In some embodiments, the second determining module 1904 is further configured to determine a third sum value among the second variables corresponding to each candidate time window of the second target sub-operation, and a fourth sum value among the first variables corresponding to each candidate time window of the first target sub-operation; set the third sum value among the second variables corresponding to each candidate time window of the second target sub-operation equal to the fourth sum value among the first variables corresponding to each candidate time window of the first target sub-operation, to obtain a third constraint relationship; the third constraint relationship is used to restrict that if the second target sub-operation is executed in any candidate time window of the corresponding at least one candidate time window, the second engine needs to execute the first target sub-operation in any candidate time window of the at least one candidate time window of the first target sub-operation; set the sum value among the second and third variables corresponding to each candidate time window of the second target sub-operation equal to the first value, to obtain a fourth constraint relationship; the fourth constraint relationship is used to restrict the second target sub-operation in any candidate time window of the corresponding at least one candidate time window. If the execution condition is met, the third type of operation associated with the target operation that precedes the first target operation does not need to be executed, or if the third type of operation associated with the target operation that precedes the first target operation is executed, the second target sub-operation does not need to be executed. A first parameter is determined based on the product of the timing value corresponding to each candidate time window of the first target sub-operation and the first variable corresponding to the candidate time window, and the product of the timing value corresponding to each candidate time window of the second target sub-operation and the second variable corresponding to the candidate time window. The first parameter indicates the execution time difference between the first and second target sub-operations when the second engine executes the first and second target sub-operations. A fifth constraint relationship is obtained by setting the fifth sum of the first and second parameters to be greater than or equal to the second value. The second parameter is the product of the second value and the third variable. The fifth constraint relationship restricts the execution time difference between the first and second target sub-operations from being greater than the second value. A first constraint condition is determined based on the third, fourth, and fifth constraint relationships.

[0291] In some embodiments, the second determining module 1904 is further configured to determine, based on the product of the amount of memory occupied by the result data of the target operation and the third variable, the third estimated amount of memory occupied by the result data of the target operation in each candidate time window of the third type of operation associated with the target operation in the time sequence.

[0292] The second constraint is determined based on the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount within each time window, as well as the memory amount of the first cache.

[0293] The second constraint also limits the sum of the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount in each time window to be less than the memory amount of the first cache.

[0294] In some embodiments, the timing arrangement device further includes a memory management module;

[0295] The memory management module is configured to determine the first type of operation and / or the second type of operation to be executed in each time window of the time sequence based on the execution timing information of the memory operation sequence; allocate memory resources of the first type of operation and / or the second type of operation to be executed in each time window; and release at least a portion of the memory resources allocated to the first type of operation and / or the second type of operation after executing the first type of operation and / or the second type of operation based on the allocated memory resources.

[0296] In some embodiments, the memory management module is further configured to: insert a fourth time window into the time series if memory resource allocation for a first type of operation and / or a second type of operation to be executed within a third time window in the time series fails; the fourth time window is the time window preceding the third time window; determine a first storage location of first target data in a first cache; wherein the first target data is operation data stored in the first cache, and the first storage location of the first target data in the first cache is not adjacent to a second storage location of operation data other than the first target data in the first cache; move the first target data from the first storage location in the first cache to a third storage location in the first cache within the fourth time window; the third storage location is a free storage location adjacent to the second storage location among multiple free storage locations in the first cache; after completing the movement of the first target data, reallocate memory resources for the first type of operation and / or the second type of operation to be executed within the third time window.

[0297] In some embodiments, the memory management module is further configured to: determine the memory usage and remaining memory of the first cache in each time window based on the first type of operation and / or the second type of operation to be executed in each time window of the time series; determine a target remaining memory based on the remaining memory corresponding to multiple time windows in the time series; the target remaining memory is the minimum value among the remaining memory corresponding to multiple time windows; set up a resident storage area in the first cache; the memory size of the resident storage area is determined by the target remaining memory, and the storage location of the resident storage area is located at the end of the first cache; determine second target data from the operation data associated with each first type of operation to be executed in the memory operation sequence; the second target data is the weight data of the first type of operation; determine at least one third target data from at least one second target data based on the memory size required by the second target data; the sum of the memory sizes of at least one third target data is less than or equal to the memory size of the resident storage area, and the memory size of each third target data is less than the memory size of the second target data other than the third target data in at least one second target data; and store at least one third target data in the resident storage area.

[0298] In some embodiments, the memory management module is further configured to, when there is at least one identical first type of operation in the memory operation sequence of the first kernel and the memory operation sequence of the second kernel, determine at least one identical fourth target data from the first operation data and the second operation data based on the at least one identical first type of operation; wherein the first operation data is the operation data of the first sub-operation associated with at least one identical first type of operation in the first kernel; the second operation data is the operation data of the first sub-operation associated with at least one identical first type of operation in the second kernel; determine a first target time window and a second target time window according to the second time window of the first sub-operation associated with the fourth target data in the first kernel and the second time window of the first sub-operation associated with the fourth target data in the second kernel; the timing of the first target time window is earlier than any second time window in the second time window of the first kernel and the second time window of the second kernel; the timing of the second target time window is later than any second time window in the second time window of the first kernel and the second time window of the second kernel; store the fourth target data in a second cache within the first target time window; wherein the fourth target data stored in the second cache can be shared by the first kernel and the second kernel; and release the memory resources occupied by the fourth target data in the second cache within the second target time window.

[0299] In some embodiments, the second determining module 1904 is further configured to determine at least two temporally adjacent third target time windows from a plurality of time windows in the time series based on the second estimated duration of each time window in the time series; the second estimated duration of each third target time window is less than a preset duration threshold; based on the earliest fourth target time window in the at least two temporally adjacent third target time windows, determine a plurality of third target variables from the second variables corresponding to the at least two temporally adjacent third target time windows; the third target variables are the second variables corresponding to the third target time windows other than the fourth target time window in the at least two temporally adjacent third target time windows; based on the latest fourth target time window in the at least two temporally adjacent third target time windows... The fifth objective time window is determined by selecting multiple fourth objective variables from the first variables corresponding to at least two temporally adjacent third objective time windows. The fourth objective variables are the first variables corresponding to the third objective time windows (excluding the fifth objective time window) within at least two temporally adjacent third objective time windows. The values ​​of the multiple third and fourth objective variables are set to be equal to their third values, resulting in the third constraint condition. The third value of the third objective variable indicates that the second sub-operation should not be executed within the candidate time window corresponding to the third objective variable. The third value of the fourth objective variable indicates that the first sub-operation should not be executed within the candidate time window corresponding to the fourth objective variable. Based on the objective function and the first, second, and third constraint conditions, the objective value is determined.

[0300] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0301] Figure 20 This is a structural block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 2000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0302] Reference Figure 20 The electronic device 2000 may include one or more of the following components: processing component 2002, memory 2004, power supply component 2006, multimedia component 2008, audio component 2010, input / output interface 2012, sensor component 2014, and communication component 2016.

[0303] Processing component 2002 typically controls the overall operation of electronic device 2000, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 2002 may include one or more processors 2020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 2002 may include one or more modules to facilitate interaction between processing component 2002 and other components. For example, processing component 2002 may include a multimedia module to facilitate interaction between multimedia component 2008 and processing component 2002.

[0304] Memory 2004 is configured to store various types of data to support operation on electronic device 2000. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 2000, contact data, phonebook data, messages, pictures, and videos. Memory 2004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0305] Power supply component 2006 provides power to various components of electronic device 2000. Power supply component 2006 may include at least one of the following: power management system, one or more power supplies, and other components associated with generating, managing and distributing power to electronic device 2000.

[0306] Multimedia component 2008 includes a screen that provides an output interface between electronic device 2000 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 2008 includes a front-facing camera and / or a rear-facing camera. When electronic device 2000 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0307] Audio component 2010 is configured to output and / or input audio signals. For example, audio component 2010 includes a microphone (MIC) configured to receive external audio signals when electronic device 2000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 2004 or transmitted via communication component 2016. In some embodiments, audio component 2010 also includes a speaker for outputting audio signals.

[0308] Input / output interface 2012 provides an interface between processing component 2002 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0309] Sensor assembly 2014 includes one or more sensors for providing state assessment of various aspects of electronic device 2000. For example, sensor assembly 2014 can detect the on / off state of electronic device 2000, the relative positioning of components, such as the display and keypad of electronic device 2000, changes in position of electronic device 2000 or one of its components, the presence or absence of user contact with electronic device 2000, orientation or acceleration / deceleration of electronic device 2000, and temperature changes of electronic device 2000. Sensor assembly 2014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 2014 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 2014 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0310] The communication component 2016 is configured to facilitate wired or wireless communication between the electronic device 2000 and other devices. The electronic device 2000 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component 2016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 2016 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0311] In an exemplary embodiment, the electronic device 2000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0312] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 2004 including executable instructions or a computer program, which can be executed by a processor 2020 of an electronic device 2000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0313] This disclosure also proposes a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to execute any of the timing arrangement methods described above in this disclosure.

[0314] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the timing arrangement methods described above in this disclosure.

[0315] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0316] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A timing arrangement method, characterized by, The method comprises: determining at least one candidate time window of a second type operation associated with each first type operation in each time window of a time sequence according to the first time window of each first type operation to be executed in the time sequence; the candidate time window of the second type operation is different from the first time window of the first type operation associated with the same second type operation; determining a first estimated duration of each time window of the time sequence according to the first time window corresponding to the first type operation; the first estimated duration of one time window is the duration required for the first type operation in the time window to be executed by the first engine; determining a second estimated duration of each time window of the time sequence according to at least one candidate time window of each second type operation; the second estimated duration of one time window is the duration required for the second type operation in the time window to be executed by the second engine; determining the second time window corresponding to each second type operation from at least one candidate time window of each second type operation according to the first estimated duration and the second estimated duration of each time window; wherein the total execution delay corresponding to the memory operation sequence when the second type operation is executed in the corresponding second time window by the second engine is less than the total execution delay corresponding to the memory operation sequence when the second type operation is executed in any candidate time window other than the corresponding second time window by the second engine; the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence; determining the execution timing information of the memory operation sequence according to the first time window of each first type operation and the second time window of the second type operation associated with the first type operation.

2. The method of claim 1, wherein, The method of determining the second estimated duration of each time window of the time sequence according to at least one candidate time window of each second type operation comprises: setting a plurality of decision variables according to at least one candidate time window of each second type operation; wherein each decision variable corresponds to a candidate time window of a second type operation; different values of the decision variable are used to indicate whether the second type operation is executed in the corresponding candidate time window by the second engine; determining the second estimated duration of each time window according to at least one second type operation expected to be executed in each time window of the time sequence, and the decision variable corresponding to the second type operation and the execution duration required for executing the second type operation; The method of determining the second time window corresponding to each second type operation from at least one candidate time window of each second type operation according to the first estimated duration and the second estimated duration of each time window comprises: determining the total execution delay corresponding to the memory operation sequence according to the first estimated duration and the second estimated duration of each time window of the time sequence; determining a target function aiming at minimizing a total execution delay corresponding to the memory operation sequence; determining target values of the plurality of decision variables according to the constraint conditions of the plurality of decision variables and the target function; wherein the target values of the decision variables are used to determine second time windows of the second type operations corresponding to the decision variables.

3. The method of claim 2, wherein, The method further comprises: determining first constraint conditions according to constraint relationships between the decision variables corresponding to the at least one candidate time window of each second type operation; determining, for each time window of the time sequence, a first estimated memory amount of the first cache area in which the first engine executes the first type operations corresponding to the time window; determining, according to the memory amount occupied by the operation data associated with each second type operation and the at least one decision variable corresponding to the second type operation, a second estimated memory amount of the first cache area occupied by the operation data associated with at least one second type operation expected to be executed in each time window; determining second constraint conditions according to the first estimated memory amount and the second estimated memory amount in each time window and the memory amount of the first cache area; the second constraint conditions limit the sum of the first estimated memory amount and the second estimated memory amount in each time window to be less than the memory amount of the first cache area; The determining target values of the plurality of decision variables according to the constraint conditions of the plurality of decision variables and the target function comprises: determining the target values of the plurality of decision variables based on the target function and the first constraint conditions and the second constraint conditions.

4. The method of claim 3, wherein, The determining, from the time sequence, at least one candidate time window of a second type operation associated with each first type operation to be executed in the memory operation sequence according to a first time window occupied by each first type operation in a time sequence of the time sequence comprises: determining, according to the first time window of each first type operation to be executed in the memory operation sequence, at least one time window earlier than the first time window in the time sequence as at least one candidate time window of a first sub-operation associated with the first type operation; wherein the execution timing of the first sub-operation is earlier than the execution timing of the first type operation associated with the first sub-operation; determining, according to the first time window of each first type operation, at least one time window later than the first time window in the time sequence as at least one candidate time window of a second sub-operation associated with the first type operation; the execution timing of the second sub-operation is later than the execution timing of the first type operation associated with the second sub-operation; wherein the decision variables comprise: a first variable corresponding to the first sub-operation and a second variable corresponding to the second sub-operation.

5. The method of claim 4, wherein, The determining first constraint conditions according to constraint relationships between the decision variables corresponding to the at least one candidate time window of each second type operation comprises: determining a first sum value between the first variables corresponding to each candidate time window of each first sub-operation; set a first sum value between each first variable of each first sub-operation to be equal to a first value, to obtain a first constraint relationship; the first constraint relationship is used to limit the second engine to execute the first sub-operation in a candidate time window of at least one candidate time window of the first sub-operation; determine a second sum value between each candidate time window corresponding second variable of each second sub-operation; set a second sum value between each second variable of each second sub-operation to be equal to the first value, to obtain a second constraint relationship; the second constraint relationship is used to limit the second engine to execute the second sub-operation in a candidate time window of at least one candidate time window of the second sub-operation; determine the first constraint condition according to the first constraint relationship and the second constraint relationship.

6. The method of claim 4, wherein, determining a second estimated memory amount occupied by operation data associated with at least one second type operation expected to be executed in each time window in the first cache area according to the memory amount occupied by the operation data associated with each second type operation and at least one decision variable corresponding to the second type operation, comprises: determining at least one first target variable of the first sub-operation in the ith time window from a plurality of first variables associated with the first sub-operation expected to be executed in the ith time window; the candidate time window corresponding to the at least one first target variable has a time sequence less than or equal to i; i is greater than or equal to 1 and less than or equal to N, N is the number of time windows included in the time sequence; determining at least one second target variable of the second sub-operation in the ith time window from a plurality of second variables associated with the second sub-operation expected to be executed in the ith time window; the candidate time window corresponding to the at least one second target variable has a time sequence greater than or equal to i; determining a second estimated memory amount occupied by operation data associated with at least one second type operation expected to be executed in the ith time window in the first cache area according to the memory amount occupied by the operation data associated with at least one first target variable of the first sub-operation in the ith time window and the first sub-operation, and the memory amount occupied by the operation data associated with at least one second target variable of the second sub-operation in the ith time window and the second sub-operation.

7. The method of claim 4, wherein, The method further comprises: in the case that there are at least two time-sequentially non-adjacent target operations in the plurality of first type operations of the memory operation sequence, and the result data of the target operation with earlier time sequence in the at least two target operations is taken as the input data of the target operation with later time sequence, determining a candidate time window of a third type operation associated with the target operation according to at least one time window between the first time windows occupied by the at least two target operations in the time sequence; A third variable is set according to a candidate time window of a third type of operation associated with the target operation; each third variable corresponds to a candidate time window of the third type of operation; different values of the third variable are used to indicate whether result data of the target operation in time sequence in front of the candidate time window of the third type of operation is cached in the first cache area.

8. The method of claim 7, wherein, The first constraint condition is determined according to a constraint relationship between the decision variables corresponding to the at least one candidate time window of each second type of operation, including: The first constraint condition is determined according to a constraint relationship between the second variables corresponding to the candidate time windows of the second target sub-operation, the first variables corresponding to the candidate time windows of the first target sub-operation, and the third variables of the third type of operation associated with the target operation; the second target sub-operation is a second sub-operation associated with the target operation in time sequence in front of the target operation; the first target sub-operation is a first sub-operation associated with the target operation in time sequence behind the target operation.

9. The method of claim 8, wherein, The first constraint condition is determined according to a constraint relationship between the second variables corresponding to the candidate time windows of the second target sub-operation, the first variables corresponding to the candidate time windows of the first target sub-operation, and the third variables of the third type of operation associated with the target operation, including: A third sum value between the second variables corresponding to the candidate time windows of the second target sub-operation and a fourth sum value between the first variables corresponding to the candidate time windows of the first target sub-operation are determined; The third sum value between the second variables corresponding to the candidate time windows of the second target sub-operation is set to be equal to the fourth sum value between the first variables corresponding to the candidate time windows of the first target sub-operation, to obtain a third constraint relationship; the third constraint relationship is used to limit that the second engine needs to execute the first target sub-operation in any candidate time window of the at least one candidate time window of the first target sub-operation in the case that the second target sub-operation is executed in any candidate time window of the at least one candidate time window corresponding thereto; A sum value between the second variables corresponding to the candidate time windows of the second target sub-operation and the third variables is set to be equal to a first value, to obtain a fourth constraint relationship; the fourth constraint relationship is used to limit that the third type of operation associated with the target operation in time sequence in front of the target operation does not need to be executed in the case that the second target sub-operation is executed in any candidate time window of the at least one candidate time window corresponding thereto, or, the second target sub-operation does not need to be executed in the case that the third type of operation associated with the target operation in time sequence in front of the target operation is executed. determining a first parameter according to a product between a time sequence value corresponding to each candidate time window of the first target sub-operation and a first variable corresponding to the candidate time window, and a product between a time sequence value corresponding to each candidate time window of the second target sub-operation and a second variable corresponding to the candidate time window; the first parameter is used to indicate an execution time difference between the first target sub-operation and the second target sub-operation in a case that a second engine executes the first target sub-operation and the second target sub-operation; setting a fifth sum of the first parameter and a second parameter to be greater than or equal to a second value, to obtain a fifth constraint relationship; the second parameter is a product of the second value and the third variable; the fifth constraint relationship is used to limit the execution time difference between the first target sub-operation and the second target sub-operation to be greater than the second value; determining the first constraint condition according to the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship.

10. The method of claim 7, wherein, The determining the second constraint condition according to the first estimated memory amount and the second estimated memory amount in each time window, and the memory amount of the first cache area comprises: determining a third estimated memory amount of result data in the first cache area in each candidate time window of a third type of operation associated with a target operation in time sequence in front, according to a product of a memory amount occupied by the result data of the target operation and the third variable; determining the second constraint condition according to the first estimated memory amount, the second estimated memory amount and the third estimated memory amount in each time window, and the memory amount of the first cache area; The second constraint condition is further used to limit a sum of the first estimated memory amount, the second estimated memory amount and the third estimated memory amount in each time window to be less than the memory amount of the first cache area.

11. The method according to any one of claims 4 to 10, characterized in that, The method further comprises: determining a first type of operation and / or a second type of operation to be executed in each time window in the time sequence according to execution time sequence information of the memory operation sequence; allocating memory resources of the first cache area to the first type of operation and / or the second type of operation to be executed in each time window; releasing at least part of the memory resources allocated to the first type of operation and / or the second type of operation after the first type of operation and / or the second type of operation is executed based on the allocated memory resources.

12. The method of claim 11, wherein, The method further comprises: if memory resource allocation fails for the first type of operation and / or the second type of operation to be executed in a third time window in the time sequence, inserting a fourth time window in the time sequence; the fourth time window is a time window in front of the third time window; determining a first storage position of first target data in the first cache area; the first target data is operation data stored in the first cache area, and the first storage position of the first target data in the first cache area is not adjacent to a second storage position of operation data other than the first target data in the first cache area; moving the first target data from the first storage location of the first cache area to a third storage location of the first cache area within the fourth time window; the third storage location is a free storage location adjacent to the second storage location among the free storage locations of the first cache area; after the moving of the first target data is completed, re-allocating memory resources for the first type of operations and / or the second type of operations to be executed within a third time window.

13. The method of claim 11, wherein, The method further comprises: determining the memory occupancy and the remaining memory of the first cache area within each time window in the time sequence according to the first type of operations and / or the second type of operations to be executed within each time window; determining a target remaining memory according to the remaining memories corresponding to the plurality of time windows in the time sequence; the target remaining memory is the minimum value among the remaining memories corresponding to the plurality of time windows; setting a resident storage area in the first cache area; the memory amount of the resident storage area is determined by the target remaining memory, and the storage location of the resident storage area is located at the tail of the first cache area; determining second target data from the operation data associated with each first type of operation to be executed in the memory operation sequence; the second target data is the weight data of the first type of operation; determining at least one third target data from the at least one second target data according to the memory amount required by the second target data; the sum of the memory amounts of the at least one third target data is less than or equal to the memory amount of the resident storage area, and the memory amount of each third target data is less than the memory amount of the second target data other than the third target data among the at least one second target data; storing the at least one third target data in the resident storage area.

14. The method according to any one of claims 4 to 10, characterized in that, The method further comprises: in the case that there is at least one same first type of operation in the memory operation sequence of the first core and the memory operation sequence of the second core, determining at least one same fourth target data from the first operation data and the second operation data based on the at least one same first type of operation; wherein the first operation data is the operation data of a first sub-operation associated with the at least one same first type of operation in the first core; the second operation data is the operation data of a first sub-operation associated with the at least one same first type of operation in the second core; determining a first target time window and a second target time window according to the second time window of the first sub-operation associated with the fourth target data in the first core and the second time window of the first sub-operation associated with the fourth target data in the second core; the time sequence of the first target time window is earlier than that of any one of the second time window of the first core and the second time window of the second core; the time sequence of the second target time window is later than that of any one of the second time window of the first core and the second time window of the second core; In the first target time window, the fourth target data is stored in a second cache area; wherein the fourth target data stored in the second cache area can be shared by the first kernel and the second kernel; And in the second target time window, the memory resource occupied by the fourth target data in the second cache area is released.

15. The method according to any one of claims 4 to 10, characterized in that, The method further comprises: According to the second estimated duration of each time window in the time sequence, at least two time-sequentially adjacent third target time windows are determined from the multiple time windows of the time sequence; the second estimated duration of each third target time window is less than a preset duration threshold; According to the fourth target time window which is the earliest in time among the at least two time-sequentially adjacent third target time windows, multiple third target variables are determined from the second variables corresponding to the at least two time-sequentially adjacent third target time windows; the third target variable is the second variable corresponding to the third target time window other than the fourth target time window among the at least two time-sequentially adjacent third target time windows; According to the fifth target time window which is the latest in time among the at least two time-sequentially adjacent third target time windows, multiple fourth target variables are determined from the first variables corresponding to the at least two time-sequentially adjacent third target time windows; the fourth target variable is the first variable corresponding to the third target time window other than the fifth target time window among the at least two time-sequentially adjacent third target time windows; The values of the multiple third target variables and the multiple fourth target variables are set to be equal to a third value, to obtain a third constraint condition; wherein the third value of the third target variable is used to indicate that the second sub-operation is not performed in the candidate time window corresponding to the third target variable; the third value of the fourth target variable is used to indicate that the first sub-operation is not performed in the candidate time window corresponding to the fourth target variable; The determination of the target values of the multiple decision variables based on the target function and the first constraint condition, the second constraint condition comprises: The determination of the target values based on the target function and the first constraint condition, the second constraint condition and the third constraint condition.

16. A timing arrangement device, characterized by Comprise: The first determination module is configured to determine at least one candidate time window of a second type of operation associated with each first type of operation in the memory operation sequence according to the first time window occupied by each to-be-executed first type of operation in the time sequence of each time window; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation; The first estimation module is configured to determine the first estimated duration of each time window in the time sequence according to the first time window corresponding to the first type of operation; The first estimated duration of one time window is the duration required for the first type of operation in the time window to be executed by the first engine; a second estimation module configured to determine a second estimated duration of each time window in the time sequence according to at least one candidate time window of each second type operation; a second estimated duration of one of the time windows is a duration required for a second type operation in the time window to be executed by the second engine; a second determination module configured to determine a second time window corresponding to each second type operation from at least one candidate time window of each second type operation according to the first estimated duration and the second estimated duration of each time window; wherein when the second type operation is executed by the second engine in the corresponding second time window, a total execution delay of the memory operation sequence is less than that when the second type operation is executed by the second engine in any candidate time window other than the corresponding second time window; and the total execution delay of the memory operation sequence is determined by a difference between the first estimated duration and the second estimated duration of each time window in the time sequence; a timing determination module configured to determine execution timing information of the memory operation sequence according to the first time window of each first type operation and the second time window of the second type operation associated with the first type operation.

17. An electronic device, comprising: comprise: a processor; a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps of the method of any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing computer programs or instructions, characterized in that, when the computer programs or instructions in the storage medium are executed by the processor, the steps of the method of any one of claims 1 to 15 are implemented.

19. A computer program product comprising computer programs or instructions, characterized in that, when the computer programs or instructions are executed by the processor, the steps of the method of any one of claims 1 to 15 are implemented.

Images