Time sequence arrangement method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a time sequence arrangement method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of determining at least one candidate time window of a second type of operation associated with each first type of operation according to a first time window occupied by each to-be-executed first type of operation in a memory operation sequence in each time window of a time sequence; determining a first estimated duration of each time window in the time sequence according to a first time window corresponding to the first type of operation; determining a second estimated duration of each time window according to at least one candidate time window of each second type of operation; determining a second time window corresponding to each second type of operation according to the first estimated duration and the second estimated duration of each time window; and determining execution time sequence information of the memory operation sequence according to the first time window of each first type of operation and the second time window of each second type of operation. The total execution delay of the memory operation sequence can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a timing arrangement method and device, electronic equipment, storage medium, and program product. Background Art

[0002] With the continuous development of computer technology, in software and hardware systems of various types and sizes, there are many software engines or hardware engines that can run independently and asynchronously and synchronize when needed. These asynchronous engines can cooperate with each other to complete certain processing tasks.

[0003] However, these asynchronous software engines or hardware engines each have specific task types and task amounts, and therefore have fixed execution times; these fixed execution times cause the asynchronous engines to increase the total execution delay when cooperating to complete certain processing tasks, such as executing the same memory operation sequence, further affecting the operating speed of the corresponding software and hardware systems. Summary of the invention

[0004] To overcome the problems existing in the related art, the embodiments of the present disclosure provide a timing arrangement method and device, electronic device, storage medium, and program product. The timing arrangement method provided by the present disclosure can reduce the total execution delay of executing a memory operation sequence.

[0005] According to a first aspect of an embodiment of the present disclosure, a time sequence arrangement method is provided, the method comprising:

[0006] According to the first time windows occupied by each first-category operation to be executed in the memory operation sequence in each time window of the time sequence, at least one candidate time window of the second-category operation associated with each first-category operation is determined from each time window of the time sequence; the candidate time window of the second-category operation is different from the first time window of the first-category operation associated with the same second-category operation;

[0007] Determine, according to the first time window corresponding to the first type of operation, a first estimated duration of each time window in the time series; the first estimated duration of one of the time windows is the duration required for the first type of operation in the time window to be executed by the first engine;

[0008] Determine, according to at least one candidate time window of each second type of operation, a second estimated duration of each time window in the time series; the second estimated duration of one of the time windows is the duration required for the second type of operation in the time window to be executed by the second engine;

[0009] According to the first estimated duration and the second estimated duration of each time window, a second time window corresponding to each second-category operation is determined from at least one candidate time window of each second-category operation; wherein, when the second-category operation is executed by the second engine in the corresponding second time window, the total execution delay corresponding to the memory operation sequence is less than the total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence;

[0010] The execution timing information of the memory operation sequence is determined according to the first time window of each of the first-type operations and the second time window of the second-type operations associated with the first-type operations.

[0011] According to a second aspect of an embodiment of the present disclosure, a time sequence arrangement device is provided, the device comprising:

[0012] A first determination module is configured to determine, according to first time windows occupied by first type operations to be executed in a memory operation sequence in each time window of a time sequence, at least one candidate time window of a second type operation associated with each first type operation from each time window of the time sequence; the candidate time window of the second type operation is different from the first time window of the first type operation associated with the same second type operation;

[0013] A first estimation module is configured to determine a first estimated duration of each time window in the time series according to a first time window corresponding to the first type of operation; the first estimated duration of a time window is a duration required for the first type of operation in the time window to be executed by the first engine;

[0014] A second estimation module is configured to determine a second estimated duration of each time window in the time series according to at least one candidate time window of each second type of operation; the second estimated duration of one time window is the duration required for the second type of operation in the time window to be executed by the second engine;

[0015] a second determination module, configured to determine, from at least one candidate time window of each second-category operation, a second time window corresponding to each second-category operation according to the first estimated duration and the second estimated duration of each time window; wherein, when the second-category operation is executed by the second engine in the corresponding second time window, the total execution delay corresponding to the memory operation sequence is less than the total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; and the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence;

[0016] The timing determination module is configured to determine the execution timing information of the memory operation sequence according to the first time window of each of the first type of operations and the second time window of the second type of operations associated with the first type of operations.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] processor;

[0019] Memory for storing computer programs or instructions;

[0020] The processor executes the computer program or instructions to implement the steps of the method proposed in the first aspect above.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the storage medium stores a computer program or instructions. When the computer program or instructions in the storage medium are executed by a processor, the steps of the method proposed in the first aspect are implemented.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method proposed in the first aspect.

[0023] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0024] In the timing arrangement method proposed in the embodiment of the present disclosure, the second time window corresponding to each second-class operation when the total execution delay of the memory operation sequence is minimized is determined based on the time required for the first engine to execute each first-class operation to be executed in the memory operation sequence, the time required for the second engine to execute each second-class operation associated with the first-class operation, the first time window corresponding to each first-class operation, and at least one candidate time window that can be selected for each second-class operation; and the execution timing information for timing the memory operation sequence is obtained based on the first time window corresponding to each first-class operation and the second time window corresponding to each second-class operation.

[0025] In this way, when determining the second time window of the second type of operation associated with the first type of operation, compared to the method of selecting a time window as the second time window of the second type of operation according to the first time window of the first type of operation and the timing relationship between the first type of operation and the second type of operation, the embodiment of the present disclosure first determines multiple candidate time windows of the second type of operation from multiple time windows of the time series, and determines the second time window from multiple candidate time windows with the goal of minimizing the total execution delay of the memory operation sequence; on the one hand, the timing arrangement of the second type of operation can be more flexible; on the other hand, because the total execution delay of the memory operation sequence when executing the second type of operation in different candidate time windows is taken into account when determining the second time window, the total execution delay of the memory operation sequence can be minimized to the greatest extent, thereby improving the speed and efficiency of the computer system in executing the memory operation sequence.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] Figure 1 It is a flowchart of a timing arrangement method according to an exemplary embodiment.

[0029] Figure 2 The figure is a schematic diagram showing an asynchronous engine execution principle according to an exemplary embodiment.

[0030] Figure 3 The diagram is a schematic diagram showing an execution principle of a memory operation sequence according to an exemplary embodiment.

[0031] Figure 4 A timing distribution of a memory operation sequence is shown according to an exemplary embodiment. Figure 1 .

[0032] Figure 5 The diagram is a schematic diagram showing configuration of a first variable and a second variable for a memory operation sequence according to an exemplary embodiment.

[0033] Figure 6 is a schematic diagram showing a method of setting a second constraint condition according to an exemplary embodiment.

[0034] Figure 7 The present invention is a schematic diagram showing a principle of determining a first estimated duration and a second estimated duration according to an exemplary embodiment.

[0035] Figure 8 is a topological diagram of a memory operation sequence according to an exemplary embodiment. Figure 1 .

[0036] Fig. 9 is a schematic diagram of setting a third variable, a first variable of a first target sub-operation, and a second variable of a second target sub-operation according to an exemplary embodiment.

[0037] Fig.10 is a topological diagram of a memory operation sequence according to an exemplary embodiment. Figure 2 .

[0038] Fig.11 The diagram is a schematic diagram of a memory management according to an exemplary embodiment.

[0039] Fig.12 The memory resource distribution of the first cache area is shown according to an exemplary embodiment. Figure 1 .

[0040] Fig.13 The memory resource distribution of the first cache area is shown according to an exemplary embodiment. Figure 2 .

[0041] Fig.14 is a schematic diagram of a resident storage area according to an exemplary embodiment Figure 1 .

[0042] Fig.15 is a schematic diagram of a resident storage area according to an exemplary embodiment Figure 2 .

[0043] Fig.16 is a timing diagram showing a method of storing fourth target data in a first core and a second core according to an exemplary embodiment.

[0044] Fig.17 is a schematic diagram of time window fusion according to an exemplary embodiment.

[0045] Fig.18A timing distribution of a memory operation sequence is shown according to an exemplary embodiment. Figure 2 .

[0046] Fig.19 It is a structural schematic diagram of a timing arrangement device according to an exemplary embodiment.

[0047] Fig. 20 It is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] An embodiment of the present disclosure provides a timing arrangement method, by which the total execution time of a memory operation sequence can be effectively reduced.

[0050] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a timing arrangement method according to an exemplary embodiment. The timing arrangement method in the present disclosure can be implemented by the following steps:

[0051] Step 101, according to the first time windows occupied by each first-category operation to be executed in the memory operation sequence in each time window of the time sequence, at least one candidate time window of the second-category operation associated with each first-category operation is determined from each time window of the time sequence; the candidate time window of the second-category operation is different from the first time window of the first-category operation associated with the same second-category operation;

[0052] Step 102, determining a first estimated duration of each time window in the time series according to a first time window corresponding to the first type of operation; the first estimated duration of a time window is the duration required for the first type of operation in the time window to be executed by the first engine;

[0053] Step 103, determining a second estimated duration of each time window in the time series according to at least one candidate time window of each second-category operation; the second estimated duration of a time window is the duration required for the second-category operation in the time window to be executed by the second engine;

[0054] Step 104, determining a second time window corresponding to each second-category operation from at least one candidate time window of each second-category operation according to the first estimated duration and the second estimated duration of each time window; wherein a total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in the corresponding second time window is less than a total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; and the total execution delay corresponding to the memory operation sequence is determined by a difference between the first estimated duration and the second estimated duration of each time window in the time sequence;

[0055] Step 105 : determining execution timing information of the memory operation sequence according to the first time window of each first-category operation and the second time window of the second-category operation associated with the first-category operation.

[0056] Here, the timing arrangement method proposed in the embodiment of the present disclosure can be applied to software and hardware systems of various types and sizes, and implemented in specific application scenarios where these software and hardware systems involve executing memory operation sequences. Among them, these software and hardware systems are located in electronic devices, and the electronic devices can be portable electronic devices, or computer devices, computer device clusters, etc., and the embodiment of the present disclosure does not limit this.

[0057] It should be noted that the memory operation sequence proposed in the embodiment of the present disclosure has memory operations that are continuous in time sequence. These memory operations can be calculation operations of input data by various network nodes in the network model. These memory operations can also be various instruction operations executed by electronic devices in response to user instructions, and of course are not limited to this.

[0058] Exemplarily, when the above-mentioned memory operation sequence is applied in an artificial intelligence (AI) network model, these memory operations are expressed as network operations performed by the network layer on each network node, and the corresponding memory operation sequence is expressed as a directed acyclic graph of the AI ​​model; wherein, an execution flow from one network node to the next network node represents the execution order (timing) of the two memory operations, and each execution flow continuously forks and merges to form a directed acyclic graph of the AI ​​model.

[0059] In the disclosed embodiments, multiple different memory operations within a memory operation sequence can be executed by at least two engines respectively. Although the at least two engines can run independently and asynchronously, the execution time required for the at least two engines to execute the corresponding memory operations is fixed and cannot be compressed. In addition, there is a synchronization point between the memory operations executed by the at least two engines, so that the at least two engines need to wait for each other at the synchronization point during the execution of the corresponding memory operations, and subsequent operations can only be continued after the at least two engines reach the synchronization point. For example, before one engine executes a certain memory operation, it needs to wait for another engine to complete the data move-in operation of the operation data associated with the memory operation before executing the memory operation. Therefore, it becomes increasingly important to arrange the execution timing of each memory operation in the memory operation sequence to reduce the total execution delay of the entire memory operation sequence.

[0060] The timing arrangement method corresponding to the above steps 101 to 105 proposed in the embodiment of the present disclosure enables the execution times of at least two engines to be hidden from each other, thereby reducing the total execution delay of the entire memory operation sequence.

[0061] In step 101, the embodiment of the present disclosure first determines the first time window of each first-category operation to be executed in the memory operation sequence in multiple time windows of the time series; wherein each first-category operation to be executed has a predetermined execution order (timing), one first time window corresponds to one first-category operation, and the corresponding first-category operation can be executed by the first engine in each first time window.

[0062] Here, the first type of operation corresponds to each memory operation in the memory operation sequence. When these memory operations are executed by the first engine, they need to utilize the memory resources in the software and hardware system. It should be noted that due to the large capacity of the memory area in the system and the slow speed of data access (reading and writing), the above-mentioned memory operations are usually performed using the memory resources of the designated cache area, wherein the designated cache area may be an on-chip cache (Cache), such as the first cache area (L1 cache) proposed below in the present disclosure, the distance between the first cache area and the processor (CPU) in the system is closer than the distance between the memory area and the CPU; the memory capacity of the first cache area is smaller than the capacity of the memory area in the system; and the speed at which the CPU accesses data to the first cache area is faster than the speed at which the CPU accesses data to the memory area in the system.

[0063] For example, in the AI ​​model, the first type of operation may be a convolution operation, a pooling operation, a nonlinear mapping operation, etc. corresponding to each network node, and the first engine corresponding to executing the first type of operation may be a computing engine, or a forward and backward reasoning engine, etc.

[0064] In some examples, the computing engine can perform convolution processing on the input data within the corresponding first time window. For example, when the input data is an image, a reduced-dimensional image can be obtained after convolution processing. The computing engine can also perform pooling processing, nonlinear mapping processing, etc. on the input data within the corresponding time window, which will not be elaborated in the embodiments of the present disclosure.

[0065] In other examples, the forward and backward reasoning engine can perform forward calculations on memory operations based on input data within the corresponding first time window to obtain result data that is output to the next memory operation in the sequence, or perform backward calculations on the memory operations to obtain result data that is output to the previous memory operation in the sequence.

[0066] In the disclosed embodiment, the first type of operation in the memory operation sequence is usually associated with the second type of operation, and the complete execution of the memory operation sequence needs to rely on the second engine to execute the second type of operation associated with the first type of operation. Continuing with the above example, if the first engine is a computing engine, the second engine can be a data moving in and out (also known as direct memory access, Direct memory access, DMA) engine; accordingly, the second type of operation can be to use the DMA engine to move the input data required for the execution of the first type of operation from the memory area to the first cache area in advance (hereinafter referred to as the data moving in operation), or to move the result data after the execution of the first type of operation from the first cache area to the memory area (hereinafter referred to as the data moving out operation); if the first engine is a forward and backward reasoning engine, the second engine can be a parameter synchronization communication engine between nodes, and accordingly, the second type of operation can be an update operation of the model parameters of the network layer of each network node.

[0067] See also Figure 2 , Figure 2It is a schematic diagram of the asynchronous engine execution principle according to an exemplary embodiment; wherein, the on-chip cache Cache (first cache area) is a one-dimensional linear physical memory space, and the capacity is generally 256KB; the memory area is a double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM), wherein the DMA engine can perform the input data transfer for the first type of operation, that is, transfer the input data from the DDR to the first cache area; after completing the transfer, the calculation engine uses the input data of the first cache area to calculate, and then outputs the result data to the first cache area for caching; and the DMA engine transfers the calculation result from the first cache area to the DDR. Of course, if the result data of the first type of operation is the input data of the next first type of operation to be executed, then the result data can also reside in the first cache area, and after being transferred to the allocated position of the first cache area according to other input data of the next first type of operation to be executed, the next first type of operation is directly executed.

[0068] For the second type of operations, if the second engine needs to occupy a time window of the time series separately for each second type of operation associated with the first type of operation, the entire execution time will increase; since the second engine and the first engine can execute asynchronously, for a first type of operation, the associated second type of operation can be executed in a time window before (or after) the timing of the first type of operation.

[0069] See also Figure 3 and Figure 4 , Figure 3 is a schematic diagram showing an execution principle of a memory operation sequence according to an exemplary embodiment; Figure 4 A timing distribution of a memory operation sequence is shown according to an exemplary embodiment. Figure 1 .

[0070] here, Figure 3 The first type of operations that can be performed by the computing engine include operations 1 to 4; the input data required for the execution of operations 1 to 4 include computing data and weight data; the output data includes result data; wherein the computing data includes the objects acted on by the network layer of each network node of the AI ​​model, and the weight data includes the model parameters of the network layer of the network node; it should be noted that in some scenarios, the intermediate outputs between adjacent operations can be directly transferred through the cache of the first cache area, for example Figure 3 Operation 2 in . Figure 4In the example, for the start time window (time window 0), the time sequence composed of each first time window (time window 1 to 4) and the end time window (time window 5); in some embodiments, it is possible to choose to perform an associated second-type operation (such as a data move-in operation, i.e., move-in operation 1 calculation data, operation 2 weight data, and operation 3 weight data) in the previous time window of the first time window corresponding to each first-type operation, and choose to perform another associated second-type operation (such as a data move-out operation, i.e., move-out operation 4 result data) in the next time window. Although such a timing arrangement reduces the execution delay to a certain extent, it has poor flexibility and may cause memory pressure in the first cache area at different time windows, resulting in the failure of the second engine to execute.

[0071] In view of this, the embodiments of the present disclosure can, for each first-category operation, use a first time window corresponding to another first-category operation different from the first-category operation, or a start time window in the time sequence for starting a memory operation sequence, or an end time window for ending the execution of a memory operation sequence, as a window in which a second-category operation associated with the first-category operation can be selected for execution, that is, determine at least one corresponding candidate time window; wherein, since time compression cannot be performed if the associated first-category operation and the second-category operation are executed in the same time window, it is set that the associated first-category operation and the second-category operation are not executed in the same time window, that is, the candidate time window of the second-category operation is different from the first time window of the first-category operation associated with the same second-category operation.

[0072] In step 102, based on the test results of executing each memory operation sequence, or the historical execution results, the time required for the first engine to execute the first type of operation in each memory operation sequence is determined; according to the first time window corresponding to each first type of operation and the execution time required, the first estimated time corresponding to each time window in the time series is obtained.

[0073] Among them, the time required to execute the first type of operation is expressed as the number of clock cycles of the first engine using the memory resources of the first cache area to execute the first type of operation; if the time window is the first time window, then the value of the first estimated duration of the time window is expressed as the estimated value of the time required to execute the first type of operation within the time window; if the time window is not the first time window, the first type of operation is not executed within the time window, and at this time the value of the first estimated duration of the time window is 0.

[0074] In step 103, based on the test results of executing each memory operation sequence, or the historical execution results, the time required for the second engine to execute the second type of operation associated with the first type of operation in each memory operation sequence is determined; based on at least one candidate time window corresponding to each first type of operation and the execution time required, the second estimated time corresponding to each time window in the time series is obtained.

[0075] Among them, the duration required for executing the second type of operation is expressed as the number of clock cycles for the second engine to execute the second type of operation using the memory resources of the first cache area; if the time window is a candidate time window corresponding to at least one second type of operation, then the value of the second estimated duration of the time window is expressed as the sum of the values ​​of the duration required for executing all second type operations that may be executed within the time window; if the time window is not a candidate time window corresponding to any second type of operation, then no second type of operation is executed within the time window, and at this time the value of the second estimated duration of the time window is 0.

[0076] It should be noted that the first estimated duration and the second estimated duration of each time window of the time series counted in step 102 and step 103 are both estimated for one execution process of a memory operation sequence.

[0077] In step 104, since there is no need to deduplicate the second-category operations for the same first-category operation, that is, in actual implementation, each second-category operation is executed by the second engine in only one time window of at least one corresponding candidate time window; so when each second-category operation selects a different candidate time window to execute, the disclosed embodiment can obtain the total execution delay of the memory operation sequence under different execution conditions based on the first estimated duration and the second estimated duration of each time window in the time series; wherein the total execution delay of the memory operation sequence is obtained by adding up the delays corresponding to each time window when the memory operation sequence is completely executed.

[0078] Among them, the delay of each time window is the absolute value of the difference between the first estimated duration and the second estimated duration of each time window, which represents the overlapping part of the execution time when the first type of operation and the second type of operation that may be executed in the corresponding time window are executed; these overlapping parts are the parts that cannot be hidden in the time window, that is, the unavoidable delays generated when the asynchronous engine is executed in the same corresponding time window; therefore, the sum of the delays of each time window can represent the total execution delay of the memory operation sequence.

[0079] In the embodiment of the present disclosure, because the total execution delay of the memory operation sequence varies when each second-category operation selects a different candidate time window to execute, if the total execution delay is minimized, it can be understood that the execution time of the asynchronous engine has achieved the greatest degree of hiding and compression in the entire timing; therefore, the embodiment of the present disclosure uses the candidate time window corresponding to each second-category operation when the total delay is minimized as the second time window for each second-category operation.

[0080] It should be noted that the embodiments of the present disclosure can use mathematical models such as data analysis and linear programming to directly analyze and evaluate the total execution delay of the memory operation sequence, so as to determine the second time window corresponding to each second-category operation; the embodiments of the present disclosure can also use the hypothesis-verification method to separately solve the total execution delay of various situations, and verify the accuracy of the candidate time window selected for each second-category operation as the second time window.

[0081] In step 105, based on the first time window of each first-category operation in the memory operation sequence and the second time window determined for each second-category operation in step 104, the execution timing information of executing the memory operation sequence using the asynchronous first engine and the second engine can be determined; wherein the execution timing information is used as a timing arrangement result to indicate the first-category operation and / or second-category operation that needs to be executed within each time window of the time series.

[0082] In the timing arrangement method proposed in the embodiment of the present disclosure, the second time window corresponding to each second-class operation when the total execution delay of the memory operation sequence is minimized is determined based on the time required for the first engine to execute each first-class operation to be executed in the memory operation sequence, the time required for the second engine to execute each second-class operation associated with the first-class operation, the first time window corresponding to each first-class operation, and at least one candidate time window that can be selected for each second-class operation; and the execution timing information for timing the memory operation sequence is obtained based on the first time window corresponding to each first-class operation and the second time window corresponding to each second-class operation. In this way, when determining the second time window of the second type of operation associated with the first type of operation, compared to the method of selecting a time window as the second time window of the second type of operation according to the first time window of the first type of operation and the timing relationship between the first type of operation and the second type of operation, the embodiment of the present disclosure first determines multiple candidate time windows of the second type of operation from multiple time windows of the time series, and determines the second time window from multiple candidate time windows with the goal of minimizing the total execution delay of the memory operation sequence; on the one hand, the timing arrangement of the second type of operation can be more flexible; on the other hand, because the total execution delay of the memory operation sequence when executing the second type of operation in different candidate time windows is taken into account when determining the second time window, the total execution delay of the memory operation sequence can be minimized to the greatest extent, thereby improving the speed and efficiency of the computer system in executing the memory operation sequence.

[0083] In some embodiments, determining the second estimated duration of each time window in the time series according to at least one candidate time window of each second-category operation includes:

[0084] According to at least one candidate time window of each second-category operation, a plurality of decision variables are set; wherein each decision variable corresponds to a candidate time window of a second-category operation; and different values ​​of the decision variables are used to indicate whether the second-category operation is executed by the second engine within the corresponding candidate time window;

[0085] Determine a second estimated duration for each time window according to at least one second-type operation expected to be performed in each time window in the time series, a decision variable corresponding to the second-type operation, and an execution duration required to perform the second-type operation;

[0086] The above-mentioned determining, according to the first estimated duration and the second estimated duration of each time window, the second time window corresponding to each second-category operation from at least one candidate time window of each second-category operation includes:

[0087] Determine a total execution delay corresponding to the memory operation sequence according to a first estimated duration and a second estimated duration of each time window in the time series;

[0088] Minimizing the total execution delay corresponding to the memory operation sequence is determined as the objective function;

[0089] According to the constraints and objective functions of the multiple decision variables, target values ​​of the multiple decision variables are determined; wherein the target values ​​of the decision variables are used to determine a second time window of the second type of operation corresponding to the decision variables.

[0090] Here, for each candidate time window in which the second type of operation can be selected for execution, a corresponding decision variable is set; these decision variables have two different, selectable values; illustratively, the value of the decision variable corresponding to each second type of operation can be 1 or 0; wherein, when the value of the decision variable is 1, it indicates that the second engine needs to execute the second type of operation corresponding to the decision variable within the candidate time window corresponding to the decision variable; when the value of the decision variable is 0, it indicates that the second type of operation corresponding to the decision variable does not need to be executed within the corresponding candidate time window.

[0091] After setting the decision variables, based on at least one second-class operation expected to be executed in each time window and the time required to execute each second-class operation, first obtain the product of each decision variable of each second-class operation in each time window and the time required to execute the corresponding second-class operation, and sum the products to obtain the summed results; sum the summed results of each second-class operation in the time window to obtain the second estimated time for each time window; and sum the delays between the first estimated time and the second estimated time in each time window of the time series to obtain the total execution delay of the memory operation sequence.

[0092] It is understandable that, because the second estimated duration is determined by the decision variables, the total execution delay of the memory operation sequence is determined by the second estimated duration of each time window, so the value of the total execution delay is different according to the different values ​​of the decision variables. In this way, the embodiment of the present disclosure takes minimizing the total execution delay as the goal, determines the objective function, and uses the constraints for multiple decision variables as restrictions to constrain the objective function to a linear problem, thereby obtaining a linear programming model; and performs linear analysis processing on the linear programming model through OR tools (OR-Tools) in related technologies, etc., to determine the target value of each decision variable when the function value of the objective function is the minimum value.

[0093] Here, in at least one decision variable of each second type of operation, the candidate time window corresponding to the target value of 1 is used as the second time window of the second type of operation.

[0094] It should be noted that the constraints on multiple decision variables may be timing constraints, memory resource constraints or environmental constraints on the execution of the first type of operations and the associated second type of operations, etc.; exemplarily, the timing constraint may be a restriction on the execution time of the second type of operations, for example, some second type of operations must be selected to be executed in a certain candidate time window, then the value of the decision variable of the candidate time window is set to 1, and the value of other existing candidate time windows is set to 0; the memory resource constraint may be a constraint on whether the second type of operation can be executed within the candidate time window based on whether the memory resources are sufficient, characterized as a constraint on the variable value corresponding to the candidate time window; the environmental constraint may be a restriction on the number / type of operations that can be executed within the time window in combination with the resource occupancy of other executing processes / threads in the computer system or the system power consumption, characterized as a constraint on the variable value corresponding to the candidate time window; of course, the specific schemes of the above-mentioned timing constraints, memory resource constraints and environmental constraints are not limited to the above-mentioned examples. In actual scenarios, there are naturally other actual constraints, and the embodiments of the present disclosure will not be further elaborated here.

[0095] In this way, the embodiment of the present disclosure sets multiple decision variables for each second-category operation, and can convert the analysis of the total execution delay of the memory operation sequence into a linear problem for solving the objective function, which can improve the rationality and accuracy of determining the second time window for the second-category operation; and, compared with the solution of calculating the minimum value of the total execution delay under different circumstances to determine each second time window, it saves calculation time, simplifies the calculation logic, and effectively improves the efficiency of timing arrangement for the memory operation sequence.

[0096] In some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0097] Determining a first constraint condition according to a constraint relationship between decision variables corresponding to at least one candidate time window of each second-type operation;

[0098] For each time window of the time series, determining a first estimated memory amount in the first cache area for a first type of operation corresponding to the first engine execution time window;

[0099] Determine, according to the amount of memory occupied by the operation data associated with each second-type operation and at least one decision variable corresponding to the second-type operation, a second estimated amount of memory occupied in the first cache area by the operation data associated with at least one second-type operation expected to be executed within each time window;

[0100] Determine a second constraint condition according to the first estimated amount of memory and the second estimated amount of memory in each time window, and the amount of memory in the first buffer area; the second constraint condition limits the sum of the first estimated amount of memory and the second estimated amount of memory in each time window to be less than the amount of memory in the first buffer area;

[0101] The above method determines the target values ​​of multiple decision variables based on the constraints and objective functions of multiple decision variables, including:

[0102] Based on the objective function and the first constraint and the second constraint, target values ​​of multiple decision variables are determined.

[0103] Here, through the timing relationship between the second-class operation and the associated first-class operation, as well as the execution requirements of the second-class operation, a first constraint condition can be set for each value of multiple decision variables of each second-class operation; wherein the first constraint condition can remove some free values ​​of multiple decision variables, and constrain the value of one decision variable among multiple decision variables, or multiple decision values.

[0104] In some examples, the first constraint condition can constrain the value of a decision variable; for example, when the second type of operation is not allowed to be executed in certain specific time windows, the decision variable of any second type of operation that may exist in this time window is set to 0; for another example, if the first type of operation is the first memory operation in a memory operation sequence, and the associated second type of operation is an input data move-in operation for the first memory operation, then the decision variable value of the data move-in operation within the start time window is set to 1; for another example, if the first type of operation is the last memory operation in the memory operation sequence, and the associated second type of operation is an operation to move the result data out of the last memory operation, then the decision variable value of the data move-out operation within the end time window is set to 1.

[0105] In other examples, the first constraint condition can also constrain multiple decision variables of a second-category operation at the same time; for example, when there are multiple candidate time windows for a second-category operation, the values ​​of the decision variables of a first preset number of candidate time windows corresponding to the first time window away from the associated first-category operation can be set to 0; wherein the first preset number can be 1 / 2 or 2 / 3 of the number of all candidate time windows corresponding to the second-category operation. Of course, the first preset number can also be other preset values, which are not limited by the present disclosure.

[0106] In the embodiment of the present disclosure, in addition to the above-mentioned first constraint condition, the embodiment of the present disclosure also determines a second constraint condition for the values ​​of multiple decision variables based on the memory capacity of the first cache area.

[0107] Since the first engine executing the first type of operations and the second engine executing the second type of operations both need to utilize the memory resources in the on-chip cache, the amount of memory occupied by executing the first type of operations and / or the second type of operations in each time window can be obtained through historical execution and testing; and the amount of memory occupied by executing the first type of operations and / or the second type of operations in each time window cannot exceed the total amount of memory in the first cache area in the time window. In this way, through the constraint of the amount of memory, the second type of operations that cannot be executed in some time windows can be constrained, that is, the value of the decision variable can be constrained.

[0108] Specifically, the embodiment of the present disclosure first estimates the first estimated memory amount of each time window of the time series based on the amount of memory in the first cache area when the first engine executes the first type of operation; and then determines the second estimated memory amount of each memory operation according to the amount of memory occupied by the operation data associated with each second type of operation, the type of the second type of operation, and at least one associated decision variable.

[0109] It should be noted that different types of second-class operations have different associated operation data; illustratively, if the second-class operation is a data-moving-in operation for the first-class operation, the associated operation data is the input data required to execute the first-class operation; if the second-class operation is a data-moving-out operation for the first-class operation, the associated operation data is the result data after the first-class operation is executed. The time window selected for the execution of the data-moving-in operation determines that the input data will start to generate memory occupancy from this time window, and the time window selected for the execution of the data-moving-out operation determines that the result data will have memory occupancy in the time window between this time window and the first time window corresponding to the first-class operation; therefore, for the second estimated memory amount in each time window, when the type of the second-class operation expected to be executed in the time window is different, it is jointly determined by the decision variables corresponding to the time window and the decision variables and operation data of the previous (or subsequent) candidate time window corresponding to the same second-class operation.

[0110] In this way, the total memory occupancy generated in each time window is determined by the first estimated memory amount and the second estimated memory amount in each time window, and the sum of the first estimated memory amount and the second estimated memory amount in each time window is less than the memory amount of the first cache area as the second constraint condition; the second constraint condition can count and constrain the amount of memory that may appear in each time window, thereby achieving restrictions on the values ​​of each decision variable.

[0111] Therefore, through the first constraint condition and the second constraint condition, the embodiment of the present disclosure can perform linear constraints on the values ​​of each decision variable in the objective function, thereby determining a more reasonable and actual target value for each decision variable.

[0112] In some embodiments, the above-mentioned determining at least one candidate time window of the second type operation associated with each first type operation from each time window of the time sequence according to the first time window occupied by each first type operation to be executed in the memory operation sequence in each time window of the time sequence includes:

[0113] According to the first time windows of each first-category operation to be executed in the memory operation sequence, at least one time window earlier than the first time window in the time sequence is determined as at least one candidate time window of a first sub-operation associated with the first-category operation; wherein the execution timing of the first sub-operation is earlier than the execution timing of the first-category operation associated with the first sub-operation;

[0114] According to the first time windows of the first type of operations, at least one time window later than the first time window in the time sequence is determined as at least one candidate time window of the second sub-operation associated with the first type of operation; the execution timing of the second sub-operation is later than the execution timing of the first type of operation associated with the second sub-operation;

[0115] The decision variables include: a first variable corresponding to the first sub-operation, and a second variable corresponding to the second sub-operation.

[0116] Here, the second type of operation associated with the first type of operation includes a first sub-operation and a second sub-operation.

[0117] In an example of a scenario of the present disclosure, when the memory operation sequence is applied to the AI ​​model training scenario, the first engine is a forward and backward reasoning engine, and the second engine is a parameter synchronization communication engine between nodes; at this time, the first type of operation includes calculation operations in two propagation directions (forward and backward); at this time, for the forward calculation operation, the first sub-operation corresponding to the second type of operation can be a weight synchronization operation of the communication engine on the weight data corresponding to the network node; at this time, because the forward calculation of the network node can only be performed after the weight update synchronization, the embodiment of the present disclosure can determine at least one time window with a timing earlier than the first time window corresponding to the forward calculation operation as at least one candidate time window for the above-mentioned weight synchronization operation; in addition, for the backward calculation operation (i.e., the backward gradient momentum parameter update operation), the second sub-operation corresponding to the second type of operation can be a gradient synchronization operation of the gradient momentum parameter by the communication engine; at this time, because the gradient momentum parameter can only be updated after calculation, the embodiment of the present disclosure can determine at least one time window with a timing later than the first time window of the backward calculation operation as at least one candidate time window for the above-mentioned gradient synchronization operation.

[0118] Correspondingly, the decision variables set in the embodiment of the present disclosure may include a first variable corresponding to the first sub-operation (for example, the first variable corresponding to the above-mentioned weight synchronization operation) and a second variable corresponding to the second sub-operation (for example, the second variable corresponding to the above-mentioned gradient synchronization operation).

[0119] In an example of another scenario of the present disclosure, when the memory operation sequence is applied to an AI model application scenario, the first engine is a computing engine and the second engine is a DMA engine. Continuing with the above example, the first type of operation can be a convolution operation, a pooling operation, a nonlinear mapping operation, etc.; the first sub-operation of the second type of operation is a data move-in operation for the input data of the first type of operation, and the second sub-operation of the second type of operation is a data move-out operation for the result data of the first type of operation.

[0120] Here, because the first type of operation needs to be executed by the first engine only when the input data is moved in, the embodiment of the present disclosure determines at least one time window that is earlier than the first time window of the first type of operation as at least one candidate time window for the data moving-in operation; in addition, because the result data is generated only after the first type of operation is executed, the embodiment of the present disclosure determines at least one time window that is later than the first time window of the first type of operation as at least one candidate time window for the data moving-out operation.

[0121] Correspondingly, the decision variables set in the embodiment of the present disclosure may include a first variable corresponding to the first sub-operation (for example, the moving-in variable corresponding to the above-mentioned data moving-in operation) and a second variable corresponding to the second sub-operation (for example, the moving-out variable corresponding to the above-mentioned data moving-out operation).

[0122] It should be noted that the embodiments of the present disclosure are suitable for the examples of the above two scenarios with respect to the first sub-operation, the second sub-operation, the decision variables, the constraints for setting the variables, and the steps for solving the objective function; the present disclosure will hereinafter introduce and illustrate an example of the second scenario (an example in which the asynchronous engine is a computing engine and a DMA engine, the first sub-operation is a data moving-in operation, and the second sub-operation is a data moving-out operation), and the steps and processes of the first example are equally applicable, and will not be elaborated on in the present disclosure hereinafter.

[0123] The disclosed embodiment considers the temporal association between different sub-operations of the second category of operations and the first category of operations, sets the candidate time window corresponding to the first sub-operation to be earlier than the first time window of the associated first category of operations, and sets the candidate time window corresponding to the second sub-operation to be later than the first time window of the associated first category of operations, thereby further making the design of decision variables more reasonable and improving the scientificity and accuracy of the timing arrangement.

[0124] In some embodiments, the first constraint condition is determined according to the constraint relationship between the decision variables corresponding to at least one candidate time window of each second type operation, including:

[0125] Determine a first sum value between first variables corresponding to each candidate time window of each first sub-operation;

[0126] The first sum value between the first variables of each first sub-operation is set to be equal to the first value, so as to obtain a first constraint relationship; the first constraint relationship is used to restrict the second engine from executing the first sub-operation in one candidate time window of at least one candidate time window of the first sub-operation;

[0127] Determine a second sum value between second variables corresponding to each candidate time window of each second sub-operation;

[0128] The second sum value between the second variables of each second sub-operation is set to be equal to the first value, so as to obtain a second constraint relationship; the second constraint relationship is used to restrict the second engine to execute the second sub-operation in one candidate time window of at least one candidate time window of the second sub-operation;

[0129] A first constraint condition is determined according to the first constraint relationship and the second constraint relationship.

[0130] For the example of the second scenario above, the first sub-operation is a data moving-in operation, and the second sub-operation is a data moving-out operation, and the above-mentioned first value can be 1; in this way, the embodiment of the present disclosure sets the first sum value between the first variables corresponding to each candidate time window of each data moving-in operation to be equal to the first value, so that in a data moving-in operation corresponding to a first-category operation, only one first variable has a value of 1, and the values ​​of other first variables are all 0, that is, a data moving-in operation of a first-category operation only selects one time window to execute; similarly, the present disclosure sets the second sum value between the second variables corresponding to each candidate time window of each data moving-out operation to be equal to the first value, so that in a data moving-out operation corresponding to a first-category operation, only one first variable has a value of 1, and the values ​​of other first variables are all 0, that is, a data moving-out operation of a first-category operation only selects one time window to execute.

[0131] It should be noted that the data moving operation corresponding to a first type of operation can be one or more, and the different input data (such as calculation data and weight data) required for the first type of operation are moved into the first buffer area. In this way, the embodiment of the present disclosure can set one or more first variables (corresponding to one or more data moving operations respectively) for each first sub-operation, and the first constraint relationship for at least one first variable of the same first sub-operation is the same as above, and the present disclosure will not be further elaborated here.

[0132] The embodiment of the present disclosure provides the above-mentioned first constraint relationship and second constraint relationship, so as to constrain the execution of the first sub-operation and the second sub-operation, thereby improving the rationality and accuracy of the timing arrangement proposed in the embodiment of the present disclosure.

[0133] In some embodiments, the determining, based on the amount of memory occupied by the operation data associated with each second-type operation and at least one decision variable corresponding to the second-type operation, a second estimated amount of memory occupied in the first cache area by the operation data associated with at least one second-type operation expected to be executed within each time window includes:

[0134] Determine at least one first target variable of the first sub-operation in the i-th time window from a plurality of first variables associated with the first sub-operation expected to be performed in the i-th time window; the time sequence of the candidate time window corresponding to the at least one first target variable is less than or equal to i; i is greater than or equal to 1 and less than or equal to N, where N is the number of time windows included in the time sequence;

[0135] Determine at least one second target variable of the second sub-operation in the i-th time window from a plurality of second variables associated with the second sub-operation expected to be executed in the i-th time window; the timing of the candidate time window corresponding to the at least one second target variable is greater than or equal to i;

[0136] Based on the amount of memory occupied by at least one first target variable of a first sub-operation and the amount of memory occupied by operation data associated with the first sub-operation in the i-th time window, and the amount of memory occupied by at least one second target variable of a second sub-operation and the operation data associated with the second sub-operation in the i-th time window, determine a second estimated amount of memory occupied in the first cache area by operation data associated with at least one second-type operation expected to be executed in the i-th time window.

[0137] In conjunction with the example of the second scenario described above, the first sub-operation is a data moving-in operation, and the second sub-operation is a data moving-out operation; wherein, since the time window selected for the execution of the data moving-in operation determines that the input data will start to generate memory occupancy from the time window, if there is a data moving-in operation that is expected to be executed in the i-th time window, then among the first variables corresponding to the data moving-in operation, the first variable corresponding to the i-th time window and the first variable in each candidate time window corresponding to the same data moving-in operation in the previous sequence are all used as the first target variable. In this way, the embodiment of the present disclosure can multiply the amount of memory occupied by the operation data associated with the data moving-in operation according to the sum of at least one first target variable corresponding to each data moving-in operation in the i-th time window, and obtain the product result; and use the sum of the product results corresponding to all possible data moving-in operations in the i-th time window as the first sub-estimated memory amount of the data moving-in operation in the i-th time window.

[0138] Similarly, since the time window in which the data moving-out operation is selected for execution determines that the result data has memory occupancy in the time window between the time window and the first time window corresponding to the first type of operation, if there is a data moving-out operation that is expected to be executed in the i-th time window, then among the second variables corresponding to the data moving-out operation, the second variable corresponding to the i-th time window and the second variable corresponding to each candidate time window of the same data moving-out operation that is later in time sequence are used as the second target variable. In this way, the disclosed embodiment can multiply the amount of memory occupied by the operation data associated with the data moving-out operation according to the sum of at least one second target variable corresponding to each output moving-out operation in the i-th time window by the sum of ...

[0139] Furthermore, based on the sum of the first sub-estimated memory amount and the second sub-estimated memory amount of the i-th time window, the second estimated memory amount occupied in the first cache area by operation data associated with at least one second type of operation expected to be performed within the i-th time window is determined.

[0140] Here, the embodiment of the present disclosure determines the second estimated amount of memory that may be generated in each time window due to executing the second type of operation by analyzing the first sub-operation and the second sub-operation that may be executed in each time window, thereby improving the comprehensiveness and accuracy of determining the second estimated amount of memory.

[0141] In the following, in conjunction with the example of the second scenario described above, an exemplary description is given of configuring the first variable and the second variable, setting the first constraint condition and the second constraint condition, and configuring the objective function in the embodiment of the present disclosure.

[0142] See also Figure 5 , Figure 5 The schematic diagram of configuring the first variable and the second variable for a memory operation sequence according to an exemplary embodiment is shown; wherein each first type of operation of the memory operation sequence includes operation 1 to operation 3; time window 0 is a start time window, time windows 1 to 3 are first time windows corresponding to operations 1 to 3 respectively, and time window 4 is an end time window. The first variable configured in the embodiment of the present disclosure includes Figure 5 The shown x1 to x4, the second variable of the configuration includes x5 to x8.

[0143] Among them, x1 corresponds to time window 0, and is the first variable of the data move-in operation corresponding to the calculation data of operation 1, and time window 0 is the candidate time window corresponding to x1; x2 corresponds to time window 0, x3 corresponds to time window 1, and x4 corresponds to time window 2. x2, x3 and x4 are three first variables of the data move-in operation corresponding to the weight data of operation 3, and time window 0, time window 1 and time window 2 are three candidate time windows for the data move-in operation corresponding to the weight data of operation 3 respectively; x5 corresponds to time window 2, x6 corresponds to time window 3, and x7 corresponds to time window 4. x5, x6 and x7 are three second variables of the data move-out operation corresponding to the result data of operation 1, and time window 2, time window 3 and time window 4 are three candidate time windows for the data move-out operation corresponding to the result data of operation 1 respectively; x8 corresponds to time window 4, and is the second variable of the data move-out operation corresponding to the result data of operation 3, and time window 4 is the candidate time window corresponding to x8.

[0144] It should be noted that, because the input data required for the first type of operations corresponding to different network nodes are different, for example, operation 1 does not require weight data, operation 2 can use the cache transfer in the first cache area to take part of the input data (or result data) residing in operation 1 as input data, and the result data of execution can also be used as input data for operation 3. Therefore, the variables of the move-in operation configured for operation 1 only involve the moving-in of calculation data, and operation 2 is not configured with the first variable and the second variable; in actual implementation, the embodiment of the present disclosure may also default to configuring the first variable for calculation data and weight data for both operation 1 and operation 2, and configuring the second variable for operation 2, and then setting the values ​​of these first variables / second variables to 0 by adding other constraint relationships to the first constraint condition.

[0145] It should also be noted that the embodiments of the present disclosure may also selectively set the above-mentioned first variable and second variable; for example, time window 0 and time window 4 do not have the first type of operation performed, so time hiding cannot be achieved by executing the data move-in operation. Therefore, the embodiments of the present disclosure may not define x2 and x7 to reduce the number of variables and simplify the constraints and objective functions.

[0146] Of course, the above Figure 5 This is only a simple example. Other situations can be designed according to the execution requirements of each first-category operation in the actual AI model, and the embodiments of the present disclosure will not be elaborated here.

[0147] For the above-mentioned first variable and second variable, the embodiment of the present disclosure also sets a first constraint condition and a second constraint condition. Among them, the first constraint condition includes a first constraint relationship and a second constraint relationship; the first constraint relationship can be set to: x2+x3+x4==1, indicating that the weight data of operation 3 must be moved in one of the three time windows from time window 0 to 2; the second constraint relationship can be set to: x5+x6+x7==1, indicating that the result data of operation 1 must be moved out of one of the three time windows from time window 2 to 4. Of course, the first constraint condition can also include other constraints, for example: x1==1, indicating that the calculation data of operation 1 can only be moved in time window 0; for another example: x8==1 indicates that the result data of operation 3 can only be moved out in time window 4.

[0148] Here, for configuring the second constraint, see Figure 6 , Figure 6 is a schematic diagram of setting a second constraint condition according to an exemplary embodiment; wherein Figure 6 Each row of represents the composition of the amount of memory occupied by operations in a time window of the time series. Figure 6 The first column represents the first estimated memory amount A, Figure 6 The second and third columns of the table together represent the second estimated memory amount, where the second column represents the second sub-estimated memory amount B of the corresponding time window, and the third column represents the first sub-estimated memory amount C of the corresponding time window. Among them, op1_out represents the memory occupied by the operation data associated with the data-out operation corresponding to operation 1, that is, the memory occupied by the result data of operation 1; op3_weight represents the memory occupied by the operation data associated with the data-in operation corresponding to operation 3, that is, the memory occupied by the weight data in the input data of operation 3.

[0149] Here, the result data of the first type of operation will be moved out in the subsequent time window, so it will generate memory usage B in the first buffer area; Figure 6 Each B shown is the product of the sum of each second target variable of the data moving out operation in the corresponding time window and op1_out; it can be understood that the second target variables corresponding to time window 2 are x5, x6 and x7. No matter which second variable among x5, x6 and x7 has a value of 1, op1_out of operation 1 will have memory occupancy in time window 2. Therefore, determining each second target variable and summing each second target variable can express the continuation of the memory occupancy of op1_out in the first buffer in terms of timing; similarly, the weight data of the first type of operation will be moved in in the previous time window, so the memory occupancy C will be generated in the first buffer in advance; Figure 6Each C in the above is the product of the sum of the first target variables of the data moving operation in the corresponding time window and op3_weight. It can be understood that the first target variables corresponding to time window 2 are x2, x3 and x4. No matter which first variable among x2, x3 and x4 takes the value of 1, the op3_weight of operation 3 will have memory occupancy in time window 2. Therefore, determining each first target variable and summing up each first target variable can express the continuation of the memory occupancy of op3_weight in the first buffer area in terms of time series.

[0150] against Figure 6 In this example, the second constraint condition is that A+B+C of each row is less than the memory capacity of the first cache area (eg, 256KB).

[0151] See also Figure 7 , Figure 7 is a schematic diagram showing a principle of determining a first estimated duration and a second estimated duration according to an exemplary embodiment; wherein Figure 7 Each row in represents the execution time of each operation in a time window of the time series. Figure 7 The first column represents the first estimated duration M, Figure 7 The second column represents the second estimated duration T; the third column represents the absolute value of the difference between the first estimated duration M and the second estimated duration T, that is, the delay |MT| of the corresponding time window.

[0152] Here, the first estimated duration M is the number of clock cycles for executing the first type of operation in the corresponding time window, such as Figure 7 In the first column, time window 0 has no first-category operation, so the first estimated duration is 0, the first estimated duration of time window 1 is 1200 of operation 1, the first estimated duration of time window 2 is 2300 of operation 2, the first estimated duration of time window 3 is 500 of operation 3, and time window 4 has no first-category operation, so the first estimated duration is 0.

[0153] For each time window, the data moving-in operation and data moving-out operation that may be executed are different, but the memory amount of the operation data of each data moving-in operation and data moving-out operation can be estimated and set. The second estimated duration T is the sum of the number of clock cycles of each second type of operation in the corresponding time window and the product of the corresponding first variable, which represents the sum of the number of clock cycles of each second type of operation that may occur in the time window; wherein, Figure 7 In the example, the clock cycle number of the calculation data of the operation 1 is 1300, the clock cycle number of the weight data of the operation 2 is 1200, the clock cycle number of the result data of the operation 1 is 2400, and the clock cycle number of the result data of the operation 3 is 1600.

[0154] here, Figure 7 The third column can represent the part of each time window that cannot be hidden from each other in the timing, namely, the delay |MT|. The embodiment of the present disclosure adds up the |MT| of each time window to obtain the total execution delay of the memory operation sequence; the objective function is obtained with the goal of minimizing the total execution delay. In this way, the execution times of the asynchronous computing engine and the DMA engine can be hidden from each other as much as possible, so that the entire delay of executing the memory operation sequence is minimized.

[0155] Here, the linear programming tools such as OR-Tools in the disclosed embodiment combine the above-mentioned first constraint condition and second constraint condition to perform linear analysis on the objective function to obtain the target values ​​of each first variable and second variable when the objective function achieves the minimum function value.

[0156] In some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0157] In a case where there are at least two target operations that are not adjacent in time sequence among the multiple first-category operations in the memory operation sequence, and result data of the target operation that is earlier in time sequence among the at least two target operations is used as input data of the target operation that is later in time sequence, determining a candidate time window of a third-category operation associated with the target operation according to at least one time window between the first time windows occupied by each of the at least two target operations in the time sequence;

[0158] A third variable is set according to the candidate time window of the third type of operation associated with the target operation; wherein each third variable corresponds to a candidate time window of a third type of operation; different values ​​of the third variable are used to indicate whether the result data of the target operation that is ahead in timing within the candidate time window of the third type of operation is stored in the first cache area.

[0159] Here, the result data of the target operation that comes earlier in time among at least two target operations is used as the input data of the target operation that comes later in time, which can be understood as the target operation that comes earlier in time being the producer and the target operation that comes later in time being the remote consumer; here, this relationship between target operations is often used in AI models with residual connection network fragments.

[0160] For example, see Figure 8 , Figure 8 is a topological diagram of a memory operation sequence according to an exemplary embodiment. Figure 1 Among them, in the AI ​​model with residual connection network fragments, the memory operations corresponding to some network nodes have a producer-consumer relationship, that is, the previous producer outputs the intermediate activation to the first buffer area, and then the subsequent consumer operation directly uses the activation without the need to move the result data out and in. Figure 8As shown, operations 1 to 7 are first-class operations of the memory operation sequence. After the output of operation 1 is used by the adjacent consumer operation 2, it also needs to be used by the remote consumer operation 7. Here, operation 1 and operation 7 can be used as two non-adjacent target operations in the embodiment of the present disclosure.

[0161] For such target operations that are not adjacent in time sequence, generally, after operation 2 is executed with the result data of operation 1 as input data, the result data needs to be released. However, moving the result data out of the first cache and moving it back in before operation 7 is executed will extend the execution time and generate excess power consumption. Therefore, the result data can be kept in the first cache until it is released after operation 7 is used. However, if the first cache has a tight memory situation, then in this process, the residence of the result data will generate great memory pressure on the first cache.

[0162] Based on this, the embodiment of the present disclosure proposes a third type of operation, determines the candidate time windows of the third type of operation, and sets a corresponding third variable in each candidate time window of the third type of operation associated with the target operation; wherein the third type of operation indicates that the result data of the producer operation resides in the first cache area, the candidate time windows of the third type of operation are the time windows between the first time window of the target operation that is earlier in timing and the first time window of the target operation that is later in timing, and the value of the third variable can be a first value (such as 1) or a third value (such as 0), so that if the value is 1, the third variable can indicate that the result data of the target operation that is earlier in timing within the candidate time window of the third type of operation is resided in the first cache area, and if the value is 0, the third variable can indicate that the result data of the target operation that is earlier in timing within the candidate time window of the third type of operation does not reside in the first cache area.

[0163] It should be noted that the same third variable is set in each candidate time window of the third type of operation associated with the same target operation, and the value of the same third variable is unified, either all the first value, for example, all 1, indicating that the result data always resides in the first cache area until the target operation with a later timing is executed; the value of the same third variable is either all the third value, for example, all 0, indicating that the result data does not reside in the first cache area.

[0164] The present disclosure can define whether the result data in non-adjacent target operations reside in the first cache area by setting a third variable, so that the residence problem can be analyzed as a variable value, so as to select a more optimized solution from the solution of retaining the result data and the solution of moving the result data out and then moving it back in through subsequent linear analysis.

[0165] In some embodiments, the first constraint condition is determined according to the constraint relationship between the decision variables corresponding to at least one candidate time window of each second type operation, including:

[0166] The first constraint condition is determined based on the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation; wherein the second target sub-operation is the second sub-operation associated with the target operation that is earlier in timing; and the first target sub-operation is the first sub-operation associated with the target operation that is later in timing.

[0167] Here, the second target sub-operation is the second sub-operation associated with the target operation that comes earlier in sequence, and the first target sub-operation is the first sub-operation associated with the target operation that comes later in sequence. Continuing with the example of the second scenario above, the second target sub-operation is a data-moving-out operation for the result data of the target operation that comes earlier in sequence, and the first target sub-operation is a data-moving-in operation for the result data.

[0168] In this way, the embodiment of the present disclosure can determine the candidate time window corresponding to the data move-out operation associated with the target operation that is earlier in time sequence, and the candidate time window corresponding to the data move-in operation associated with the target operation that is later in time sequence.

[0169] In some examples, the embodiments of the present disclosure may use the time windows between the first time window of the target operation that is earlier in sequence and the first time window of the target operation that is later in sequence as candidate time windows for the aforementioned data move-in operation and candidate time windows for the aforementioned data move-out operation; and determine the constraint relationship between the values ​​of the relevant variables based on the producer-consumer relationship, the logical relationship between the moved-out result data and the re-moved-in and resident result data, the first variables of the candidate time windows for the aforementioned data move-in operation and the second variables of the candidate time windows for the aforementioned data move-out operation.

[0170] For example, combined with Figure 8, operation 2 is a consumer adjacent to operation 1, so operation 2 needs to use the result data of operation 1 as calculation data. Therefore, the aforementioned data moving-out operation and data moving-in operation should not occur in time window 1. In this way, the disclosed embodiment can set the values ​​of the first variable and the second variable corresponding to this time window to 0 as a constraint relationship; in addition, since the result data can be used only after it is moved into the cache area, the disclosed embodiment can set the product of the first variable of the data moving-in operation for the result data and the timing value of the corresponding candidate time window to be greater than the product of the second variable of the data moving-out operation and the timing value of the corresponding candidate window as a constraint relationship; furthermore, because the third type of operation and the second type of operation will not all be executed between non-adjacent target operations, a constraint relationship can be determined according to this logic; in this way, by limiting the above-mentioned constraint relationships, a first constraint condition can be obtained, thereby limiting the values ​​of each first variable and second variable through the first constraint condition.

[0171] In other examples, the embodiments of the present disclosure may also select candidate time windows and determine corresponding first and second variables in each time window between the first time window of the target operation that comes earlier in sequence and the first time window of the target operation that comes later in sequence based on the producer-consumer relationship; for example, the first time window among these time windows is not used as a candidate time window for the aforementioned data-moving-in operation; and the last time window among these time windows is not used as a candidate time window for the aforementioned data-moving-out operation; in this way, the embodiments of the present disclosure may reduce the setting of decision variables and the setting of corresponding constraints, thereby simplifying the resulting linear programming model.

[0172] Here, the embodiment of the present disclosure can constrain the first variable and the second variable of the second type of operation associated with the above-mentioned non-adjacent target operations according to the third variable and the first constraint condition, so that a more optimized solution can be selected through linear analysis later from the solution of retaining the result data and the solution of moving out the result data and then moving it back in.

[0173] In some embodiments, the first constraint condition is determined according to the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation, including:

[0174] Determine a third sum value between second variables corresponding to each candidate time window of the second target sub-operation, and a third sum value between first variables corresponding to each candidate time window of the first target sub-operation;

[0175] The third sum value between the second variables corresponding to each candidate time window of the second target sub-operation is set to be equal to the fourth sum value between the first variables corresponding to each candidate time window of the first target sub-operation, to obtain a third constraint relationship; the third constraint relationship is used to restrict that when the second target sub-operation is executed in any candidate time window of the corresponding at least one candidate time window, the second engine needs to execute the first target sub-operation in any candidate time window of at least one candidate time window of the first target sub-operation;

[0176] The sum of the second variable and the third variable corresponding to each candidate time window of the second target sub-operation is set to be equal to the first value, so as to obtain a fourth constraint relationship; the fourth constraint relationship is used to restrict that when the second target sub-operation is executed in any candidate time window of the corresponding at least one candidate time window, the third type of operation associated with the target operation preceding the timing sequence does not need to be executed, or when the third type of operation associated with the target operation preceding the timing sequence is executed, the second target sub-operation does not need to be executed;

[0177] Determine a first parameter according to the product of the timing value corresponding to each candidate time window of the first target sub-operation and the first variable corresponding to the candidate time window, and the product of the timing value corresponding to each candidate time window of the second target sub-operation and the second variable corresponding to the candidate time window; the first parameter is used to indicate the execution time difference between the first target sub-operation and the second target sub-operation when the second engine executes the first target sub-operation and the second target sub-operation;

[0178] The fifth sum of the first parameter and the second parameter is set to be greater than or equal to the second value, thereby obtaining a fifth constraint relationship; the second parameter is the product of the second value and the third variable; the fifth constraint relationship is used to limit the execution time difference between the first target sub-operation and the second target sub-operation to be greater than the second value;

[0179] The first constraint condition is determined according to the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship.

[0180] Here, for the target operation with a later timing, if the second target sub-operation (such as the aforementioned data moving-out operation for the result data) is executed in any time window between the first time window of the target operation with an earlier timing and the first time window of the target operation with a later timing, then the first target sub-operation (the aforementioned data moving-in operation) must also be executed in a corresponding candidate time window; in this way, the result data of the target operation with an earlier timing can be used by the target operation with a later timing. Therefore, the embodiment of the present disclosure sets the third sum value between the second variables corresponding to each candidate time window of the aforementioned moving-out operation to be equal to the fourth sum value between the first variables corresponding to each candidate time window of the aforementioned moving-in operation, as the third constraint relationship, which is used to constrain the value relationship between the second variable and the first variable.

[0181] For an example following the second scenario above, see Fig. 9 , Fig. 9 is a schematic diagram of setting a third variable, a first variable of a first target sub-operation, and a second variable of a second target sub-operation according to an exemplary embodiment; wherein, in combination with the above Figure 8 and Fig. 9 In the example, operations 1 to 7 correspond to first-category operations, respectively, and time windows 0 to 6 correspond to first time windows of operations 1 to 7, respectively; here, operation 1 is a target operation in the front sequence, and operation 2 is a target operation in the back time window. The present disclosure sets the second variables (out0_store2 and out0_store3) of the aforementioned data moving-out operation in time window 2 and time window 3, sets the second variables (out0_load3, out0_load4 and out0_load5) of the aforementioned data moving-out operation in time window 3, time window 4 and time window 5, and sets the third variable out0_reside in time window 2 to time window 5.

[0182] Among them, in out0_store(n), n is the timing value of the candidate time window corresponding to the variable, such as out0_store2 is the second variable corresponding to time window 2 and related to the aforementioned data moving-out operation for the result data; similarly, in out0_load(m), m is also the timing value of the candidate time window corresponding to the variable, such as out0_store4 is the first variable corresponding to time window 4 and related to the aforementioned data moving-in operation for the result data; the interpretations of other variables are similar, and the present disclosure will not elaborate on them here.

[0183] according to Figure 8 and Fig. 9For example, the embodiment of the present disclosure takes out0_store2+out0_store3 as the third sum value, out0_load3+out0_load4+out0_load5 as the fourth sum value, and sets out0_store2+out0_store3==out0_load3+out0_load4+out0_load5 as the third constraint relationship.

[0184] In addition, for the above-mentioned non-adjacent target operations, because the target parameters with a later timing must use the result data, if the second target sub-operation (such as the aforementioned data moving-out operation for the result data) is performed in the time window between the first time window of the target operation with a previous timing and the first time window of the target operation with a later timing, there is no need to perform the third type of operation; if the third type of operation is performed, there is no need to perform the aforementioned data moving-out operation; that is, the result data either resides in the first cache area, or is moved out of the first cache area by the second engine and moved back in. In this way, the embodiment of the present disclosure sets the sum of each second variable corresponding to each candidate time window of the aforementioned moving-out operation and a third variable to the first value, as the third constraint relationship, which is used to constrain the value relationship between the second variable and the first variable. Wherein, in combination with the above disclosure, the first value can be 1.

[0185] according to Figure 8 and Fig. 9 For example, the embodiment of the present disclosure sets out0_reside+out0_store2+out0_store3==1 as the fourth constraint relationship.

[0186] At the same time, since the moving-out and moving-in operations of the result data occur in two adjacent time windows, the memory resource shortage in the first cache area cannot be alleviated; considering the memory constraints of the first cache area, the embodiment of the present disclosure can further target the time window of the aforementioned data moving-in operation of the result data to be at least a second preset number of time windows later than the time window after the aforementioned data moving-out operation; the value of the second preset number is a second value, and the second value is greater than or equal to 2.

[0187] In this way, the embodiment of the present disclosure performs subtraction processing on the product between the timing value corresponding to each candidate time window of the first target sub-operation (such as the aforementioned data move-in operation for the result data) and the first variable corresponding to the candidate time window, and the product between the timing value corresponding to each candidate time window of the second target sub-operation (such as the aforementioned data move-out operation) and the second variable corresponding to the candidate time window to obtain a first parameter. In this way, the first parameter can reflect the number of time windows between the second engine executing the aforementioned data move-out operation and executing the aforementioned data move-in operation, that is, reflect the execution time difference; and because only one of the third type of operation and the aforementioned data move-out operation occurs, the embodiment of the present disclosure uses the product of the second value and the third variable as the second parameter, and sets the fifth sum of the first parameter and the second parameter to be greater than or equal to the second value to obtain the fifth constraint relationship, which can reflect that the aforementioned data move-in operation and the aforementioned data move-out operation for the result data are at least separated by two time windows, or that the third type of operation occurs within these interval time windows, that is, the residence of the result data occurs.

[0188] according to Figure 8 and Fig. 9 For example, the embodiment of the present disclosure takes (5*out0_load5+4*out0_load4+3*out0_load3)-(2*out0_store2+3*out0_store3) as the first parameter; takes 2*out0_reside as the second parameter; and takes (5*out0_load5+4*out0_load4+3*out0_load3)-(2*out0_store2+3*out0_store3)+2*out0_reside>=2 as the fifth constraint relationship.

[0189] In this way, the embodiment of the present disclosure can determine the first constraint condition through the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship, and obtain the target values ​​of the first variable and the second variable through the linear programming model obtained by the first constraint condition and the objective function. Correspondingly, the value of the third variable can also be determined according to the value of the first variable of the first target operation and the value of the second variable of the second target operation, and then a solution with a smaller total execution delay of the memory operation sequence can be obtained in the solution of retaining the result data and the solution of moving out the result data and then moving it back in.

[0190] It should be noted that if the value of the third variable is the first value (such as 1), the first target sub-operation and the second target sub-operation of the target operation will not occur, and the execution time of the second engine will not be generated. Therefore, the third type of operation itself does not affect the execution delay; therefore, the embodiment of the present disclosure still defines the objective function through the first estimated duration and the second estimated duration, and analyzes the objective function in combination with the first constraint condition determined by the above-mentioned third constraint relationship, the fourth constraint relationship and the fifth constraint relationship, so as to obtain the values ​​of the first variable and the second variable.

[0191] In some embodiments, determining the second constraint condition according to the first estimated memory amount and the second estimated memory amount in each time window and the memory amount of the first buffer area includes:

[0192] Determine, according to the product of the memory amount occupied by the result data of the target operation and the third variable, a third estimated memory amount occupied by the result data in the first buffer area in each candidate time window of the third type of operation associated with the target operation that precedes the timing;

[0193] Determine a second constraint condition according to the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount in each time window, and the memory amount of the first buffer area;

[0194] The second constraint condition is also used to limit the sum of the first estimated memory amount, the second estimated memory amount and the third estimated memory amount in each time window to be less than the memory amount of the first cache area.

[0195] Here, because the third type of operation represents the residence of the result data of the target operation that is preceding in sequence in the first cache area, the resident result data has a memory amount, which will cause memory occupancy in the first cache area; thus, the disclosed embodiment will use the product of the memory amount occupied by the result data of the target operation and the third variable as the third estimated memory amount occupied by the result data in the first cache area in each candidate time window of the third type of operation associated with the target operation that is preceding in sequence.

[0196] For example, according to Figure 8 and Fig. 9 In the example of , op1_out*out0_size is set as the third estimated memory amount D corresponding to each candidate time window of the third type of operation; here, op1_out is the memory amount occupied by the result data of operation 1, and out0_size is the third variable. Figure 6For example, the disclosed embodiment can use the sum of the first estimated memory amount A, the second estimated memory amount (the sum of the first sub-estimated memory amount C and the second sub-estimated memory amount B) and the third estimated memory amount D in each time window as the second constraint condition; wherein, if the corresponding third variable is not set in the time window, the value of the third estimated memory amount is 0.

[0197] Here, the embodiment of the present disclosure improves the second constraint condition in the scenario by considering the continuous occupation of the result data of the target operation in the first cache area due to the third type of operation, thereby improving the rationality of the obtained linear programming model, thereby making the determined decision variables and the third variable more accurate.

[0198] For the above-mentioned at least two non-adjacent target operations, in actual implementation, there is also a situation where the result data of the target operation in the previous sequence can be used as the input data of the at least two target operations in the subsequent sequence. Fig.10 , Fig.10 is a topological diagram of a memory operation sequence according to an exemplary embodiment. Figure 2 ;in, Fig.10 Operations 1 to 14 are all the first type of operations in the memory operation sequence. The operation number indicates the order in which the operations are executed, and the arrow indicates the producer-consumer relationship, that is, the result data of the operation sent by the arrow can be used as the input data of the operation pointed to by the arrow. Here, the result data of operation 1 can be used as the input data of operation 2, operation 6, and operation 10. Therefore, operation 1 and operation 6, operation 1 and operation 10 can all constitute the above-mentioned non-adjacent target operations; similarly, the result data of operation 5 can be used as the input data of operation 14. Therefore, operation 5 and operation 14 can constitute the above-mentioned non-adjacent target operations; the result data of operation 9 can be used as the input data of operation 14. Therefore, operation 9 and operation 14 can constitute the above-mentioned non-adjacent target operations.

[0199] Here, if the target operation in the previous sequence has a target operation in the subsequent sequence, such as the above-mentioned operation 5 and operation 14, then the embodiment of the present disclosure can set the corresponding third variable, the corresponding first constraint and the second constraint for operation 5 and operation 14, so as to obtain the second variable of the data moving-out operation of operation 5, the first variable of the data moving-in operation of operation 14, and the result of the corresponding third variable according to the minimization objective function, so as to make a more optimized choice in the residence scheme for the result data of operation 5 and the scheme of moving out first and then moving in. Similarly, for operation 9 and operation 14, they can also be set as above, and the embodiment of the present disclosure will not be repeated here.

[0200] In addition, if a target operation that is earlier in time sequence has at least two target operations that are later in time sequence (for example, operation 1 and operation 6, operation 10, then the embodiment of the present disclosure may set two variable groups, one variable group includes the second variable of the data moving-out operation of operation 1, the first variable of the data moving-in operation of operation 6, and the third variable in the time window between the time window corresponding to operation 2 and the time window corresponding to operation 5; the other variable group includes the second variable of the data moving-out operation of operation 1, the first variable of the data moving-in operation of operation 10, and the third variable in the time window between the time window corresponding to operation 7 and the time window corresponding to operation 9. Here, in each variable group, the variables in the group have the first constraint condition formed by the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship and the second constraint condition determined by the third estimated variable, the second estimated variable and the first estimated variable.

[0201] It should be noted that, in actual implementation, the embodiments of the present disclosure may also set constraints between the above-mentioned different variable groups. For example, the sum of the sum of the first variables of each variable group is a first value, and the first value is 1. This limits the result data of the producer operation to be moved out only once between the target operation that comes before the sequence (producer) and the last target operation (the farthest consumer) in the sequence. For another example, if the sum of the second variables of the previous variable group is 1, then a data move-out operation of the result data has occurred before the first consumer uses it. In order to reduce memory usage, the third variables of subsequent variable groups are all 0. In this way, when subsequent consumers use it, only the second engine is used to move the result data back in.

[0202] In some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0203] Determine, according to the execution timing information of the memory operation sequence, the first type of operation and / or the second type of operation to be executed in each time window in the time sequence;

[0204] In each time window, memory resources of the first buffer area are allocated for the first type of operation and / or the second type of operation to be executed in the time window;

[0205] After the first type of operation and / or the second type of operation is executed based on the allocated memory resources, at least part of the memory resources allocated to the first type of operation and / or the second type of operation is released.

[0206] Here, after obtaining the execution timing information, the first type of operation and / or the second type of operation that can be implemented in each time window of the time series can be determined. In this way, the embodiment of the present disclosure can allocate memory resources for the first type of operation when it is determined that the first type of operation needs to be executed in each time window, so that the first engine can execute the first type of operation based on the memory resources; and when the second type of operation needs to be executed, memory resources can be allocated for the second type of operation, so that the second engine can execute the second type of operation based on the memory resources.

[0207] It should be noted that the size of memory resources allocated for the first type of operation is the same as the amount of memory occupied by executing the first type of operation in the first cache area; the size of memory resources allocated for the second type of operation is the same as the amount of memory occupied by the operation data of the second type of operation in the first cache area.

[0208] When executing first-category operations and / or second-category operations, in order to ensure that the first cache area has sufficient resources for subsequent second-category operations and first-category operations, the embodiments of the present disclosure may release memory resources allocated to the currently executed first-category operations and / or second-category operations in the first cache area, except for data that needs to reside in the first cache area for sharing, and execute the next sequential first-category operation after the release is completed.

[0209] For an example following the second scenario above, see Fig.11 , Fig.11 is a schematic diagram of a memory management according to an exemplary embodiment; here, Fig.11 yes Figure 5 The values ​​of the first and second variables are shown as follows; the values ​​of the first variables x1 and x3 are 1, and the values ​​of the other first variables are 0; the values ​​of the second variables x6 and x8 are 1, and the values ​​of the other second variables are 0; here, the value of the first variable is 1, indicating that the DMA engine needs to move the calculation data or weight data of operation 1 to the first buffer area, and the value of the second variable is 1, indicating that the DMA engine needs to move the result data from the first buffer area. Here, Fig.11 It shows the memory resource allocation, memory occupancy and resource release of the first cache area from time window 0 to time window 4 when the values ​​of the first variables x1 and x3 are 1, the values ​​of other first variables are 0, the values ​​of the second variables x6 and x8 are 1, and the values ​​of other second variables are 0.

[0210] It should be noted that if the memory resource does not need to be resident, it will be released in the time window after use. For example, in time window 1, for the calculation data of operation 1 ( Fig.111 input in the time window), and is released immediately after use; if the memory resource needs to be resident, it is not released in the current time window, for example, the result data of operation 2 (op2 output) is not released in the second buffer; it should also be noted that the first buffer can also allocate temporary cache resources (op3buffer) for some special operations. For example, in time window 3, temporary cache resources are allocated for operation 3, and after operation 3 is calculated, it is released in time window 3.

[0211] In the disclosed embodiment, by adjusting the allocation and release of resources in the first cache area, the safe and effective execution of the memory operation sequence can be guaranteed.

[0212] In the embodiment of the present disclosure, due to the limitation of the second constraint condition mentioned above, the execution timing information of the memory operation obtained can determine the amount of memory resources allocated for the first type of operation and / or the second type of operation in each time window, and ensure that the amount of memory resources allocated does not exceed the total memory amount of the first cache area in the time window. However, in actual implementation, see Fig.12 , Fig.12 The memory resource distribution of the first cache area is shown according to an exemplary embodiment. Figure 1 ; The shaded areas 3000 and 1000 are the memory amounts for the allocated operating data, and the other areas are free memory; at this time, Fig.12 There is a gap between the storage location corresponding to the operation data with a memory size of 1000 and the memory resource operation data with a memory size of 3000, that is, the operation data with a memory size of 1000 is regarded as memory fragmentation. Therefore, if the memory operation to be executed within the time window needs to allocate a memory resource with a memory size of 4000, the total amount of free memory in the first cache area (3000 and 1000 in the blank area) is sufficient, but the memory resources cannot be allocated successfully.

[0213] In view of this, in order to solve the problem of memory resource allocation failure caused by the above-mentioned memory fragmentation problem, it is necessary to reasonably eliminate memory fragmentation or reduce the generation of memory fragmentation.

[0214] In some embodiments, the timing arrangement method proposed in the embodiments of the present disclosure further includes:

[0215] If the memory resource allocation for the first type of operation and / or the second type of operation to be executed in a third time window in the time sequence fails, inserting a fourth time window in the time sequence; the fourth time window is a time window before the third time window;

[0216] Determine a first storage location of first target data in the first cache area; wherein the first target data is operation data stored in the first cache area, and the first storage location of the first target data in the first cache area is not adjacent to a second storage location of operation data other than the first target data in the first cache area;

[0217] Moving the first target data from the first storage position of the first cache area to the third storage position of the first cache area within a fourth time window; the third storage position is an idle storage position adjacent to the second storage position among a plurality of idle storage positions of the first cache area;

[0218] After the first target data is moved, memory resources are reallocated for the first type of operations and / or the second type of operations to be executed within the third time window.

[0219] Here, for a third time window in a time series in which there are first-category operations and / or second-category operations, if there is a first cache area without a whole block of free memory, memory resources are allocated for the first-category operations and / or second-category operations, and it is necessary to first eliminate the memory fragments in the first cache area; at this time, a time window is inserted between the third time window and the adjacent previous time window as the fourth time window, and the storage position of the operation data stored in the first cache area is adjusted within the fourth time window.

[0220] Among them, first determine the first target data existing in the first cache area, and move the first target data from the first storage position to the third storage position within the inserted third time window. Here, the first target data is the operation data stored in the first cache area, and the first storage position of the first target data in the first cache area is not adjacent to the second storage position of the operation data other than the first target data in the first cache area, that is, the first target data is a memory fragment in the first cache area (such as Fig.12 because the third storage location is an idle storage location adjacent to the second storage location among multiple idle storage locations in the first cache area, the disclosed embodiment concentrates the memory fragments into non-memory fragments stored in the first cache area, so that after shifting at least one target operation in the first cache area, the memory fragments in the first cache area can be eliminated, so that the memory resources allocated in the first cache area (that is, the occupied operation data) are concentrated at the head of the first cache area, or at the tail of the first cache area.

[0221] In this way, after the movement of the first target data is completed within the fourth time window, memory resources can be reallocated for the first type of operations and / or the second type of operations to be executed within the third time window.

[0222] For example, see Fig.13 , Fig.13The memory resource distribution of the first cache area is shown according to an exemplary embodiment. Figure 2 ; Here, by Fig.12 The operation data with a memory capacity of 1000 is moved to the location of the operation data with a memory capacity of 3000, and we get Fig.13 As shown in the memory resource distribution, a memory resource of 4000 in size can be allocated in the first buffer area for use in the operation of the next time window.

[0223] In this way, the embodiment of the present disclosure can improve the success rate of memory resource allocation and further improve the efficiency of executing the memory operation sequence by adjusting the storage position of the first target data in the first cache area.

[0224] In some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0225] Determine the memory usage and remaining memory amount of the first buffer area in each time window according to the first type of operation and / or the second type of operation to be executed in each time window in the time series;

[0226] According to the remaining memory amounts corresponding to multiple time windows in the time series, a target remaining memory amount is determined; the target remaining memory amount is the minimum value of the remaining memory amounts corresponding to the multiple time windows;

[0227] A resident storage area is set in the first buffer area; the memory amount of the resident storage area is determined by the target remaining memory amount, and the storage position of the resident storage area is located at the end of the first buffer area;

[0228] Determine second target data from the operation data associated with each first-category operation to be executed in the memory operation sequence; the second target data is weight data of the first-category operation;

[0229] Determine at least one third target data from at least one second target data according to the amount of memory required by the second target data; the sum of the memory amounts of the at least one third target data is less than or equal to the memory amount of the resident storage area, and the memory amount of each third target data is less than the memory amount of the second target data other than the third target data in the at least one second target data;

[0230] At least one third target data is stored in the resident storage area.

[0231] Here, based on the execution timing information of the memory operation sequence, after determining the first type of operation and / or second type of operation to be executed in each time window, the memory occupancy of the first cache area in each time window can be estimated; in this way, the remaining memory amount in each time window can be further determined based on the difference between the memory amount of the first cache area in each time window and the above memory occupancy.

[0232] See also Fig.14 and Fig.15 , Fig.14 is a schematic diagram of a resident storage area according to an exemplary embodiment Figure 1 , Fig.15 is a schematic diagram of a resident storage area according to an exemplary embodiment Figure 2 ;in, Fig.14 The length of each line in represents the memory size of the first cache area, where each line indicates the memory size A' occupied by the first engine to perform the first type of operation, the memory size B' occupied by the second engine to perform the first sub-operation, the memory size C' occupied by the second engine to perform the second sub-operation, and the remaining memory size E (blank area) in each time window. Fig.14 It can be seen that the remaining memory amount E corresponding to multiple time windows in the time series can determine the minimum remaining memory amount as the target remaining memory amount. The target remaining memory amount represents the capacity of the space area that will not be used in the first cache area when the memory operation sequence is executed according to the execution timing information; thus, the embodiment of the present disclosure can be set according to the target remaining memory amount as follows Fig.14 and Fig.15 The resident storage area X shown in the figure can store some weight data with a small amount of memory in the resident storage area X for subsequent use by the first type of operations that are needed.

[0233] It should be noted that the embodiment of the present disclosure can set the resident storage area at the tail of the first cache area, so that the resident storage area will not form memory fragments, and in actual implementation, resources can be allocated to the operations of each time window in sequence from the head to the tail of the first cache area; it should be noted that after the operation of each time window is executed, the non-resident memory resources need to be released to reduce the generation of memory fragments.

[0234] In the case where there are multiple identical first-category operations in the memory operation sequence, although the data processed by the first engine when executing multiple first-category operations may be different, the weight data of the first-category operations used by the first engine when executing the first-category operations is fixed. In order to reduce the number of times the second engine performs the second-category operations on the weight data of the first-category operations, the disclosed embodiment can make full use of the space of the resident storage area and store the weight data of the first-category operations in the resident storage area, so that the second engine does not need to frequently perform the second-category operations on the weight data of multiple identical first-category operations, thereby effectively reducing not only the overall execution delay but also the power consumption of the system.

[0235] Considering that the memory space of the resident storage area is limited, the weight data required for each first type of operation in the memory operation sequence can be screened to determine the weight data that can be stored in the resident storage area.

[0236] Specifically, the disclosed embodiment determines the second target data from the operation data associated with each first-class operation to be executed in the memory operation sequence; the second target data is the above-mentioned weight data. Of course, in the above-mentioned first scenario, the weight data is also model parameters, gradient data, etc.; in addition, the second target data can also be operation data that can be shared by at least one first-class operation, such as label data in a supervised learning model.

[0237] After determining all the second target data, the disclosed embodiment can sort from small to large according to the amount of memory that the second target data needs to occupy, and obtain an ascending sequence; for the ascending sequence, the amount of memory occupied by each second target data in the sequence is added to the amount of memory before the sequence to obtain the addition result; if the addition result corresponding to the previous second target data in the adjacent second target data in the sequence is less than or equal to the amount of memory in the resident storage area, and the addition result corresponding to the next second target data is greater than the amount of memory in the resident storage area, then the previous second target data and the second target data before it are both used as the third target data. In this way, because the sum of the memory amounts of each third target data is less than or equal to the amount of memory in the resident storage area, and because of the ascending sequence, the memory amount of each third target data will be less than the memory amount of the second target data other than the third target data, so all third target data can be resident in the above-mentioned resident storage area.

[0238] In the embodiment of the present disclosure, the third target data determined above can be moved to the first cache area in advance for storage. Here, the embodiment of the present disclosure can store the third target data in the resident cache area in descending order according to the memory amount of the third target data. In addition, the third target data can be released from the first cache area after the memory operation sequence is completed, or released immediately when the third target data is no longer needed to be used by the subsequent first-type operation data.

[0239] It can be understood that the embodiment of the present disclosure can reduce memory fragmentation caused by the untimely release of these data by setting up a resident cache area to store the second target data with a small memory capacity; and the embodiment of the present disclosure sorts the second target data from small to large according to the amount of memory required to be occupied by the second target data, and determines the second target data sorted in front as the third target data, so that more weight data required for executing the first type of operation can be stored in the resident storage area, thereby reducing the delay caused by moving these data in and out, and reducing unnecessary power consumption generated by the operations of moving these data in and out, compared with storing the second target data with a larger memory capacity.

[0240] In some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0241] In a case where there is at least one identical first-category operation in the memory operation sequence of the first kernel and the memory operation sequence of the second kernel, at least one identical fourth target data is determined from the first operation data and the second operation data based on the at least one identical first-category operation; wherein the first operation data is operation data of a first sub-operation associated with at least one identical first-category operation in the first kernel; and the second operation data is operation data of a first sub-operation associated with at least one identical first-category operation in the second kernel;

[0242] Determine the first target time window and the second target time window according to the second time window of the first sub-operation associated with the fourth target data in the first kernel and the second time window of the first sub-operation associated with the fourth target data in the second kernel; the timing of the first target time window is earlier than any second time window in the second time window of the first kernel and the second time window of the second kernel; the timing of the second target time window is later than any second time window in the second time window of the first kernel and the second time window of the second kernel;

[0243] Within the first target time window, storing the fourth target data in the second buffer area; wherein the fourth target data stored in the second buffer area can be shared by the first kernel and the second kernel;

[0244] And within the second target time window, the memory resources occupied by the fourth target data are released in the second buffer area.

[0245] Here, the memory operation sequence proposed in the embodiment of the present disclosure can be executed by the first core and the second core in the processor that can execute in parallel; wherein, the first core and the second core can execute the same memory operation sequence in parallel, or can execute different memory sequences in parallel, and the embodiment of the present disclosure does not limit this.

[0246] In actual implementation, if the same first-category operation exists in the memory operation sequences executed by the first kernel and the second kernel respectively, it can be determined whether the same first-category operation requires the same input data; at this time, the embodiment of the present disclosure determines the operation data of the first sub-operation associated with the above-mentioned first-category operation in the first kernel (such as the data moving-in operation in the above example) as the first operation data, and determines the operation data of the first sub-operation associated with the same first-category operation in the second kernel (such as the data moving-in operation) as the second operation data; and then determines one or more identical fourth target data from the first operation data and the second operation data.

[0247] Here, the fourth target data may be the weight data required for execution of each network node in the AI ​​model or some shared input data such as label data, wherein the weight data may include bias data.

[0248] Since these fourth target data are data required by both the first core and the second core, but if both need to read the fourth target data from the memory area, the access speed is slow and the power consumption is high; therefore, the embodiment of the present disclosure can set another on-chip cache with a smaller memory capacity than the memory area and a higher access speed than the memory area, such as the second cache area (L2 cache), and the data stored in the L2 cache can be accessed and moved by the first core and the second core; in this way, the embodiment of the present disclosure can first store the fourth target data in the second cache area as needed, so that the first core and the second core can directly access the fourth target data from the second cache area, thereby improving the access speed and reducing excess power consumption.

[0249] Specifically, the embodiment of the present disclosure first determines the second time window of the first sub-operation associated with the fourth target data in the first kernel, and the second time window of the first sub-operation associated with the fourth target data in the second kernel, and in the time series, a time window that is earlier than the second time window of the first kernel and the second time window of the second kernel determined above is used as the first target time window; and in actual implementation, within the first target time window, the fourth target data is stored in the second cache area.

[0250] Among them, the method for determining the first target time window in the embodiment of the present disclosure can be: compare the timing of the second time window of the first kernel determined above and the second time window of the second kernel, and take the previous time window of the second time window with an earlier timing as the above first target time window.

[0251] In addition, in order to reduce the memory resource usage of the second cache area, the embodiment of the present disclosure can release the fourth target data from the second cache area after the first sub-operation corresponding to the first kernel and the second kernel is executed, that is, after the fourth target data is accessed. Specifically, the embodiment of the present disclosure can use a time window that is later than the second time window of the first kernel and the second time window of the second kernel determined above as the second target time window in the time series; and in actual implementation, the fourth target data is released from the second cache area within the second target time window.

[0252] Among them, the method for determining the second target time window in the embodiment of the present disclosure can be: compare the timing of the second time window of the first kernel and the second time window of the second kernel determined above, and take the next time window of the second time window with a later timing as the first target time window mentioned above.

[0253] For example, see Fig.16 , Fig.16It is a timing diagram of storing fourth target data to a first core and a second core according to an exemplary embodiment; wherein, the first core core1 and the second core core2 both have the same first-class operation: operation 3; each time window of the first core core1 and the second core core2 corresponds to each time window of the time series; wherein, operation 3 in both cores requires the same weight data (weight3); through the execution timing information determined by core1 and the execution timing information determined by core2, it is obtained that core1 can move weight3 in time window 1, and core1 can move weight3 in time window 2; in this way, the embodiment of the present disclosure uses time window 0, which is earlier than time window 1 and time window 2, as the first target time window, and uses the DMA engine to move weight3 from the memory area (DDR) to the second cache area (L2 cache) for storage in time window 0, and then in time window 2, the first core uses the DMA engine to move weight3 from the second cache area to the first cache area; in time window 3, the second core uses the DMA engine to move weight3 from the second cache area to the first cache area. In addition, the embodiment of the present disclosure uses time window 3 which is later than time window 1 and time window 2 as the second target time window, and releases weight 3 from the second buffer area in time window 3.

[0254] The disclosed embodiment sets a second cache area for the parallel cores and moves the same fourth target data required by the parallel cores from the memory area in advance, thereby increasing the speed at which the parallel cores acquire the fourth target data and reducing the power consumption of the system executing a memory operation sequence.

[0255] In the disclosed embodiment, for some memory operation sequences, there is a situation where the execution time required for multiple consecutive first-category operations is very short, but the execution time required for associated second-category operations is very long. At this time, the first time windows corresponding to these first-category operations cannot be hidden from the second-category operations occurring within the time window, that is, the delay generated within the corresponding time window is relatively large.

[0256] In view of this, in some embodiments, the timing arrangement method proposed in the present disclosure further includes:

[0257] According to the second estimated duration of each time window in the time series, at least two third target time windows adjacent in time sequence are determined from the multiple time windows of the time series; the second estimated duration of each third target time window is less than a preset duration threshold;

[0258] According to the fourth target time window with the earliest time sequence among at least two third target time windows adjacent in time sequence, a plurality of third target variables are determined from the second variables corresponding to at least two third target time windows adjacent in time sequence; the third target variable is the second variable corresponding to the third target time window other than the fourth target time window among at least two third target time windows adjacent in time sequence;

[0259] According to the fifth target time window with the latest timing among at least two third target time windows adjacent in time sequence, a plurality of fourth target variables are determined from the first variables corresponding to at least two third target time windows adjacent in time sequence; the fourth target variable is the first variable corresponding to the third target time window other than the fifth target time window among at least two third target time windows adjacent in time sequence;

[0260] The values ​​of the plurality of third target variables and the plurality of fourth target variables are all set to be equal to the third value, thereby obtaining a third constraint condition; wherein the third value of the third target variable is used to indicate that the second sub-operation is not performed within the candidate time window corresponding to the third target variable; and the third value of the fourth target variable is used to indicate that the first sub-operation is not performed within the candidate time window corresponding to the fourth target variable;

[0261] The above-mentioned target values ​​of multiple decision variables are determined based on the objective function and the first constraint condition and the second constraint condition, including:

[0262] Based on the objective function and the first constraint, the second constraint, and the third constraint, a target value is determined.

[0263] In an embodiment of the present disclosure, in order to determine the situation where the execution time required for the second type of operation corresponding to at least two consecutive time windows is very long, the present disclosure determines at least two third target time windows that are adjacent in time sequence in multiple time windows of the time series based on the second estimated duration of each time window in the time series; here, the second estimated duration of each third target time window is less than a preset duration threshold, and the preset duration threshold is used to reflect that the total duration required to execute the second type of operation in the corresponding third time window far exceeds the time required to execute the first type of operation in the time window, thereby seriously increasing the total execution delay of the memory operation sequence.

[0264] Here, each time window can have a unique corresponding preset duration threshold, which is determined by the first estimated duration of the corresponding time window; the preset duration thresholds corresponding to each first time window may also be the same, and the preset duration threshold may be an empirical value determined by evaluating the execution duration of the memory operation in each time window during the historical execution or test of the same memory operation sequence. The embodiments of the present disclosure do not impose any restrictions on this.

[0265] After determining the third target time windows that are consecutive in time sequence, the embodiment of the present disclosure can merge these third target time windows that are consecutive in time sequence into the same time window, thereby re-planning the time sequence for the operations that can be executed in each time window.

[0266] Here, taking the example of the second scenario mentioned above as an example, the steps of merging these sequentially continuous third target time windows into the same time window are as follows: determine the earliest fourth target time window (the first third time window) in at least two sequentially adjacent third target time windows, and remove the second variable corresponding to the data moving-out operation of the fourth target time window from the second variable corresponding to the data moving-out operation of the at least two sequentially adjacent third target time windows, and obtain multiple third target variables. At the same time, determine the latest fifth target time window (the last third time window) in at least two sequentially adjacent third target time windows, and remove the first variable corresponding to the data moving-in operation of the fifth target time window from the first variable corresponding to the data moving-in operation of the fifth target time window, and obtain multiple fourth target variables; after determining multiple fourth target variables and multiple fifth target variables, set them all to the third value as the third constraint condition.

[0267] The third value may be 0, indicating that the data moving-out operation is not performed in the candidate time window corresponding to the third target variable, and the data moving-in operation is not performed in the candidate time window corresponding to the fourth target variable. In this way, by setting all the first variables and the second variables of at least two third time windows that are continuous in time sequence to 0 except for the second variable of the data moving-out operation in the first time window and the first variable of the data moving-in operation in the last time window, these third time windows are merged into one time window.

[0268] For example, see Fig.17 , Fig.17 is a schematic diagram of time window fusion according to an exemplary embodiment; wherein, through the third constraint condition, time window 2, time window 3 and time window 4 can be fused to form a new time window.

[0269] In the embodiment of the present disclosure, by adding the third constraint condition, the first constraint condition, the second constraint condition, the third constraint condition and the objective function can be used to form an updated linear programming model, which can make the values ​​of each decision variable obtained by the linear programming model more reasonable and more practical.

[0270] In the embodiment of the present disclosure, the timing arrangement method for the memory operation sequence can be applied to the AI ​​model, combined with Figure 2For operations in different network nodes in the AI ​​model that require asynchronous execution by the asynchronous engine, the DMA engine can move input data and weight data from the memory area to the first cache area. After the computing engine calculates each first-class operation, it may need to move the result data from the first cache area to the memory area. The two asynchronous engines execute commands in a specific order. The most basic constraint is to meet data dependencies. For example, before each memory operation is calculated, the DMA engine is required to move its input data to the first cache area in advance; if the result data of a memory operation needs to be moved out, the DMA engine is scheduled to perform the data move operation only after the memory operation is completed. In addition, for example Figure 3 , the weight data of operation 3 needs to be moved into the first buffer in advance, and at this time, operation 2 may be executing, and the first buffer also needs to allocate corresponding memory space for its input data and result data. It is even possible that the result data of operation 1 needs to be moved out to the memory area at this time, that is, when operation 2 is executed, the space occupied by the result of operation 1 in the first buffer must also be guaranteed. This means that the input data or result data of multiple adjacent memory operations share the limited amount of memory in the first buffer with the input data or result data of the current operation. In this way, each time window of the memory operation sequence needs to have enough memory allocated in the first buffer to accommodate the above tensors. At the same time, because both the computing engine and the DMA engine have execution time, how to arrange the timing to minimize the total execution delay of the memory operation sequence depends on whether the execution time of the parallel computing engine and DMA engine in each time window can be hidden from each other as a whole.

[0271] In the related art, the timing arrangement is hard-coded, which can be implemented by programming with complex conditional loops and other control processes. For example, the basic model may be: sort the memory operations in the memory operation sequence corresponding to the AI ​​model in depth-first topological order, determine a time window for a memory operation, and combine the start time window and the end time window to obtain the following: Figure 4 The six time windows shown in FIG. 1 and FIG. 2 are the same as those in FIG. 2 ; and for each memory operation, it can be Figure 4 , choose to move input data in a time window before the time window corresponding to the first type of operation, or move result data out in a time window after the time window; however, in actual situations, there are execution methods that use less memory and have a smaller overall delay; see Fig.18 , Fig.18 A timing distribution of a memory operation sequence is shown according to an exemplary embodiment. Figure 2 ; Fig.18 and Figure 4 This is a timing arrangement method for each operation of the same memory operation sequence. After actual testing, Fig.18 The overall latency shown is smaller and the memory usage of the first buffer area is smaller.

[0272] Therefore, if we need to consider the possible time windows in which weight data, calculation data, and result data can be imported, these possibilities will increase the complexity of hard coding, and all tensors may have similar multiple options, resulting in many overall decision points. It is impossible for the code to judge and process each decision one by one, so it can only be done as follows Figure 4 As shown, the time window is fixedly advanced or delayed by one time window, but such a fixed arrangement cannot obtain the optimal solution.

[0273] In view of this, an embodiment of the present disclosure proposes a timing arrangement method, and the specific process can be found in the above text of the present disclosure, wherein the timing arrangement scheme of the present disclosure takes each operation in the memory operation sequence in the AI ​​model that needs to be asynchronously executed by the asynchronous engine as an optimization decision point, and abstracts the optimization decision point into a decision variable; in this way, by setting constraints and objective functions, a linear programming model that is effective for various optimization decision points is obtained by taking a comprehensive view of the overall situation; through the linear programming model, the decision results of each optimization decision point are obtained, so that the overall delay for the execution of the memory operation sequence is smaller, and the memory shortage problem is improved, and it can be applied to scenarios such as multi-core data sharing and resident sharing of small memory weight data.

[0274] Here, when the above-mentioned AI model is a resnet50 model, the embodiment of the present disclosure uses the above-mentioned timing arrangement method to perform timing arrangement processing on the resnet50 training process, and trains resnet50 according to the obtained execution timing information. Through testing, the overall execution time obtained by forward training resnet50 using the execution timing information is 8409.05us, which is 15% higher than the overall time of 9893.63us without timing arrangement. In addition, the overall execution time obtained by backward training resnet50 using the execution timing information is 299120.60us, which is 36% higher than the overall time of 473676.13us without timing arrangement.

[0275] The present disclosure also provides a time sequence arrangement device. Fig.19 , Fig.19 1 is a schematic diagram of a timing arrangement device according to an exemplary embodiment. The timing arrangement device 1900 mainly includes:

[0276] The first determining module 1901 is configured to determine, from each time window of the time sequence, at least one candidate time window of a second type of operation associated with each first type of operation according to the first time window occupied by each first type of operation to be executed in the memory operation sequence in each time window of the time sequence; the candidate time window of the second type of operation is different from the first time window of the first type of operation associated with the same second type of operation;

[0277] The first estimation module 1902 is configured to determine a first estimated duration of each time window in the time series according to the first time window corresponding to the first type of operation; the first estimated duration of a time window is the duration required for the first engine to execute the first type of operation in the time window;

[0278] The second estimation module 1903 is configured to determine a second estimated duration of each time window in the time series according to at least one candidate time window of each second type of operation; the second estimated duration of a time window is the duration required for the second type of operation in the time window to be executed by the second engine;

[0279] The second determination module 1904 is configured to determine, from at least one candidate time window of each second-category operation, a second time window corresponding to each second-category operation according to the first estimated duration and the second estimated duration of each time window; wherein, when the second-category operation is executed by the second engine in the corresponding second time window, the total execution delay corresponding to the memory operation sequence is less than the total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence;

[0280] The timing determination module 1905 is configured to determine the execution timing information of the memory operation sequence according to the first time window of each first-type operation and the second time window of the second-type operation associated with the first-type operation.

[0281] In some embodiments, the second estimation module 1903 is further configured to set a plurality of decision variables according to at least one candidate time window of each second type of operation; wherein each decision variable corresponds to a candidate time window of a second type of operation; different values ​​of the decision variables are used to indicate whether the second type of operation is executed by the second engine within the corresponding candidate time window; and determine the second estimated duration of each time window according to at least one second type of operation expected to be executed within each time window in the time series, the decision variables corresponding to the second type of operation, and the execution duration required to execute the second type of operation;

[0282] The second determination module 1904 is further configured to determine the total execution delay corresponding to the memory operation sequence based on the first estimated duration and the second estimated duration of each time window in the time series; determine the minimization of the total execution delay corresponding to the memory operation sequence as the objective function; determine the target values ​​of multiple decision variables based on the constraints of multiple decision variables and the objective function; wherein the target values ​​of the decision variables are used to determine the second time window of the second type of operation corresponding to the decision variables.

[0283] In some embodiments, the second determination module 1904 is further configured to determine a first constraint condition based on a constraint relationship between decision variables corresponding to at least one candidate time window of each second-category operation; determine, for each time window of the time series, a first estimated amount of memory in the first cache area for the first-category operation corresponding to the first engine execution time window; determine a second estimated amount of memory in the first cache area for operation data associated with at least one second-category operation expected to be executed within each time window based on the amount of memory occupied by the operation data associated with each second-category operation and at least one decision variable corresponding to the second-category operation; determine a second constraint condition based on the first estimated amount of memory and the second estimated amount of memory within each time window and the amount of memory in the first cache area; the second constraint condition limits the sum of the first estimated amount of memory and the second estimated amount of memory within each time window to be less than the amount of memory in the first cache area; and determine target values ​​of multiple decision variables based on the objective function and the first constraint condition and the second constraint condition.

[0284] In some embodiments, the first determination module 1901 is further configured to determine, based on the first time windows of each first-category operation to be executed in the memory operation sequence, at least one time window in the time sequence that is earlier than the first time window as at least one candidate time window of the first sub-operation associated with the first-category operation; wherein the execution timing of the first sub-operation is earlier than the execution timing of the first-category operation associated with the first sub-operation; based on the first time windows of each first-category operation, at least one time window in the time sequence that is later than the first time window is determined as at least one candidate time window of the second sub-operation associated with the first-category operation; the execution timing of the second sub-operation is later than the execution timing of the first-category operation associated with the second sub-operation; wherein the decision variables include: a first variable corresponding to the first sub-operation, and a second variable corresponding to the second sub-operation.

[0285] In some embodiments, the second determination module 1904 is further configured to determine the first sum value between the first variables corresponding to each candidate time window of each first sub-operation; set the first sum value between the first variables of each first sub-operation equal to the first value to obtain a first constraint relationship; the first constraint relationship is used to limit the second engine to execute the first sub-operation in one candidate time window of at least one candidate time window of a first sub-operation; determine the second sum value between the second variables corresponding to each candidate time window of each second sub-operation; set the second sum value between the second variables of each second sub-operation equal to the first value to obtain a second constraint relationship; the second constraint relationship is used to limit the second engine to execute the second sub-operation in one candidate time window of at least one candidate time window of a second sub-operation; determine the first constraint condition based on the first constraint relationship and the second constraint relationship.

[0286] In some embodiments, the second determination module 1904 is further configured to determine at least one first target variable of the first sub-operation in the i-th time window from a plurality of first variables associated with the first sub-operation expected to be executed in the i-th time window; the timing of the candidate time window corresponding to the at least one first target variable is less than or equal to i; i is greater than or equal to 1, and less than or equal to N, where N is the number of time windows contained in the time series; determine at least one second target variable of the second sub-operation in the i-th time window from a plurality of second variables associated with the second sub-operation expected to be executed in the i-th time window; the timing of the candidate time window corresponding to the at least one second target variable is greater than or equal to i; determine the second estimated amount of memory occupied by the operation data associated with at least one second type of operation expected to be executed in the i-th time window in the first cache area based on the amount of memory occupied by the operation data associated with the at least one first target variable of the first sub-operation and the first sub-operation in the i-th time window, and the amount of memory occupied by the operation data associated with the at least one second target variable of the second sub-operation and the second sub-operation in the i-th time window.

[0287] In some embodiments, the timing arrangement device further includes: a setting module;

[0288] A setting module is configured to determine a candidate time window of a third type of operation associated with the target operation according to at least one time window between the first time windows occupied by each of the at least two target operations in the time sequence, when there are at least two target operations with non-adjacent timings among multiple first type operations in a memory operation sequence, and the result data of the target operation that is earlier in timing among the at least two target operations is used as the input data of the target operation that is later in timing; and to set a third variable according to the candidate time window of the third type of operation associated with the target operation; wherein each third variable corresponds to a candidate time window of a third type of operation; and different values ​​of the third variable are used to indicate whether the result data of the target operation that is earlier in timing within the candidate time window of the third type of operation is stored in the first cache area.

[0289] In some embodiments, the second determination module 1904 is further configured to determine the first constraint condition based on the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation; wherein the second target sub-operation is the second sub-operation associated with the target operation that is earlier in timing; and the first target sub-operation is the first sub-operation associated with the target operation that is later in timing.

[0290] In some embodiments, the second determination module 1904 is further configured to determine a third sum value between the second variables corresponding to each candidate time window of the second target sub-operation, and a third sum value between the first variables corresponding to each candidate time window of the first target sub-operation; set the third sum value between the second variables corresponding to each candidate time window of the second target sub-operation equal to the fourth sum value between the first variables corresponding to each candidate time window of the first target sub-operation, to obtain a third constraint relationship; the third constraint relationship is used to limit the second engine to execute the first target sub-operation in any candidate time window of at least one candidate time window of the corresponding second target sub-operation when the second target sub-operation is executed; set the sum value between the second variable and the third variable corresponding to each candidate time window of the second target sub-operation equal to the first value, to obtain a fourth constraint relationship; the fourth constraint relationship is used to limit the second target sub-operation in any candidate time window of the corresponding at least one candidate time window When the third type of operation associated with the target operation that is previously executed does not need to be executed, or, when the third type of operation associated with the target operation that is previously executed is executed, the second target sub-operation does not need to be executed; the first parameter is determined according to the product between the timing value corresponding to each candidate time window of the first target sub-operation and the first variable corresponding to the candidate time window, and the product between the timing value corresponding to each candidate time window of the second target sub-operation and the second variable corresponding to the candidate time window; the first parameter is used to indicate the execution time difference between the first target sub-operation and the second target sub-operation when the second engine executes the first target sub-operation and the second target sub-operation; the fifth sum of the first parameter and the second parameter is set to be greater than or equal to the second value to obtain the fifth constraint relationship; the second parameter is the product of the second value and the third variable; the fifth constraint relationship is used to limit the execution time difference between the first target sub-operation and the second target sub-operation to be greater than the second value; the first constraint condition is determined according to the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship.

[0291] In some embodiments, the second determination module 1904 is further configured to determine a third estimated amount of memory occupied by the result data in the first buffer area in each candidate time window of the third type of operation associated with the target operation preceding in time sequence, according to the product of the amount of memory occupied by the result data of the target operation and the third variable;

[0292] Determine a second constraint condition according to the first estimated memory amount, the second estimated memory amount, and the third estimated memory amount in each time window, and the memory amount of the first buffer area;

[0293] The second constraint condition is also used to limit the sum of the first estimated memory amount, the second estimated memory amount and the third estimated memory amount in each time window to be less than the memory amount of the first cache area.

[0294] In some embodiments, the timing arrangement device further includes: a memory management module;

[0295] The memory management module is configured to determine the first-category operation and / or the second-category operation to be executed in each time window in the time sequence according to the execution timing information of the memory operation sequence; in each time window, allocate memory resources of the first cache area to the first-category operation and / or the second-category operation to be executed in the time window; after executing the first-category operation and / or the second-category operation based on the allocated memory resources, release at least part of the memory resources allocated to the first-category operation and / or the second-category operation.

[0296] In some embodiments, the memory management module is also configured to insert a fourth time window in the time sequence if the allocation of memory resources for the first type of operation and / or the second type of operation to be executed within a third time window in the time sequence fails; the fourth time window is the previous time window of the third time window; determine the first storage position of the first target data in the first cache area; wherein the first target data is the operation data stored in the first cache area, and the first storage position of the first target data in the first cache area is not adjacent to the second storage position of the operation data other than the first target data in the first cache area; move the first target data from the first storage position of the first cache area to the third storage position of the first cache area within the fourth time window; the third storage position is an idle storage position adjacent to the second storage position among multiple idle storage positions in the first cache area; after completing the movement of the first target data, reallocate memory resources for the first type of operation and / or the second type of operation to be executed in the third time window.

[0297] In some embodiments, the memory management module is further configured to determine the memory occupancy and remaining memory amount of the first cache area in each time window according to the first type of operation and / or the second type of operation to be executed in each time window in the time series; determine the target remaining memory amount according to the remaining memory amounts corresponding to multiple time windows in the time series; the target remaining memory amount is the minimum value of the remaining memory amounts corresponding to the multiple time windows; set a resident storage area in the first cache area; the memory amount of the resident storage area is determined by the target remaining memory amount, and the storage position of the resident storage area is located at the end of the first cache area; determine the second target data from the operation data associated with each first type of operation to be executed in the memory operation sequence; the second target data is the weight data of the first type of operation; determine at least one third target data from at least one second target data according to the memory amount required to be occupied by the second target data; the sum of the memory amounts of at least one third target data is less than or equal to the memory amount of the resident storage area, and the memory amount of each third target data is less than the memory amount of the second target data other than the third target data in at least one second target data; store at least one third target data in the resident storage area.

[0298] In some embodiments, the memory management module is further configured to determine at least one identical fourth target data from the first operation data and the second operation data based on at least one identical first operation when there is at least one identical first operation in the memory operation sequence of the first kernel and the memory operation sequence of the second kernel; wherein the first operation data is the operation data of the first sub-operation associated with at least one identical first operation in the first kernel; the second operation data is the operation data of the first sub-operation associated with at least one identical first operation in the second kernel; the first target time window and the second target time window are determined according to the second time window of the first sub-operation associated with the fourth target data in the first kernel and the second time window of the first sub-operation associated with the fourth target data in the second kernel; the timing of the first target time window is earlier than any second time window in the second time window of the first kernel and the second time window of the second kernel; the timing of the second target time window is later than any second time window in the second time window of the first kernel and the second time window of the second kernel; within the first target time window, the fourth target data is stored in the second cache area; wherein the fourth target data stored in the second cache area can be shared by the first kernel and the second kernel; and within the second target time window, the memory resources occupied by the fourth target data are released in the second cache area.

[0299] In some embodiments, the second determination module 1904 is further configured to determine at least two third target time windows adjacent in time sequence from multiple time windows of the time series according to the second estimated duration of each time window in the time series; the second estimated duration of each third target time window is less than a preset duration threshold; according to the fourth target time window with the earliest duration in at least two third target time windows adjacent in time sequence, multiple third target variables are determined from the second variables corresponding to at least two third target time windows adjacent in time sequence; the third target variable is the second variable corresponding to the third target time window other than the fourth target time window from at least two third target time windows adjacent in time sequence; according to the fourth target time window with the latest duration in at least two third target time windows adjacent in time sequence The fifth target time window determines multiple fourth target variables from the first variables corresponding to at least two third target time windows adjacent in time sequence; the fourth target variable is the first variable corresponding to the third target time window other than the fifth target time window from at least two third target time windows adjacent in time sequence; the values ​​of multiple third target variables and multiple fourth target variables are set to be equal to the third value to obtain the third constraint condition; wherein the third value of the third target variable is used to indicate that the second sub-operation is not performed within the candidate time window corresponding to the third target variable; the third value of the fourth target variable is used to indicate that the first sub-operation is not performed within the candidate time window corresponding to the fourth target variable; the target value is determined based on the objective function and the first constraint condition, the second constraint condition, and the third constraint condition.

[0300] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0301] Fig. 20 2 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 2000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0302] Reference Fig. 20 The electronic device 2000 may include one or more of the following components: a processing component 2002 , a memory 2004 , a power component 2006 , a multimedia component 2008 , an audio component 2010 , an input / output interface 2012 , a sensor component 2014 , and a communication component 2016 .

[0303] The processing component 2002 generally controls the overall operation of the electronic device 2000, such as operations associated with at least one of display, phone calls, data communications, camera operations, and recording operations. The processing component 2002 may include one or more processors 2020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 2002 may include one or more modules to facilitate the interaction between the processing component 2002 and other components. For example, the processing component 2002 may include a multimedia module to facilitate the interaction between the multimedia component 2008 and the processing component 2002.

[0304] The memory 2004 is configured to store various types of data to support operations on the electronic device 2000. Examples of such data include at least one of the following: instructions for any application or method operating on the electronic device 2000, contact data, phone book data, messages, pictures, and videos. The memory 2004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PRO M), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0305] The power supply component 2006 provides power to various components of the electronic device 2000. The power supply component 2006 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 2000.

[0306] The multimedia component 2008 includes a screen that provides an output interface between the electronic device 2000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 2008 includes a front camera and / or a rear camera. When the electronic device 2000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0307] The audio component 2010 is configured to output and / or input audio signals. For example, the audio component 2010 includes a microphone (MIC), and when the electronic device 2000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 2004 or sent via the communication component 2016. In some embodiments, the audio component 2010 also includes a speaker for outputting audio signals.

[0308] The input / output interface 2012 provides an interface between the processing component 2002 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0309] The sensor assembly 2014 includes one or more sensors for providing various aspects of status assessment for the electronic device 2000. For example, the sensor assembly 2014 can detect the open / closed state of the electronic device 2000, the relative positioning of the components, such as the display and keypad of the electronic device 2000, and the sensor assembly 2014 can also detect the position change of the electronic device 2000 or a component in the electronic device 2000, the presence or absence of contact between the user and the electronic device 2000, the orientation or acceleration / deceleration of the electronic device 2000, and the temperature change of the electronic device 2000. The sensor assembly 2014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 2014 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 2014 may also include, but is not limited to, at least one of the following: an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, and a temperature sensor.

[0310] The communication component 2016 is configured to facilitate communication between the electronic device 2000 and other devices in a wired or wireless manner. The electronic device 2000 can access a wireless network based on a communication standard, such as Wi-Fi, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 2016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 2016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies to achieve.

[0311] In an exemplary embodiment, the electronic device 2000 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components.

[0312] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 2004 including executable instructions or a computer program, which can be executed by a processor 2020 of an electronic device 2000 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disk read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0313] The embodiments of the present disclosure also provide a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can execute any one of the timing arrangement methods described above in the embodiments of the present disclosure.

[0314] The embodiment of the present disclosure provides a computer program product, which includes: a computer program or an executable instruction, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the executable instruction from the computer-readable storage medium, and the processor executes the computer program or the executable instruction, so that the computer device executes any one of the above-mentioned timing arrangement methods of the embodiment of the present disclosure.

[0315] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0316] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A time sequence arrangement method, characterized in that: include: According to the first time windows occupied by each first-category operation to be executed in the memory operation sequence in each time window of the time sequence, at least one candidate time window of the second-category operation associated with each first-category operation is determined from each time window of the time sequence; the candidate time window of the second-category operation is different from the first time window of the first-category operation associated with the same second-category operation; Determining a first estimated duration of each time window in the time series according to a first time window corresponding to the first type of operation; A first estimated duration of the time window is a duration required for the first type of operation in the time window to be executed by the first engine; Determine, according to at least one candidate time window of each second type of operation, a second estimated duration of each time window in the time series; the second estimated duration of one of the time windows is the duration required for the second type of operation in the time window to be executed by the second engine; According to the first estimated duration and the second estimated duration of each time window, a second time window corresponding to each second-category operation is determined from at least one candidate time window of each second-category operation; wherein, when the second-category operation is executed by the second engine in the corresponding second time window, the total execution delay corresponding to the memory operation sequence is less than the total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence; The execution timing information of the memory operation sequence is determined according to the first time window of each of the first-type operations and the second time window of the second-type operations associated with the first-type operations.

2. The method according to claim 1, characterized in that The determining, according to at least one candidate time window of each second-category operation, a second estimated duration of each time window in the time series comprises: According to at least one candidate time window of each second-category operation, a plurality of decision variables are set; wherein each decision variable corresponds to a candidate time window of a second-category operation; and different values ​​of the decision variables are used to indicate whether the second-category operation is executed by the second engine within the corresponding candidate time window; Determine the second estimated duration of each time window according to at least one second-type operation expected to be performed in each time window in the time series, a decision variable corresponding to the second-type operation, and an execution duration required to perform the second-type operation; The determining, according to the first estimated duration and the second estimated duration of each time window, a second time window corresponding to each second-category operation from at least one candidate time window of each second-category operation includes: Determining a total execution delay corresponding to the memory operation sequence according to the first estimated duration and the second estimated duration of each time window in the time sequence; Minimizing the total execution delay corresponding to the memory operation sequence is determined as an objective function; According to the constraints of the multiple decision variables and the objective function, target values ​​of the multiple decision variables are determined; wherein the target values ​​of the decision variables are used to determine a second time window of the second type of operation corresponding to the decision variables.

3. The method according to claim 2, characterized in that The method further comprises: Determining a first constraint condition according to a constraint relationship between decision variables corresponding to at least one candidate time window of each second-type operation; For each time window of the time series, determining a first estimated amount of memory in the first cache area for the first engine to execute a first type of operation corresponding to the time window; Determine, according to the amount of memory occupied by the operation data associated with each second-type operation and at least one decision variable corresponding to the second-type operation, a second estimated amount of memory occupied in the first cache area by the operation data associated with at least one second-type operation expected to be executed within each time window; Determine a second constraint condition according to the first estimated amount of memory and the second estimated amount of memory in each time window, and the amount of memory in the first cache area; the second constraint condition limits the sum of the first estimated amount of memory and the second estimated amount of memory in each time window to be less than the amount of memory in the first cache area; Determining target values ​​of the multiple decision variables according to the constraint conditions of the multiple decision variables and the objective function includes: Based on the objective function and the first constraint condition and the second constraint condition, target values ​​of the multiple decision variables are determined.

4. The method according to claim 3, characterized in that The determining, according to the first time windows occupied by each first-category operation to be executed in the memory operation sequence in each time window of the time sequence, at least one candidate time window of the second-category operation associated with each first-category operation from each time window of the time sequence comprises: According to the first time windows of each first-category operation to be executed in the memory operation sequence, at least one time window in the time sequence that is earlier than the first time window is determined as at least one candidate time window of a first sub-operation associated with the first-category operation; wherein the execution timing of the first sub-operation is earlier than the execution timing of the first-category operation associated with the first sub-operation; According to the first time windows of each of the first-category operations, at least one time window in the time sequence that is later than the first time window is determined as at least one candidate time window of a second sub-operation associated with the first-category operation; the execution timing of the second sub-operation is later than the execution timing of the first-category operation associated with the second sub-operation; The decision variables include: a first variable corresponding to the first sub-operation, and a second variable corresponding to the second sub-operation.

5. The method according to claim 4, characterized in that The determining of the first constraint condition according to the constraint relationship between the decision variables corresponding to at least one candidate time window of each second type operation includes: Determine a first sum value between first variables corresponding to each candidate time window of each first sub-operation; Setting a first sum value between each first variable of each first sub-operation to be equal to a first value, to obtain a first constraint relationship; the first constraint relationship is used to restrict the second engine from executing the first sub-operation in one candidate time window of at least one candidate time window of the first sub-operation; Determine a second sum value between second variables corresponding to each candidate time window of each second sub-operation; Setting a second sum value between each second variable of each second sub-operation equal to the first value to obtain a second constraint relationship; the second constraint relationship is used to restrict the second engine to execute the second sub-operation in one candidate time window of at least one candidate time window of the second sub-operation; The first constraint condition is determined according to the first constraint relationship and the second constraint relationship.

6. The method according to claim 4, characterized in that The determining, according to the amount of memory occupied by the operation data associated with each second-type operation and at least one decision variable corresponding to the second-type operation, a second estimated amount of memory occupied in the first cache area by the operation data associated with at least one second-type operation expected to be executed within each time window comprises: Determine, from a plurality of first variables associated with a first sub-operation expected to be performed in an i-th time window, at least one first target variable of the first sub-operation in the i-th time window; the time sequence of the candidate time window corresponding to the at least one first target variable is less than or equal to i; i is greater than or equal to 1 and less than or equal to N, where N is the number of time windows included in the time sequence; Determine, from a plurality of second variables associated with the second sub-operation expected to be performed in the i-th time window, at least one second target variable of the second sub-operation in the i-th time window; the timing of the candidate time window corresponding to the at least one second target variable is greater than or equal to i; Based on the amount of memory occupied by at least one first target variable of the first sub-operation and the operation data associated with the first sub-operation in the i-th time window, and the amount of memory occupied by at least one second target variable of the second sub-operation and the operation data associated with the second sub-operation in the i-th time window, determine a second estimated amount of memory occupied in the first cache area by the operation data associated with at least one second-type operation expected to be executed in the i-th time window.

7. The method according to claim 4, characterized in that The method further comprises: In a case where there are at least two target operations whose timings are not adjacent in the plurality of first-category operations of the memory operation sequence, and result data of the target operation that is earlier in timing among the at least two target operations is used as input data of the target operation that is later in timing, determining a candidate time window of a third-category operation associated with the target operation according to at least one time window between the first time windows occupied by the at least two target operations in the time sequence; A third variable is set according to the candidate time window of the third type of operation associated with the target operation; wherein each third variable corresponds to a candidate time window of a third type of operation; and different values ​​of the third variable are used to indicate whether the result data of the target operation that is earlier in timing within the candidate time window of the third type of operation is stored in the first cache area.

8. The method according to claim 7, characterized in that The determining of the first constraint condition according to the constraint relationship between the decision variables corresponding to at least one candidate time window of each second type operation includes: The first constraint condition is determined based on the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation; wherein the second target sub-operation is the second sub-operation associated with the target operation that is earlier in timing; and the first target sub-operation is the first sub-operation associated with the target operation that is later in timing.

9. The method according to claim 8, characterized in that The determining the first constraint condition according to the constraint relationship between the second variable corresponding to each candidate time window of the second target sub-operation, the first variable corresponding to each candidate time window of the first target sub-operation, and the third variable of the third type of operation associated with the target operation includes: Determine a third sum value between second variables corresponding to each candidate time window of the second target sub-operation, and a third sum value between first variables corresponding to each candidate time window of the first target sub-operation; The third sum value between the second variables corresponding to each candidate time window of the second target sub-operation is set to be equal to the fourth sum value between the first variables corresponding to each candidate time window of the first target sub-operation, to obtain a third constraint relationship; the third constraint relationship is used to restrict that when the second target sub-operation is executed in any candidate time window of the corresponding at least one candidate time window, the second engine needs to execute the first target sub-operation in any candidate time window of at least one candidate time window of the first target sub-operation; The sum of the second variable and the third variable corresponding to each candidate time window of the second target sub-operation is set to be equal to the first value, to obtain a fourth constraint relationship; the fourth constraint relationship is used to restrict that when the second target sub-operation is executed in any candidate time window of the corresponding at least one candidate time window, the third type of operation associated with the target operation preceding the timing sequence does not need to be executed, or when the third type of operation associated with the target operation preceding the timing sequence is executed, the second target sub-operation does not need to be executed; Determine a first parameter according to the product of a timing value corresponding to each candidate time window of the first target sub-operation and a first variable corresponding to the candidate time window, and a product of a timing value corresponding to each candidate time window of the second target sub-operation and a second variable corresponding to the candidate time window; the first parameter is used to indicate an execution time difference between the first target sub-operation and the second target sub-operation when the second engine executes the first target sub-operation and the second target sub-operation; The fifth sum of the first parameter and the second parameter is set to be greater than or equal to the second value, thereby obtaining a fifth constraint relationship; the second parameter is the product of the second value and the third variable; the fifth constraint relationship is used to limit the execution time difference between the first target sub-operation and the second target sub-operation to be greater than the second value; The first constraint condition is determined according to the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship.

10. The method according to claim 7, characterized in that The determining of the second constraint condition according to the first estimated memory amount and the second estimated memory amount in each time window and the memory amount of the first cache area includes: Determine, according to the product of the memory amount occupied by the result data of the target operation and the third variable, a third estimated memory amount occupied by the result data in the first buffer area in each candidate time window of the third type of operation associated with the target operation preceding in time sequence; Determine the second constraint condition according to the first estimated memory amount, the second estimated memory amount, the third estimated memory amount, and the memory amount of the first cache area in each time window; The second constraint condition is also used to limit the sum of the first estimated memory amount, the second estimated memory amount and the third estimated memory amount in each time window to be less than the memory amount of the first cache area.

11. The method according to any one of claims 4 to 10, characterized in that: The method further comprises: Determine, according to the execution timing information of the memory operation sequence, the first type of operation and / or the second type of operation to be executed in each time window in the time sequence; In each time window, allocating memory resources of the first cache area to first-category operations and / or second-category operations to be performed in the time window; After executing the first type of operation and / or the second type of operation based on the allocated memory resources, at least part of the memory resources allocated to the first type of operation and / or the second type of operation are released.

12. The method according to claim 11, characterized in that The method further comprises: If the memory resource allocation for the first type of operation and / or the second type of operation to be executed in a third time window in the time sequence fails, inserting a fourth time window in the time sequence; the fourth time window is a time window before the third time window; Determine a first storage location of first target data in the first cache area; wherein the first target data is operation data stored in the first cache area, and the first storage location of the first target data in the first cache area is not adjacent to a second storage location of operation data other than the first target data in the first cache area; Moving the first target data from the first storage position of the first cache area to the third storage position of the first cache area within the fourth time window; the third storage position is an idle storage position adjacent to the second storage position among a plurality of idle storage positions of the first cache area; After completing the movement of the first target data, memory resources are reallocated for the first type of operations and / or the second type of operations to be executed within the third time window.

13. The method according to claim 11, characterized in that The method further comprises: Determine the memory usage and remaining memory amount of the first buffer area in each time window according to the first type of operation and / or the second type of operation to be performed in each time window in the time series; Determine a target remaining memory amount according to the remaining memory amounts corresponding to multiple time windows in the time series; the target remaining memory amount is the minimum value of the remaining memory amounts corresponding to the multiple time windows; A resident storage area is set in the first buffer area; the memory amount of the resident storage area is determined by the target remaining memory amount, and the storage position of the resident storage area is located at the end of the first buffer area; Determine second target data from the operation data associated with each first-type operation to be executed in the memory operation sequence; the second target data is weight data of the first-type operation; Determine at least one third target data from the at least one second target data according to the amount of memory required by the second target data; the sum of the memory amounts of the at least one third target data is less than or equal to the memory amount of the resident storage area, and the memory amount of each third target data is less than the memory amount of the second target data other than the third target data in the at least one second target data; The at least one third target data is stored in the resident storage area.

14. The method according to any one of claims 4 to 10, characterized in that The method further comprises: In a case where there is at least one identical first-category operation in the memory operation sequence of the first kernel and the memory operation sequence of the second kernel, at least one identical fourth target data is determined from the first operation data and the second operation data based on the at least one identical first-category operation; wherein the first operation data is the operation data of the first sub-operation associated with the at least one identical first-category operation in the first kernel; and the second operation data is the operation data of the first sub-operation associated with the at least one identical first-category operation in the second kernel; Determine a first target time window and a second target time window according to a second time window of a first sub-operation associated with the fourth target data in the first kernel and a second time window of a first sub-operation associated with the fourth target data in the second kernel; a timing of the first target time window is earlier than any second time window of the second time window of the first kernel and the second time window of the second kernel; a timing of the second target time window is later than any second time window of the second time window of the first kernel and the second time window of the second kernel; Within the first target time window, storing the fourth target data in a second cache area; wherein the fourth target data stored in the second cache area can be shared by the first kernel and the second kernel; And within the second target time window, the memory resources occupied by the fourth target data are released in the second buffer area.

15. The method according to any one of claims 4 to 10, characterized in that The method further comprises: According to the second estimated duration of each time window in the time series, at least two third target time windows adjacent in time sequence are determined from multiple time windows in the time series; the second estimated duration of each third target time window is less than a preset duration threshold; According to the fourth target time window with the earliest time sequence among the at least two third target time windows adjacent in time sequence, a plurality of third target variables are determined from the second variables corresponding to the at least two third target time windows adjacent in time sequence; the third target variables are the second variables corresponding to the third target time windows other than the fourth target time window among the at least two third target time windows adjacent in time sequence; According to the fifth target time window with the latest timing among the at least two third target time windows adjacent in time sequence, a plurality of fourth target variables are determined from the first variables corresponding to the at least two third target time windows adjacent in time sequence; the fourth target variables are the first variables corresponding to the third target time windows other than the fifth target time window among the at least two third target time windows adjacent in time sequence; The values ​​of the plurality of third target variables and the plurality of fourth target variables are all set to be equal to the third value, so as to obtain a third constraint condition; wherein the third value of the third target variable is used to indicate that the second sub-operation is not performed within the candidate time window corresponding to the third target variable; and the third value of the fourth target variable is used to indicate that the first sub-operation is not performed within the candidate time window corresponding to the fourth target variable; The determining target values ​​of the plurality of decision variables based on the objective function and the first constraint condition and the second constraint condition includes: The target value is determined based on the objective function and the first constraint, the second constraint, and the third constraint.

16. A time sequence arrangement device, characterized in that: include: A first determination module is configured to determine, according to first time windows occupied by first type operations to be executed in a memory operation sequence in each time window of a time sequence, at least one candidate time window of a second type operation associated with each first type operation from each time window of the time sequence; the candidate time window of the second type operation is different from the first time window of the first type operation associated with the same second type operation; A first estimation module, configured to determine a first estimated duration of each time window in the time series according to a first time window corresponding to the first type of operation; A first estimated duration of the time window is a duration required for the first type of operation in the time window to be executed by the first engine; A second estimation module is configured to determine a second estimated duration of each time window in the time series according to at least one candidate time window of each second type of operation; A second estimated duration of the time window is a duration required for the second engine to execute the second type of operation within the time window; a second determination module, configured to determine, from at least one candidate time window of each second-category operation, a second time window corresponding to each second-category operation according to the first estimated duration and the second estimated duration of each time window; wherein, when the second-category operation is executed by the second engine in the corresponding second time window, the total execution delay corresponding to the memory operation sequence is less than the total execution delay corresponding to the memory operation sequence when the second-category operation is executed by the second engine in any candidate time window other than the corresponding second time window; and the total execution delay corresponding to the memory operation sequence is determined by the difference between the first estimated duration and the second estimated duration of each time window in the time sequence; The timing determination module is configured to determine the execution timing information of the memory operation sequence according to the first time window of each of the first type of operations and the second time window of the second type of operations associated with the first type of operations.

17. An electronic device, characterized in that: include: processor; Memory for storing computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing a computer program or instruction, characterized in that: When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images