Data processing apparatus, method, electronic device and storage medium

Abstract

This disclosure provides a data processing apparatus, relating to the field of artificial intelligence technology, particularly to the field of deep learning technology and chip technology. The apparatus includes: a storage unit comprising multiple initial storage spaces, each initial storage space corresponding to multiple tensors of a target model, wherein the multiple tensors include a current input tensor of the target model and output tensors of multiple operators; a processor configured to: read the current input tensor; determine the data size of the output tensors of the multiple operators based on the data size of the current input tensor; generate reallocation information in response to determining that the data size of the output tensor of an operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, wherein the reallocation information is used to indicate the reallocation of storage space for the tensors; and determine multiple target storage regions in the storage unit based on the reallocation information. This disclosure also provides a data processing method, an electronic device, and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of deep learning technology and chip technology. More specifically, this disclosure provides a data processing apparatus, method, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, deep learning models can include dynamic deep learning models and static deep learning models. The size (shape) of the input tensor in a static deep learning model can be fixed. The size of the input tensor in a dynamic deep learning model can be variable. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for multi-operator parallel processing of deep learning models.

[0004] According to one aspect of this disclosure, a data processing apparatus is provided, comprising: a storage unit including a plurality of initial storage spaces, the plurality of initial storage spaces corresponding to a plurality of tensors of a target model, wherein the target model includes a plurality of operators, and the plurality of tensors include a current input tensor of the target model and output tensors of each of the plurality of operators; a processor configured to: read the current input tensor; determine the data size of each output tensor of the plurality of operators based on the data size of the current input tensor; generate reallocation information in response to determining that the data size of the output tensor of an operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, wherein the reallocation information is used to indicate the reallocation of storage space for the tensors; and determine a plurality of target storage regions in the storage unit based on the reallocation information.

[0005] According to another aspect of this disclosure, a data processing method is provided, the method comprising: reading a current input tensor of a target model from a storage unit, wherein the storage unit includes a plurality of initial storage spaces, the plurality of initial storage spaces corresponding to a plurality of tensors of the target model, the target model including a plurality of operators, and the plurality of tensors including a current input tensor and output tensors of each of the plurality of operators; determining the data size of the output tensors of each of the plurality of operators based on the data size of the current input tensor; generating reallocation information in response to determining that the data size of the output tensor of an operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, wherein the reallocation information is used to indicate the reallocation of storage space for the tensors; and determining a plurality of target storage regions in the storage unit based on the reallocation information.

[0006] According to another aspect of this disclosure, an electronic device is provided, including the data processing apparatus provided in this disclosure.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided according to this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0013] Figure 2 This is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 3 This is a diagram illustrating the effect of model inference using input tensors of different batch sizes according to an embodiment of the present disclosure;

[0015] Figure 4 This is a schematic diagram of a data processing apparatus according to another embodiment of the present disclosure;

[0016] Figure 5 This is a block diagram of an electronic device according to an embodiment of the present disclosure;

[0017] Figure 6 This is a flowchart of a data processing method according to an embodiment of the present disclosure; and

[0018] Figure 7 This is a block diagram of an electronic device that can apply a multi-operator parallel processing method for deep learning models according to an embodiment of the present disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] Deep learning models can be categorized into static and dynamic deep learning models. The size of the input tensor in a static deep learning model can be fixed. For example, the dimension of the input tensor processed by the deep learning model can be the same in different scenarios. Therefore, during model initialization, the size of various intermediate tensors during the static deep learning model's operation can be efficiently determined, allowing for the allocation of storage space required for these intermediate tensors in the memory. Accelerated Linear Algebra (XLA), TVM, and TensorRT engines can provide excellent storage optimization for the model's intermediate tensors, better utilizing different levels of caching to achieve better inference or training results.

[0021] Compared to static deep learning models, dynamic deep learning models can adapt to a wider range of scenarios. However, the size of the input tensors in dynamic deep learning models is not fixed. Therefore, it is difficult to determine the size of various intermediate tensors during model initialization, making it difficult to optimize storage space before inference, and also difficult to effectively utilize caches or efficiently use global storage units.

[0022] In some embodiments, storage space can be determined for tensors based on the maximum batch size input by the user. Thus, the storage space for each intermediate tensor can be the same, corresponding to the maximum batch size. However, the storage space required for intermediate tensors can be less than the space corresponding to the maximum batch size. Allocating space for tensors according to the maximum batch size can lead to decreased storage utilization.

[0023] In some embodiments, storage space can be dynamically allocated and dynamically freed based on the size of the tensor. However, dynamically allocating or freeing space can lead to storage fragmentation. Furthermore, frequent allocation and freeing of storage space also reduces inference efficiency.

[0024] To efficiently run dynamic deep learning models, this disclosure provides a data processing apparatus, which will be described in detail below.

[0025] Figure 1 This is a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0026] like Figure 1 As shown, the device 100 may include a storage unit 110 and a processor 120.

[0027] Storage unit 11O includes multiple initial storage spaces.

[0028] In embodiments of this disclosure, multiple initial storage spaces correspond to multiple tensors of the target model. For example, initial storage spaces can be determined for multiple tensors in various ways.

[0029] In this embodiment of the disclosure, the storage unit may include at least one of a Level 3 cache unit (L3) and a global memory unit.

[0030] In embodiments of this disclosure, the target model may include multiple operators. The multiple tensors may include the current input tensor of the target model and the respective output tensors of the multiple operators. For example, the target model may be a dynamic deep learning model. The size of the current input tensor may be indeterminate.

[0031] Processor 120 can be configured to read the current input tensor. For example, the current input tensor can be read from storage unit 110.

[0032] The processor 120 can also be configured to determine the data size of the output tensors of multiple operators based on the data size of the current input tensor. For example, the data size of the output tensors of multiple operators can be determined using a preset function based on the data size of the current input tensor. The preset function can be a scale derivation (infershape) function.

[0033] The processor 120 can also be configured to generate reallocation information in response to determining that the data size of the output tensor of an operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator. For example, the reallocation information may indicate the reallocation of storage space for the tensor. For an operator in the target model, if the data size of the output tensor of the operator is greater than the capacity of the operator's initial storage space, it can be determined that the capacity of the initial storage space is insufficient, and reallocation information can be generated to reallocate storage space for the output tensor.

[0034] The processor 120 can also be configured to determine multiple target storage regions in the storage units based on redistribution information. For example, based on the redistribution information, at least one instruction corresponding to the storage space of the redistributed tensor can be executed to determine the multiple target storage regions. Each target storage region can store the output tensor of at least one operator. The number of target storage regions can be less than or equal to the number of output tensors of all operators.

[0035] In this embodiment, when the data size of the operator's output tensor exceeds the capacity of the initial storage space corresponding to the operator's output tensor, the tensor size space is reallocated, which optimizes storage utilization efficiency. Furthermore, it eliminates the need for users to set maximum batch size parameters, reducing manual costs. After reallocating the storage area, if the size of subsequent input tensors of the current input tensor remains unchanged, the output tensors of multiple operators corresponding to those subsequent input tensors can be stored in the target storage area, reducing the number of times storage space is reallocated and minimizing the resource overhead required for reallocation.

[0036] As can be understood, the data processing apparatus of this disclosure has been described above, and some methods by which the processor generates redistribution information will be described below.

[0037] Figure 2 This is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0038] like Figure 2 As shown, the processor can be configured to execute at least one instruction to implement operation S201. In operation S201, it is determined whether the data size of the operator's output tensor is deterministic.

[0039] In this embodiment of the disclosure, when the data size of the output tensors of each of the multiple operators in the target model is determined, the processor can be configured to execute at least one instruction to implement operation S202. In operation S202, multiple storage spaces are determined based on the data size of the output tensors of each of the multiple operators. For example, when the target model is a static deep learning model, the data size of the output tensors is known. Therefore, storage spaces can be determined separately for the output tensors of each of the multiple operators.

[0040] In this embodiment of the disclosure, when the data volume of the output tensors of the multiple operators of the target model is uncertain, the processor can be configured to execute at least one instruction to implement operation S210. In operation S210, the current input tensor is read. For example, the current input tensor can be read from a storage unit. After reading the current input tensor, the data volume of the current input tensor can be determined.

[0041] In this embodiment of the disclosure, the processor can also be configured to: determine the data size of the output tensor of each of the multiple operators based on the data size of the current input tensor. For example, the processor can be configured to execute at least one instruction to implement operation S220. In operation S220, the data size of the output tensor of the current operator of the target model is determined. For example, taking the current operator as the first operator of the target model as an example, the data size of the output tensor of the current operator can be determined based on the current input tensor using a preset function.

[0042] In some embodiments, the processor may also be configured to generate reallocation information. This will be described below in conjunction with operations S231 to S236.

[0043] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S231. In operation S231, it is determined whether the data size of the output tensor of the current operator is greater than the initial storage space corresponding to the output tensor of the current operator. For example, the following explanation further illustrates this by assuming that the data size of the output tensor of the first operator is greater than the initial storage space corresponding to the output tensor of the first operator.

[0044] In this embodiment of the disclosure, in response to determining that the data size of the output tensor of the current operator is greater than the initial storage space corresponding to the output tensor of the current operator, the processor may further be configured to release the initial storage space corresponding to the output tensor of the operator. For example, the processor may be configured to execute at least one instruction to implement operation S232. In operation S232, the initial storage space corresponding to the output tensor of the current operator is released. The initial storage space corresponding to the first operator may be located in a level 3 cache unit. The initial storage space located in the level 3 cache unit may be released.

[0045] In this embodiment of the disclosure, the processor may further be configured to: determine, in a storage unit, a temporary storage space corresponding to the output tensor of the operator, based on the data size of the output tensor of the operator. For example, the processor may be configured to execute at least one instruction to implement operation S233. In operation S233, a temporary storage space corresponding to the output tensor of the current operator is determined. A first temporary storage space corresponding to the output tensor of the first operator may be determined in the global storage unit.

[0046] In this embodiment of the disclosure, the processor can also be configured to generate redistribution information. For example, the processor can be configured to execute at least one instruction to implement operation S234. In operation S234, redistribution information is generated. The target model can correspond to the redistribution variable value. The redistribution variable value is false by default. The redistribution variable value can be set to true to generate redistribution information.

[0047] In this embodiment of the disclosure, the processor can also be configured to: determine the output tensors of each of the multiple operators in the current processing cycle based on the current input tensor. For example, the processor can be configured to execute at least one instruction to implement operation S235. In operation S235, the current operator is run. The first operator can be run to obtain the output tensor of the first operator. This output tensor can be written to the first temporary storage space described above.

[0048] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S236. In operation S236, it is determined whether all operators have completed execution. If the target model includes multiple operators, after the first operator completes execution, it can be determined that all operators have not completed execution. The process can then return to operation S231, using the second operator as the current operator, and execute operations S231 through S236 until all operators have completed execution. If the data size of the output tensor of the second operator is also greater than its corresponding initial storage space, a second temporary storage space corresponding to the output tensor of the second operator can be determined in the global storage unit.

[0049] As can be understood, some methods for generating reallocation information have been described above. In some embodiments, the processor may also be configured to determine multiple target storage regions in the storage unit based on the reallocation information. The following will describe this in conjunction with operations S241 to S243.

[0050] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S241. In operation S241, it is determined whether to reallocate memory space. For example, if the reallocation variable is detected to be true, memory space may be reallocated.

[0051] In this embodiment of the disclosure, the processor may further be configured to write the output tensors of each of the multiple operators into a first preset storage area according to the reallocation information. For example, the processor may also be configured to execute at least one instruction to implement operation S242. In operation S242, the output tensors of each of the multiple operators are written into the first preset storage area. The first preset storage area may include multiple first preset storage spaces. The first preset storage area may be located in a global storage unit. The first preset storage area may temporarily store the output variables of all operators.

[0052] In this embodiment of the disclosure, the processor can also be configured to determine multiple target storage regions in the storage unit based on the data volume of the output tensors of each of the multiple operators. For example, the processor can also be configured to execute at least one instruction to implement operation S243. In operation S243, multiple target storage regions are determined in the storage unit. After the above-described operations S231 to S236 are executed, at least one initial storage space and at least one temporary storage space may exist in the L3 cache. These storage spaces can be released. Next, the multiple target storage regions can be reallocated in the L3 cache.

[0053] In this embodiment of the disclosure, the processor can also be configured to write the output tensors of each of the multiple operators from a first preset storage area to multiple target storage areas. For example, the processor can also be configured to execute at least one instruction to implement operation S250. In operation S250, the output tensors of each of the multiple operators are written to the target storage areas. As described above, the first preset storage unit can temporarily store all output tensors. After the target storage areas are reallocated, the output tensors of each of the multiple operators can be restored to the multiple target storage areas. Through this embodiment of the disclosure, by first temporarily storing the tensors and then restoring the tensors to the reallocated target storage areas, all tensor data can be efficiently saved, and data loss can be effectively prevented.

[0054] In this embodiment of the disclosure, the processor can also be configured to: process the subsequent input tensor of the current input tensor using the target model to obtain the output tensors of multiple operators in the subsequent processing cycle of the current processing cycle. The output tensors of the multiple operators in the subsequent processing cycle are written into multiple target storage areas. For example, the processor can also be configured to execute at least one instruction to implement operation S260. In operation S260, the subsequent input tensor of the current input tensor is processed using the target model. The data size of the subsequent input tensor can be the same as the data size of the current input tensor. During model inference or model training using the subsequent input tensor, the output tensors of the operators can be written into the aforementioned target storage areas.

[0055] In other embodiments of this disclosure, if the value of a reallocated variable is detected as false, the storage space may be reallocated. For example, if the output variable of any operator is less than or equal to the initial storage space corresponding to that operator, reallocation may not be performed, and operation S260 may be executed directly. After the reallocation is completed, the value of the reallocated variable may be restored to its default value.

[0056] In other embodiments of this disclosure, multiple input tensors with consistent data volumes can be used for model inference or training. When the first input tensor is used as the current input tensor, storage space can be reallocated during the processor's execution of the above operations. When the second input tensor is used as the current input tensor, reallocation information may not be generated; instead, the operator's output tensor can be stored using multiple target storage regions allocated based on the first input tensor. Alternatively, operation S241 can be executed directly followed by operation S260.

[0057] It is understood that the above description uses the example of operations S220 and S231 to S236 being executed sequentially to illustrate this disclosure. However, this disclosure is not limited to this; operations S231 to S236 can also be executed after multiple executions of operation S220 to obtain the data volume of the output tensors of all operators.

[0058] It is understood that the above description uses multiple tensors, including the output tensor of an operator, as an example to illustrate this disclosure. However, this disclosure is not limited to this; the multiple tensors may also include multiple preset tensors for each operator. For example, the preset tensor of an operator may include a weight tensor. In dynamic deep learning models, the data size of the preset tensor of an operator may be fixed. In some embodiments, the target storage area of ​​the preset tensor may also be determined based on redistribution information.

[0059] It is understandable that multiple input tensors with the same batch size can be used for multiple model training or inference operations. When using the first input tensor for model training or inference, reallocation information can be generated to determine the target storage region. When using input tensors with the same batch size as the first input tensor for model training or inference, multiple target storage regions allocated based on the first input tensor can be used to store the output tensor.

[0060] It can also be understood that during model training or inference using input tensors of different batch sizes, as the batch size increases, the data volume of the model's input tensors can also increase, as can the data volume of the output tensors of multiple operators. This may result in multiple reallocation messages, leading to multiple determinations of the target storage space. The following will combine... Figure 3 Please provide an explanation.

[0061] Figure 3 This is a diagram illustrating the effect of model inference using input tensors of different batch sizes according to an embodiment of the present disclosure.

[0062] Model inference can be performed using the first, second, third, and fourth input tensors respectively. The size of the first input tensor can be [1, 3, 224, 224]. For the first input tensor, the first batch size (batch 1) is 1, the number of channels is 3, the data height is 224, and the data width is 224. For the second input tensor, the second batch size (batch 2) is 2, the number of channels is 3, the data height is 224, and the data width is 224. For the third input tensor, the third batch size (batch 4) is 4, the number of channels is 3, the data height is 224, and the data width is 224. For the fourth input tensor, the fourth batch size (batch 8) is 8, the number of channels is 3, the data height is 224, and the data width is 224. It can be understood that the first to fourth input tensors can correspond to RGB images respectively. In this embodiment, the operator's preset tensor can be stored in the initial storage space. When determining the target storage area based on the redistribution information, the target storage area can also be determined for the preset tensor.

[0063] Multiple inferences can be performed using different first input tensors. During the first inference (batch 1, 1st infer) using the first input tensor, redistribution information is generated, identifying multiple target storage regions in the storage unit. Thus, the first inference takes 194.34 milliseconds (ms). During the second and third inferences, the data size of the first input tensor remains unchanged, and these multiple target storage regions can be used to store the output tensor. The inference time for the second inference is approximately 2.508 milliseconds. The inference time for the third inference is approximately 2.357 milliseconds.

[0064] Next, during the first inference (batch 2, 1st infer) using the second input tensor, the second batch size (2) of the second input tensor is larger than the first batch size (1) of the first input tensor. This allows for the generation of redistribution information, and multiple target storage regions are redefined in the storage unit. Therefore, the first inference takes 153.228 milliseconds. During the second and third inference processes, the data size of the second input tensor remains unchanged, and these multiple target storage regions can be used to store the output tensor. The inference time for the second inference is approximately 3.604 milliseconds. The inference time for the third inference is approximately 3.451 milliseconds.

[0065] Next, during the first inference (batch 4, 1st infer) using the third input tensor, the third batch size (4) of the third input tensor is larger than the second batch size (2) of the second input tensor. This allows for the generation of redistribution information, and multiple target storage regions are redefined in the storage unit. Therefore, the first inference takes 148.431 milliseconds. During the second and third inference processes, the data volume of the third input tensor remains unchanged, and these multiple target storage regions can be used to store the output tensor. The inference time for the second inference is approximately 5.694 milliseconds. The inference time for the third inference is approximately 5.604 milliseconds.

[0066] Next, during the first inference (batch 8, 1stinfer) using the fourth input tensor, the fourth batch size (8) of the fourth input tensor is larger than the third batch size (4) of the third input tensor. This allows for the generation of redistribution information, and multiple target storage regions are redefined in the storage unit. Therefore, the first inference takes 172.809 milliseconds. During the second and third inference processes, the data size of the fourth input tensor remains unchanged, and the output tensor can be stored using the redistributed target storage regions. The inference time for the second inference is approximately 10.168 milliseconds. The inference time for the third inference is approximately 10.368 milliseconds.

[0067] It is understandable that when performing multiple model inferences using input tensors of the same batch size, the input tensors can be different. For example, three different first input tensors can be used for three inferences.

[0068] like Figure 3 As shown, as the size of the input tensor increases, the storage space required for the operator's output tensor also increases, necessitating reallocation and resulting in a longer inference time for the first inference. This phenomenon can be called the glitch phenomenon.

[0069] To mitigate glitches, the storage unit may further include multiple second preset storage spaces. The second preset storage spaces are configured to store preset tensors of operators. In embodiments of this disclosure, the preset tensors may have target parameter values. The following will describe... Figure 4 Further explanation is needed.

[0070] Figure 4 This is a schematic diagram of a data processing apparatus according to another embodiment of the present disclosure.

[0071] like Figure 4 As shown, the processor can be configured to execute at least one instruction to implement operation S401'. In operation S401', it is determined whether the data size of the operator's tensor is deterministic.

[0072] In this embodiment of the disclosure, when the data size of the output tensors and preset tensors of the multiple operators in the target model is determined, the processor can be configured to execute at least one instruction to implement operation S402'. In operation S402', multiple storage spaces are determined based on the data size of the tensors of the multiple operators. For example, when the target model is a static deep learning model, the data size of the output tensors is known. Therefore, storage spaces can be determined for the output tensors and preset tensors of the multiple operators respectively.

[0073] In this embodiment of the disclosure, when the data volume of the output tensors of the multiple operators of the target model is uncertain, the processor can be configured to execute at least one instruction to implement operation S403. In operation S403, the preset tensors of the multiple operators are written into multiple second preset storage spaces. For example, the multiple second preset storage spaces can be located in a level 3 cache unit or in a global storage unit.

[0074] In this embodiment of the disclosure, the processor may be configured to execute at least one instruction to implement operation S410. In operation S410, the current input tensor is read. For example, the current input tensor may be read from a memory unit. After reading the current input tensor, the amount of data in the current input tensor can be determined.

[0075] In this embodiment of the disclosure, the processor can also be configured to: determine the data size of the output tensor of each of the multiple operators based on the data size of the current input tensor. For example, the processor can be configured to execute at least one instruction to implement operation S420. In operation S420, the data size of the output tensor of the current operator of the target model is determined. For example, taking the current operator as the first operator of the target model as an example, the data size of the output tensor of the current operator can be determined based on the current input tensor using a preset function.

[0076] In some embodiments, the processor may also be configured to generate reallocation information. This will be described below in conjunction with operations S431 to S436.

[0077] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S431. In operation S431, it is determined whether the data size of the output tensor of the current operator is greater than the initial storage space corresponding to the output tensor of the current operator. For example, the data size of the output tensor of the first operator may be greater than the capacity of the initial storage space corresponding to the output tensor of the first operator.

[0078] In this embodiment of the disclosure, in response to determining that the data size of the output tensor of the current operator is greater than the initial storage space corresponding to the output tensor of the current operator, the processor may further be configured to release the initial storage space corresponding to the output tensor of the operator. For example, the processor may be configured to execute at least one instruction to implement operation S432. In operation S432, the initial storage space corresponding to the output tensor of the current operator is released. The initial storage space corresponding to the first operator may be located in a level 3 cache unit. The initial storage space located in the level 3 cache unit may be released.

[0079] In this embodiment of the disclosure, the processor can also be configured to: determine a temporary storage space corresponding to the output tensor of the operator in a storage unit based on the data size of the output tensor of the operator. For example, the processor can be configured to execute at least one instruction to implement operation S433. In operation S433, a temporary storage space corresponding to the output tensor of the current operator is determined. A first temporary storage space corresponding to the output tensor of the first operator can be determined in the global storage unit.

[0080] In this embodiment of the disclosure, the processor may also be configured to generate redistribution information. For example, the processor may be configured to execute at least one instruction to implement operation S434. In operation S434, redistribution information is generated. The target model may correspond to the redistribution variable value. The redistribution variable value may be set to true to generate redistribution information.

[0081] In this embodiment of the disclosure, the processor can also be configured to: determine the output tensors of each of the multiple operators in the current processing cycle based on the current input tensor. For example, the processor can be configured to execute at least one instruction to implement operation S435'. In operation S435', the current operator is run.

[0082] Unlike operation S235, in this embodiment, the processor can also be configured to: read the preset tensor of the operator from the second preset storage space in response to determining that the target parameter value of the preset tensor of the operator is greater than or equal to a preset parameter threshold. For example, the preset parameter threshold can be 1. If model inference or model training is performed using three input tensors with the same amount of data, and the current input tensor is the first input tensor, the target parameter value can be 3. When the target parameter value is greater than the preset parameter threshold, the preset tensor of the first operator can be read from the second preset storage space. Through this embodiment, the preset tensor is stored in the second preset storage space, and a target parameter value is set for the preset tensor. Thus, when the target parameter value is greater than the preset parameter threshold, the preset tensor can be efficiently saved, preventing the loss of tensors such as weight tensors. In addition, it can also save the resource overhead and time cost required to reallocate space for the preset tensor, which helps to improve the dynamic inference capability of the data processing device.

[0083] In this embodiment of the disclosure, the processor can also be configured to: process the input tensor of the operator using a preset tensor of the operator to obtain the output tensor of the operator. For example, if the preset tensor is a weight tensor, the input tensor of the operator can be weighted using the preset tensor to obtain the output tensor of the current operator. This output tensor can be written into the first temporary storage space described above.

[0084] In this embodiment of the disclosure, the processor can also be configured to: update the target parameter value using a preset processing parameter. For example, the preset processing parameter can be 1. The processed target parameter value is obtained by subtracting the preset parameter value from the target parameter value.

[0085] In another embodiment of this disclosure, the processor may further be configured to: release a second preset storage space corresponding to the preset tensor in response to determining that the target parameter value of the preset tensor of the operator is less than a preset parameter threshold. For example, if three input tensors with the same amount of data are used for model inference or model training, after using the third input tensor for model training or inference, the target parameter is adjusted to 0, which is less than the preset parameter threshold, and the second preset storage space corresponding to the preset tensor can be released.

[0086] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S436. In operation S436, it is determined whether all operators have completed their execution. If the target model includes multiple operators, after the first operator has completed its execution, it can be determined that all operators have not completed their execution. The process can then return to operation S431, using the second operator as the current operator, and execute operations S431 through S436 until all operators have completed their execution.

[0087] As can be understood, some methods for generating reallocation information have been described above. In some embodiments, the processor may also be configured to determine multiple target storage regions in the storage unit based on the reallocation information. This will be explained below in conjunction with operations S441 to S443.

[0088] In this embodiment of the disclosure, the processor may also be configured to execute at least one instruction to implement operation S441. In operation S441, it is determined whether to reallocate memory space. For example, if the reallocation variable is detected to be true, memory space may be reallocated.

[0089] In this embodiment of the disclosure, the processor may further be configured to write the output tensors of each of the multiple operators into a first preset storage area according to the reallocation information. For example, the processor may also be configured to execute at least one instruction to implement operation S442. In operation S442, the output tensors of each of the multiple operators are written into the first preset storage area. The first preset storage area may include multiple first preset storage spaces. The first preset storage area may be located in a global storage unit. The first preset storage area may temporarily store the output variables of all operators.

[0090] In this embodiment of the disclosure, the processor can also be configured to determine multiple target storage regions in the storage unit based on the data volume of the output tensors of each of the multiple operators. For example, the processor can also be configured to execute at least one instruction to implement operation S443. In operation S443, multiple target storage regions are determined in the storage unit. After the above-described operations S431 to S436 are executed, at least one initial storage space and at least one temporary storage space may exist in the L3 cache. These storage spaces can be released. Next, multiple target storage regions can be reallocated in the L3 cache unit. It is understood that the preset tensors of the multiple operators are stored in multiple second preset storage spaces. Target storage regions can be determined only for the output tensors. This reduces the resource overhead required for reallocation and helps to improve the performance of the data processing device.

[0091] In this embodiment of the disclosure, the processor may further be configured to write the output tensors of each of the plurality of operators from a first preset storage area to a plurality of target storage areas. For example, the processor may also be configured to execute at least one instruction to implement operation S450. In operation S450, the output tensors of each of the plurality of operators are written to the target storage areas.

[0092] In embodiments of this disclosure, the processor may further be configured to: process the subsequent input tensors of the current input tensor using the target model to obtain the output tensors of multiple operators in subsequent processing cycles of the current processing cycle. The output tensors of the multiple operators in the subsequent processing cycles are written to multiple target storage areas. For example, the processor may also be configured to execute at least one instruction to implement operation S460. In operation S460, the subsequent input tensors of the current input tensor are processed using the target model. It is understood that the descriptions of operations S450 and S460 can be found in the detailed descriptions of operations S250 and S260 described above, and will not be repeated here.

[0093] It is understood that the above description uses the example of operations S420 and S431 to S436 being executed sequentially to illustrate this disclosure. However, this disclosure is not limited to this; operations S431 to S436 can also be executed after multiple executions of operation S420 to obtain the data volume of the output tensors of all operators.

[0094] It is understood that the processor of this disclosure has been further described above, and the target storage area of ​​this disclosure will be further described below.

[0095] In this embodiment of the disclosure, the target storage region includes multiple target storage spaces. Each target storage space is configured to store the output tensors of operators. There are no dependencies between the output tensors of the multiple operators stored within the target storage region. A dependency can be characterized by the fact that the output tensor of a subsequent operator is obtained by inputting the output tensor of the current operator into the subsequent operator. For example, the output tensor of the first operator can be used as the input tensor of the second and fourth operators, respectively. The output tensor of the second operator can be used as the input tensor of the third operator. The output tensor of the third operator can also be used as the input tensor of the fourth operator. When the current operator is the first operator, there is a dependency between the output tensor of the second operator and the output tensor of the first operator, and a dependency between the output tensor of the fourth operator and the output tensor of the first operator. The output tensor of the first operator can be stored in the first target storage space of the first target storage region. The output tensor of the second operator can be stored in the second target storage space of the second target storage region. The output tensor of the third operator can be stored in the third target storage space of the third target storage region. The output tensor of the fourth operator can be stored in the fourth target storage space of the second target storage region. There is no dependency between the output tensors of the second and fourth operators in the second target storage region.

[0096] In embodiments of this disclosure, the processor can further be configured to: in response to determining that all output tensors that are dependent on the output tensor of the target operator are stored in at least one target storage region, release the target storage space of the output tensor of the target operator. For example, if the output tensors of the second operator and the fourth operator already exist in the second target storage region, the aforementioned first target storage space can be released. Through embodiments of this disclosure, storage space can be dynamically released, which helps to achieve efficient reuse of storage space and further optimize the utilization efficiency of storage space.

[0097] It is understood that the target storage area of ​​this disclosure has been described above, and the effects of the data processing device of this disclosure will be further explained below.

[0098] In some embodiments, a preset tensor can be stored in a second preset storage space. When determining the target storage region based on the reallocation information, a target storage region can also be determined for the output tensor, while the preset tensor remains stored in the second preset storage space.

[0099] During the first inference using the first input tensor, redistribution information is generated, identifying multiple target storage regions within the storage unit. Therefore, the first inference takes 7.6 milliseconds. During the second and third inferences, the data size of the first input tensor remains unchanged, and the output tensor can be stored using the redistributed target storage regions. The inference time for the second and third inferences is approximately 2.5 milliseconds.

[0100] During the first inference using the second input tensor, redistribution information is generated, identifying multiple target storage regions within the storage unit. Therefore, the first inference takes 8.5 milliseconds. During the second and third inference processes, the data size of the second input tensor remains unchanged, and the output tensor can be stored using the redistributed target storage regions. The inference time for the second and third inference processes is approximately 3.6 milliseconds.

[0101] During the first inference using the third input tensor, redistribution information is generated, identifying multiple target storage regions within the storage unit. Therefore, the first inference takes 6.1 milliseconds. During the second and third inference processes, the data size of the third input tensor remains unchanged, and the output tensor can be stored using the redistributed target storage regions. The inference time for the second and third inference processes is approximately 5.7 milliseconds.

[0102] During the first inference using the fourth input tensor, redistribution information is generated, identifying multiple target storage regions within the storage unit. Therefore, the first inference takes 10.91 milliseconds. During the second and third inferences, the data size of the fourth input tensor remains unchanged, and the output tensor can be stored using the redistributed target storage regions. The inference time for the second and third inferences is approximately 10.8 milliseconds.

[0103] Through the embodiments of this disclosure, the time required for the first inference is significantly reduced during model inference using different input tensors with consistent data volume. Therefore, storing the preset tensor in a second preset storage space can effectively alleviate glitches.

[0104] It is understood that the target model mentioned above is a dynamic deep learning model. In some other embodiments, the target model is set as a static deep learning model with the same number of parameters as the dynamic deep learning model, and the static deep learning model is used to perform model inference using the first to fourth input tensors respectively. For example, in the process of performing three model inferences using the first input tensor, the inference time required by the static deep learning model is 2.4 milliseconds each. In the process of performing three model inferences using the second input tensor, the inference time required by the static deep learning model is 3.4 milliseconds each. In the process of performing three model inferences using the third input tensor, the inference time required by the static deep learning model is 5.6 milliseconds each. In the process of performing three model inferences using the fourth input tensor, the inference time required by the static deep learning model is 10.1 milliseconds each.

[0105] Therefore, by storing the preset tensor in the second preset storage space, the resource overhead and time cost required for reallocation are greatly reduced, the inference time of the dynamic deep learning model is greatly reduced, and the inference capability is close to that of the static deep learning model.

[0106] It is understood that the data processing apparatus of this disclosure has been described above, and the electronic equipment including the data processing apparatus will be described below.

[0107] Figure 5 This is a schematic block diagram of an electronic device according to another embodiment of the present disclosure.

[0108] like Figure 5 As shown, the device 50 may include a data processing unit 500. The data processing unit 500 may be the data processing unit 100 described above.

[0109] It is understood that the above text has described an electronic device of this disclosure, and the following text will describe the data processing method of this disclosure.

[0110] Figure 6 This is a flowchart of a data processing method according to an embodiment of the present disclosure.

[0111] like Figure 6 As shown, method 600 may include operations S610 to S640.

[0112] In operation S610, the current input tensor of the target model is read from the storage unit.

[0113] In this embodiment of the disclosure, the storage unit includes multiple initial storage spaces. These multiple initial storage spaces correspond to multiple tensors of the target model. The target model includes multiple operators. The multiple tensors include a current input tensor and the output tensors of each of the multiple operators.

[0114] In operation S620, the data volume of the output tensor of each of the multiple operators is determined based on the data volume of the current input tensor.

[0115] In operation S630, in response to the determination that the data size of the output tensor of the operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, reallocation information is generated.

[0116] In this embodiment of the disclosure, the reallocation information is used to indicate the reallocation of storage space for tensors.

[0117] In operation S640, multiple target storage regions are determined in the storage unit based on the reallocation information.

[0118] It is understandable that the processor 120 described above can be used to execute multiple instructions to implement method 600.

[0119] In some embodiments, in response to determining that the data size of the operator's output tensor is greater than the capacity of the initial storage space corresponding to the operator's output tensor, generating reallocation information includes: releasing the initial storage space corresponding to the operator's output tensor in response to determining that the data size of the operator's output tensor is greater than the capacity of the initial storage space corresponding to the operator's output tensor; determining a temporary storage space corresponding to the operator's output tensor in the storage unit based on the data size of the operator's output tensor; and generating reallocation information.

[0120] In some embodiments, the method further includes: determining the output tensors of each of the multiple operators in the current processing cycle based on the current input tensor.

[0121] In some embodiments, determining multiple target storage regions in the storage unit based on reallocation information includes: writing the output tensors of each of the multiple operators into a first preset storage region based on the reallocation information. For example, the first preset storage region includes multiple first preset storage spaces. Multiple target storage regions are determined in the storage unit based on the data size of the output tensors of each of the multiple operators.

[0122] In some embodiments, the method further includes: writing the output tensors of the plurality of operators from a first preset storage area to a plurality of target storage areas.

[0123] In some embodiments, the storage unit further includes a plurality of second preset storage spaces, and the plurality of tensors further includes preset tensors for each of the plurality of operators. The second preset storage spaces are configured to store the preset tensors of the operators. Determining the output tensors of the plurality of operators in the current processing cycle based on the current input tensor includes: reading the preset tensor of the operator from the second preset storage space in response to determining that the target parameter value of the preset tensor of the operator is greater than or equal to a preset parameter threshold; processing the input tensor of the operator using the preset tensor of the operator to obtain the output tensor of the operator; and updating the target parameter value using preset processing parameters.

[0124] In some embodiments, determining the output tensors of the multiple operators in the current processing cycle based on the current input tensor further includes: releasing a second preset storage space corresponding to the preset tensor in response to determining that the target parameter value is less than a preset parameter threshold.

[0125] In some embodiments, the method further includes: processing the subsequent input tensors of the current input tensor using the target model to obtain the output tensors of multiple operators in subsequent processing cycles of the current processing cycle. The output tensors of the multiple operators in the subsequent processing cycles are then written into multiple target storage areas.

[0126] In some embodiments, the target storage region includes multiple target storage spaces configured to store the output tensors of operators. There are no dependencies between the output tensors of the multiple operators stored within the target storage region. Dependencies are used to characterize the fact that the output tensor of a subsequent operator is obtained by inputting the output tensor of the current operator into the subsequent operator.

[0127] In some embodiments, the method further includes: in response to determining that all output tensors that are dependent on the output tensors of the target operator are stored in at least one target storage region, releasing the target storage space used to store the output tensors of the target operator.

[0128] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0132] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) monitor or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing apparatus, comprising: The storage unit includes multiple initial storage spaces and multiple second preset storage spaces. The multiple initial storage spaces correspond to multiple tensors of the target model. The target model includes multiple operators. The multiple tensors include the current input tensor of the target model, the output tensors of each of the multiple operators, and the preset tensors of each of the multiple operators. The second preset storage spaces are configured to store the preset tensors of the operators. The preset tensors of the operators include the weight tensors of the operators. The processor is configured as follows: Read the current input tensor; Based on the data volume of the current input tensor, determine the data volume of the output tensor of each of the multiple operators; In response to determining that the data size of the output tensor of the operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, the initial storage space corresponding to the output tensor of the operator is released; Based on the data volume of the output tensor of the operator, a temporary storage space corresponding to the output tensor of the operator is determined in the storage unit; Generate reallocation information, wherein the reallocation information is used to indicate the reallocation of storage space for tensors; Based on the reallocation information, multiple target storage regions are determined in the storage unit; and Determining the output tensors of each of the multiple operators in the current processing cycle based on the current input tensor includes: in response to determining that the target parameter value of the preset tensor of the operator is greater than or equal to a preset parameter threshold, reading the preset tensor of the operator from the second preset storage space; processing the input tensor of the operator using the preset tensor of the operator to obtain the output tensor of the operator; and updating the target parameter value using preset processing parameters.

2. The apparatus according to claim 1, wherein, The processor is also configured to: According to the redistribution information, the output tensors of each of the multiple operators are written into a first preset storage area, wherein the first preset storage area includes multiple first preset storage spaces; Based on the data volume of the output tensors of each of the multiple operators, a plurality of target storage regions are determined in the storage unit.

3. The apparatus according to claim 2, wherein, The processor is also configured to: The output tensors of each of the multiple operators are written from the first preset storage area to the multiple target storage areas.

4. The apparatus according to claim 1, wherein, The processor is also configured to: In response to determining that the target parameter value is less than the preset parameter threshold, the second preset storage space corresponding to the preset tensor is released.

5. The apparatus according to claim 1, wherein, The processor is also configured to: The target model is used to process the subsequent input tensors of the current input tensor to obtain the output tensors of each of the operators in the subsequent processing cycles of the current processing cycle. The output tensors of the various operators in the post-processing cycle are written into the various target storage areas.

6. The apparatus according to claim 1, wherein, The target storage region includes multiple target storage spaces, and the target storage spaces are configured to store the output tensors of the operator; There is no dependency relationship between the output tensors of the multiple operators stored in the target storage area. The dependency relationship is used to characterize that the output tensor of the subsequent operator of the current operator is obtained by inputting the output tensor of the current operator into the subsequent operator.

7. The apparatus according to claim 6, wherein, The processor is also configured to: In response to determining that at least one of the target storage regions has stored all output tensors that have the dependency relationship with the output tensor of the target operator, the target storage space used to store the output tensors of the target operator is released.

8. A data processing method, comprising: The current input tensor of the target model is read from the storage unit, wherein the storage unit includes multiple initial storage spaces and multiple second preset storage spaces, the multiple initial storage spaces correspond to multiple tensors of the target model, the target model includes multiple operators, the multiple tensors include the current input tensor, the output tensors of each of the multiple operators, and the preset tensors of each of the multiple operators, the second preset storage space is configured to store the preset tensors of the operators, and the preset tensors of the operators include the weight tensors of the operators; Based on the data volume of the current input tensor, determine the data volume of the output tensor of each of the multiple operators; In response to determining that the data size of the output tensor of the operator is greater than the capacity of the initial storage space corresponding to the output tensor of the operator, the initial storage space corresponding to the output tensor of the operator is released; Based on the data volume of the output tensor of the operator, a temporary storage space corresponding to the output tensor of the operator is determined in the storage unit; Generate reallocation information, wherein the reallocation information is used to indicate the reallocation of storage space for tensors; and Based on the reallocation information, multiple target storage regions are determined in the storage unit; Determining the output tensors of each of the multiple operators in the current processing cycle based on the current input tensor includes: in response to determining that the target parameter value of the preset tensor of the operator is greater than or equal to a preset parameter threshold, reading the preset tensor of the operator from the second preset storage space; processing the input tensor of the operator using the preset tensor of the operator to obtain the output tensor of the operator; and updating the target parameter value using preset processing parameters.

9. The method according to claim 8, wherein, The step of determining multiple target storage regions in the storage unit based on the reallocation information includes: According to the redistribution information, the output tensors of each of the multiple operators are written into a first preset storage area, wherein the first preset storage area includes multiple first preset storage spaces; Based on the data volume of the output tensors of each of the multiple operators, a plurality of target storage regions are determined in the storage unit.

10. The method of claim 9, further comprising: The output tensors of each of the multiple operators are written from the first preset storage area to the multiple target storage areas.

11. The method according to claim 8, wherein, The step of determining the output tensors of the multiple operators in the current processing cycle based on the current input tensor further includes: In response to determining that the target parameter value is less than the preset parameter threshold, the second preset storage space corresponding to the preset tensor is released.

12. The method according to claim 8, further comprising: The target model is used to process the subsequent input tensors of the current input tensor to obtain the output tensors of each of the operators in the subsequent processing cycles of the current processing cycle. The output tensors of the various operators in the post-processing cycle are written into the various target storage areas.

13. The method according to claim 8, wherein, The target storage region includes multiple target storage spaces, and the target storage spaces are configured to store the output tensors of the operator; There is no dependency relationship between the output tensors of the multiple operators stored in the target storage area. The dependency relationship is used to characterize that the output tensor of the subsequent operator of the current operator is obtained by inputting the output tensor of the current operator into the subsequent operator.

14. The method according to claim 13, wherein, The method further includes: In response to determining that at least one of the target storage regions has stored all output tensors that have the dependency relationship with the output tensor of the target operator, the target storage space used to store the output tensors of the target operator is released.

15. An electronic device comprising: The data processing apparatus according to any one of claims 1 to 7.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 8 to 14.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 8 to 14.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 8 to 14.

Images