Memory allocation method and device, computer equipment and storage medium

Abstract

The invention relates to a memory allocation method and device, computer equipment and a storage medium. The method comprises the steps that a target model is acquired, tensor information of each tensor generated in a topological structure based on the target model is determined, the tensor information comprises a use cycle, and the use cycle indicates an input operator and an output operator of the tensor; according to the tensor information of each tensor and the topological structure of the target model, operator information of operators contained in the topological structure is determined, and each operator is matched with at least one tensor; sorting the operators according to the operator information, and determining the priority of each tensor according to the sorting result of the operators and the tensor matched with the operators; and in response to the priority of the tensor, determining that the tensor which is not subjected to memory allocation exists in the tensor, and performing memory allocation on the tensor which is not subjected to memory allocation at least according to the input operator and the output operator indicated by the use cycle of the tensor which is not subjected to memory allocation. By adopting the method, memory resources can be saved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a memory allocation method, device, computer equipment, and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, computing models are deployed on electronic devices such as smartphones and tablets to enhance the processing capabilities of various aspects of electronic devices.

[0003] However, the computing model has limited resources during reasoning, and it is necessary to improve the efficiency of reasoning while saving memory space.

[0004] In this case, the memory usage of each operator in the reasoning process is often planned and allocated in advance through static memory allocation to avoid repeated application for memory release during actual calculation and reasoning, thereby improving efficiency. However, how to plan the memory to save memory resources is a big problem at this time. Summary of the invention

[0005] Based on this, it is necessary to provide a memory allocation method, device, computer equipment, and storage medium to address the above technical problems.

[0006] In a first aspect, the present disclosure provides a memory allocation method. The method comprises:

[0007] Acquire a target model, and determine tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage period, and the usage period is used to indicate an input operator and an output operator of the tensor;

[0008] Determining operator information of operators contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor;

[0009] According to the operator information, the operators are sorted, and according to the sorting result of the operators, a tensor matching the operators is determined, and the priority of each of the tensors is determined;

[0010] In response to determining that there are tensors for which memory allocation has not been performed among the tensors according to the priorities of the tensors, memory allocation is performed on the tensors for which memory allocation has not been performed at least according to input operators and output operators indicated by a usage cycle of the tensors for which memory allocation has not been performed.

[0011] In one embodiment, the types of the input operator and the output operator include: operators that support running inference acceleration; the memory allocation for the tensor that is not memory allocated is performed at least according to the input operator and the output operator indicated by the usage cycle of the tensor that is not memory allocated, including:

[0012] In response to the type of the input operator being an operator that supports running inference acceleration, memory is allocated to the tensor that has not been memory allocated according to a preset allocated memory;

[0013] In response to the input operator not being an operator that supports running inference acceleration and the output operator being an operator that supports running inference acceleration, memory is allocated for the tensor that has not been memory allocated according to the first input memory of the output operator.

[0014] In one embodiment, the performing memory allocation on the tensor that has not been memory allocated at least according to the input operator and the output operator indicated by the usage cycle of the tensor that has not been memory allocated, further comprises:

[0015] In response to the input operator not being an operator that supports running inference acceleration and the output operator not being an operator that supports running inference acceleration, determining operator characteristics of the input operator and the output operator;

[0016] In response to the operator characteristic supporting in-place operation, determining a second input memory of the operator supporting in-place operation;

[0017] When there is no conflict in the second input memory, memory allocation is performed on the tensor for which memory allocation has not been performed according to the second input memory.

[0018] In one embodiment, the method further comprises:

[0019] In the case where there is a conflict in the second input memory and there is a reused memory area in the cache pool of the memory, determine to use tensor information of tensors in each of the reused storage areas;

[0020] Determine a target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated;

[0021] Memory is allocated for the tensor that has not been memory allocated according to the reused storage area used by the target tensor.

[0022] In one embodiment, the tensor information further includes: size information; and determining the target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated includes:

[0023] Determine a first tensor among the tensors using the reused storage area, whose size information is greater than the size information of the tensor for which memory allocation is not performed;

[0024] Determine a target tensor in the first tensor according to a size difference between the size information of the first tensor and the size information of the tensor that is not memory allocated.

[0025] In one embodiment, determining the target tensor according to the tensor information of the tensor using each of the reused storage areas and the tensor information of the tensor that is not memory allocated includes:

[0026] Determine a usage cycle difference between a tensor using the reused storage area and a tensor not performing memory allocation according to a usage cycle of the tensor using the reused storage area and a usage cycle of the tensor not performing memory allocation;

[0027] In response to the use period of the tensor that generates the smallest use period difference being different from the input operator and the output operator indicated by the use period of the tensor that does not perform memory allocation, determining a target tensor according to the tensor that generates the smallest use period difference;

[0028] In response to the fact that the usage cycle of the tensor that produces the smallest usage cycle difference is the same as any one of the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, according to the size of the usage cycle difference, until a target usage cycle is determined whose usage cycle is different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, and the target tensor is determined according to the tensor corresponding to the target usage cycle.

[0029] In a second aspect, the present disclosure further provides a memory allocation device. The device comprises:

[0030] A tensor information determination module, used to obtain a target model, determine tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage cycle, and the usage cycle is used to indicate an input operator and an output operator of the tensor;

[0031] An operator information determination module, used to determine the operator information of the operator contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor;

[0032] A priority determination module, used to sort the operators according to the operator information, and determine the priority of each tensor according to a tensor matching the operator and the sorting result of the operators;

[0033] A memory allocation module is used to determine, in response to the priority of the tensor, that there are tensors among the tensors for which memory allocation has not been performed, and to allocate memory for the tensors for which memory allocation has not been performed at least according to input operators and output operators indicated by the usage cycle of the tensors for which memory allocation has not been performed.

[0034] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0035] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0036] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0037] In the above embodiments, since it is necessary to reasonably allocate memory for the reasoning process of the target model, and the target model will generate different tensors according to its topological structure during reasoning, the tensor information of each tensor generated in the topological structure of the target model can be determined. Since the tensors generated by the target model are generated by operators, the operator information can also be determined according to the tensor information and the topological structure. In order to reasonably allocate memory to the tensors, the operators can be sorted according to the operator information, and then the priority of each tensor can be determined according to the tensors matching the operators. The memory can be allocated according to the priority to ensure the effect of memory reuse. Finally, the input operators and output operators indicated by the use cycle of the tensors that are not allocated memory can be allocated to the tensors that are not allocated memory, and the memory corresponding to the input operators and output operators can be reused, and can be reused to reduce memory consumption. And because the operator information and tensor information used are used for priority adjustment, various information of operators and tensors is fully considered, and unnecessary memory consumption is further reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A first diagram is an application environment of a memory allocation method in one embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of a memory allocation method in one embodiment;

[0041] Figure 3 is a schematic diagram of the topological structure of a model in one embodiment;

[0042] Figure 4 A schematic diagram of a part of step S208 in one embodiment;

[0043] Figure 5 This is a flow chart of another part of step S208 in one embodiment;

[0044] Figure 6 A schematic diagram of a process of using a reused storage area to perform memory allocation in one embodiment;

[0045] Figure 7 A schematic diagram of a part of step S504 in one embodiment;

[0046] Figure 8 This is a flow chart of another part of step S504 in one embodiment;

[0047] Fig. 9 A schematic diagram of a flow chart of a memory allocation method in another embodiment;

[0048] Fig.10 is a schematic block diagram of the structure of a memory allocation device in one embodiment;

[0049] Fig.11 is a schematic diagram of the internal structure of a computer device in one embodiment;

[0050] Fig.12 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0054] As described in the background technology, model reasoning is actually the process of reading the model structure and weight information, performing calculations according to the model input, and outputting the calculation results. In the process of model reasoning, nrt (npu runtime) is usually used for processing. nrt (npu runtime) is a model reasoning running software based on the NPU heterogeneous platform, that is, nrt is responsible for reading the model structure and weight information, performing calculations according to the model input, and outputting the calculation results. The algorithm currently used by NPU is to traverse the operators one by one according to the order of the network topology of the model, and count the reference counts of the memory buffers occupied by the input and output of each operator to determine whether the current operator has a reusable buffer. If the buffer reference count is 0, it means that the buffer is not occupied by other tensors, so it can be reused, which can reduce a lot of memory consumption.

[0055] However, this strategy considers the time information of the network (according to the network topological order, in fact, the topological order is the execution order of the model, so this order represents the execution time information of the operators in the network). Although a lot of memory consumption has been optimized compared to the case of no memory reuse, there is still room for optimization due to the lack of consideration of the spatial information of the operator itself (that is, the size information of the related tensors).

[0056] Therefore, in order to solve the above problems, the present disclosure provides a memory allocation method, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can obtain the target model in the server 104. The terminal 102 can determine the tensor information of each tensor generated in the topological structure based on the target model, and the tensor information includes a usage cycle, and the usage cycle is used to indicate the input operator and output operator of the tensor. The terminal 102 can determine the operator information of the operator contained in the topological structure based on the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor. The terminal 102 sorts the operators according to the operator information, and determines the priority of each of the tensors according to the sorting result of the operators and a tensor that matches the operator. The terminal 102 determines that there are tensors that have not been allocated memory in the tensors according to the priority of the tensors, and allocates memory to the tensors that have not been allocated memory at least according to the input operators and output operators indicated by the usage cycle of the tensors that have not been allocated memory. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be implemented as an independent server or a server cluster composed of multiple servers. It should be noted that this method can also be applied to servers.

[0057] In one embodiment, Figure 2 As shown, a memory allocation method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0058] S202, acquiring a target model, and determining tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage cycle, and the usage cycle is used to indicate an input operator and an output operator of the tensor.

[0059] Among them, the target model can usually be a model for model reasoning, such as a neural network model for image processing or a neural network model for text, speech, etc. In machine learning and deep learning, tensors refer to multidimensional data structures input into the model. Tensor is a mathematical concept that represents a collection of numbers, which can be a scalar (0-dimensional), vector (1-dimensional), matrix (2-dimensional) or a higher-dimensional data structure. Specifically, input tensors are usually used to represent the following types of data: Image: An image may be represented as a three-dimensional tensor, and the dimensions are usually (height, width, color channel). Text: Text data can be converted into vectors through word embeddings, usually represented as a two-dimensional tensor. Time series: Time series data can be represented as a one-dimensional or two-dimensional tensor, depending on the structure of the data. Operators usually refer to specific operations or functions used to define and process input data. The types of input operators and output operators can include: Placeholder: In some frameworks, such as TensorFlow, placeholders are used to define the shape and type of inputs, but specific data is not provided when the model is built. Input Layer: In frameworks such as Keras, the input layer is explicitly defined, including the input shape and type. Data loading and preprocessing: This includes batch processing, normalization, data enhancement, and other operations on the input data, which are usually implemented as a series of data processing functions. Conversion operators: For example, converting text into numerical form (such as word embedding) or resizing images to the size required by the model. Such operators are executed before the data flows into the model. In some exemplary embodiments, such as Figure 3 As shown in the figure, taking the model topology A->C->D->B as an example, the usage cycle of a certain tensor can be determined in the following way: first determine the index of the operator that produces this tensor and the index of the operator that consumes this tensor. The index refers to the index of the operator in the network topology. For example, in the topology A->C->D->B, the indexes of the four operators are 0, 1, 2, and 3 respectively. Then the input operator of the A_out2 tensor is 0, the output operator is 1, and the usage cycle of the A_out2 tensor is (0, 1), that is, the producer is 0 and the consumer is 1. The tensor information can include the usage cycle, and can also include the name of the tensor and the size of the tensor, that is, the size information of the tensor. The size of a tensor refers to the size of the tensor in each dimension. Specifically, a tensor is a multidimensional array that can be used to represent scalars (0 dimensions), vectors (1 dimensions), matrices (2 dimensions) and higher-dimensional structures. The size of a tensor describes how many elements the tensor contains in each dimension.

[0060] Specifically, the target model can be obtained, the target model can be analyzed, and the topological structure of the target model can be determined. In addition, it should be noted that the present disclosure is based on the NPU heterogeneous platform, and some background information on the memory allocation of this heterogeneous platform needs to be introduced first. First, the code that generates the model structure file will change the initial input tensor of the model to an input type operator, whose input tensor number is 0, and the output is the number of tensors that should be input. Since the input of an operator is the output of the previous operator, it is actually guaranteed that the output of each operator is allocated, that is, the static memory allocation of the entire model is completed. Therefore, the terminal can determine the multiple tensors generated based on the topological structure of the target model based on the NPU heterogeneous platform and the topological structure of the target model, and determine the tensor information of each tensor.

[0061] In some exemplary embodiments, all tensor tensor_record records can be counted to obtain the tensor information of each tensor, including the name, size, and usage cycle of the corresponding tensor. tensor_record is a format for storing and reading large tensor data, mainly used for efficient data input and processing, especially in the context of deep learning and machine learning. Features of tensor_record: Efficient storage: The tensor_record format usually stores data in binary, which can effectively compress and store large data sets. Support for large-scale data: Suitable for processing large-scale data sets, especially when the data size exceeds the memory, it can be effectively processed by reading in blocks. Easy to process in parallel: Since the data is stored as independent records, frameworks such as TensorFlow can easily read and process it in parallel, which increases the speed of data input. Flexibility: Supports multiple types of data (such as images, text, audio, etc.), and users can customize the data format according to their needs.

[0062] S204, determining operator information of operators contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor.

[0063] Among them, continue to refer to Figure 3, the operators and tensors mentioned in some embodiments of the present disclosure are explained. When operator C is running, operator A has finished running and generated two outputs A_out1 and A_out_2. Then the minimum memory usage required when operator C is running is the space required for the two tensors A_out1 and Aout_2 that must exist in the memory at this time plus the output C_out of C. This space is the width of the operator, and these three tensors are the tensors that must exist in the memory during the operation of the C operator. Therefore, the tensors that match the C operator are A_out1, Aout_2, and C_out. Operator information may include: operator name, operator width. Therefore, the operator width is a measure of the minimum amount of space required when the current operator is running.

[0064] Specifically, the topological structure of the target model can be used to determine the matching relationship between operators and tensors, and then the matching relationship and the usage cycle in the tensor information can be used to calculate the operator width of each current operator. According to the width of the operator and the operator name, the operator information of the operator is determined.

[0065] In some exemplary embodiments, the layer_profile records of all operators can be counted according to the usage period, including the operator name and operator width, to obtain operator information. The layer_profile record is usually a tool used in deep learning frameworks to analyze and optimize model performance. It can provide detailed data on the computation time, memory usage, and other relevant information of each layer during forward propagation and backpropagation.

[0066] S206, sorting the operators according to the operator information, and determining the priority of each tensor according to a tensor matching the operator and the sorting result of the operators.

[0067] Specifically, operators with larger width information can be placed in front according to the width information contained in the operator information to sort the operators. Since each operator matches at least one tensor, after the operators are sorted, the tensors matching the operators can be sorted according to the tensor information to determine the priority of each tensor.

[0068] In some exemplary embodiments, continue with Figure 3Take an example to illustrate, the four operators A, B, C, and D are sorted according to the width information of the operator, and the sorted order is D, C, B, and A. Then, according to the tensors matched by each operator, the priority of the tensor is determined according to the tensor information of the tensor. For example, the tensors matched by the D operator are A_out1, A_out_2, C_out, and D_out, the tensors matched by the C operator are A_out1, Aout_2, and C_out, the tensors matched by the B operator are A_out1, Aout_2, and B_out, and the tensors matched by the A operator are A_out1 and Aout_2. If D_out>C_out>A_out1>B_out>A_out_2, the operator matched by the tensor of the D operator is started first, and the priority of the tensors from high to low is D_out, C_out, A_out1, and A_out_2. Since there is no B_out tensor in the D operator, and only the B_out tensor is left in the tensor, B_out can be determined as the lowest priority. In addition, the minimum memory usage required by the target model can also be determined through the operator width corresponding to the D operator.

[0069] S208, in response to determining that there are tensors for which memory allocation has not been performed among the tensors according to the priority of the tensors, memory allocation is performed on the tensors for which memory allocation has not been performed at least according to input operators and output operators indicated by the usage cycle of the tensors for which memory allocation has not been performed.

[0070] Specifically, since the priorities of tensors are different, the effects of the memory reuse process are also different. Therefore, in order to ensure the effect of memory reuse, it is possible to determine, according to the priorities of the tensors, whether there are tensors in the tensors that have not been memory allocated, and then perform memory allocation on the tensors that have not been memory allocated at least based on the input operators and output operators indicated by the usage cycles of the tensors that have not been memory allocated. For example, the memory indicated by the input operator or the output operator can be used to allocate memory to the tensors that have not been memory allocated, that is, the memory indicated by the input operator or the output operator is reused.

[0071] In the above memory allocation method, since it is necessary to reasonably allocate memory for the reasoning process of the target model, and the target model will generate different tensors according to its topological structure during reasoning, the tensor information of each tensor generated in the topological structure of the target model can be determined. Since the tensors generated by the target model are generated by operators, the operator information can also be determined according to the tensor information and the topological structure. In order to reasonably allocate memory to the tensors, the operators can be sorted according to the operator information, and then the priority of each tensor can be determined according to the tensors matching the operators. Memory allocation can be performed according to the priority to ensure the effect of memory reuse. Finally, the input operators and output operators indicated by the use cycle of the tensors that have not been allocated memory can be allocated memory for the tensors that have not been allocated memory, and the memory corresponding to the input operators and output operators can be reused, which can be reused to reduce memory consumption, and because the operator information and tensor information used are used for priority adjustment, various information of operators and tensors is fully considered, further reducing unnecessary memory consumption.

[0072] In one embodiment, the types of input operators and output operators include: operators that support running inference acceleration; it should be noted that in the implementation of NPU heterogeneous platform operators, operators are divided into different types,

[0073] The first type is that the NPU directly supports running inference acceleration operators. The memory information occupied by the input and output tensors of these operators will be planned in the compiler, that is, the memory that these operators can use will be pre-allocated in the compiler.

[0074] The other part is operators that support in-place operations, that is, the output can reuse the input memory without trampling, such as sinh. In-place operation means that when performing a certain calculation, the original data is modified directly at the memory location, rather than creating a new copy of the data. This operation can save memory and improve performance because it reduces the number of times memory needs to be allocated and released. Features of in-place operations: Save memory: In-place operations do not require allocation of additional memory space to store results, which is particularly important when processing large-scale data. Improve performance: In-place operations usually have higher execution efficiency due to the reduction of memory allocation and copying overhead. Such as Figure 4 As shown, the memory allocation for the tensor that has not been memory allocated is performed at least according to the input operator and the output operator indicated by the usage cycle of the tensor that has not been memory allocated, including:

[0075] S302: In response to the type of the input operator being an operator that supports running inference acceleration, memory is allocated for the tensor that has not been memory allocated according to a preset allocated memory.

[0076] Specifically, when the input operator type is an operator that supports running inference acceleration, the memory of the current tensor is directly allocated according to the memory planned by the compiler. The pre-set allocated memory can be the memory planned by the compiler.

[0077] S304: In response to the input operator not being an operator that supports running inference acceleration and the output operator being an operator that supports running inference acceleration, memory is allocated for the tensor that has not been memory allocated according to the first input memory of the output operator.

[0078] Specifically, when the type of the input operator is not an operator that supports running inference acceleration, but the type of the output operator is an operator that supports running inference acceleration, the memory of the current tensor can be allocated as the first input memory of this output operator to achieve zero copy and improve performance.

[0079] In this embodiment, by judging the type of the input operator and / or the output operator, memory allocation can be performed accurately according to different situations, thereby reducing the waste of memory resources and saving memory resources.

[0080] In one embodiment, Figure 5 As shown, the memory allocation for the tensor that has not been memory allocated is performed at least according to the input operator and the output operator indicated by the usage cycle of the tensor that has not been memory allocated, and also includes:

[0081] S402: In response to the input operator not being an operator that supports running inference acceleration and the output operator not being an operator that supports running inference acceleration, determine operator characteristics of the input operator and the output operator.

[0082] Specifically, when the types of the input operator and the output operator are not operators that support running inference acceleration, it is necessary to determine the operator characteristics of the input operator and the output operator.

[0083] S404: In response to the operator characteristic supporting in-place operation, determine a second input memory of the operator supporting in-place operation.

[0084] S406: When there is no conflict in the second input memory, perform memory allocation on the tensor that has not been memory allocated according to the second input memory.

[0085] Specifically, when the operator characteristics of the input operator and / or the output operator support in-place operation, the second input memory of the operator supporting in-place operation is determined, and when the second input memory does not conflict with other allocation results, the second input memory can be reused, and the tensor that has not been memory allocated is allocated according to the second input memory.

[0086] In this embodiment, when the operator is an operator that supports in-place operation and there is no conflict in the second input memory of the operator, the second input memory can be reused to reduce the waste of memory resources.

[0087] In one embodiment, Figure 6 As shown, the method also includes:

[0088] S502: When there is a conflict in the second input memory and there is a reused memory area in the cache pool of the memory, determine tensor information of tensors using each of the reused storage areas.

[0089] Among them, the buffer pool can usually be a buffer pool. The buffer pool is a memory pool that manages multiple buffers, allowing the system to reuse these buffers efficiently. In databases and high-performance computing frameworks, the buffer pool can reduce the overhead of memory allocation and improve data reuse. The main features of the buffer pool: Reuse: The buffer pool can reuse the space in the buffer to avoid frequent allocation and release of memory, thereby improving efficiency. Unified management: Through the buffer pool, the system can uniformly manage multiple buffers, facilitate control of total memory usage, optimize performance, and implement more efficient caching strategies. The reused memory area can usually be a buffer that can be reused.

[0090] Specifically, when there is a conflict in the second input memory and it cannot be reused, and there is a buffer that can be reused in the memory cache pool, at this time, the tensor information of the tensor that has used the reused memory area can be determined.

[0091] S504: Determine a target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated.

[0092] S506: Allocate memory for the tensor that has not been memory allocated according to the reused storage area used by the target tensor.

[0093] Specifically, according to the tensor information of the tensor of each of the reused storage areas and the tensor information of the tensor that is not memory allocated, a tensor with the highest similarity to the tensor information of the tensor that is not memory allocated is determined, and the tensor may be a target tensor. Then, according to the reused memory area used by the target tensor, reuse is performed. That is, the reused memory area used by the target tensor is given to the tensor that is not memory allocated.

[0094] In this embodiment, by reusing the memory area, the waste of memory resources can be reduced.

[0095] In one embodiment, Figure 7As shown, the tensor information also includes: size information; the target tensor is determined according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated, including:

[0096] S602, determining a first tensor among the tensors using the reused storage area, whose size information is greater than the size information of the tensor that is not memory allocated;

[0097] S604: Determine a target tensor in the first tensor according to a size difference between the size information of the first tensor and the size information of the tensor that is not memory allocated.

[0098] Specifically, a first tensor whose size information is greater than the size information of the tensor that does not perform memory allocation can be determined among the tensors using the reused storage area. Then, the size difference between the size information of the first tensor and the size information of the tensor that does not perform memory allocation is determined, and the first tensor corresponding to the smallest size difference is found, which is the target tensor.

[0099] In this embodiment, the size information is used to judge and find the first tensor whose size information is larger than the tensor to which memory allocation is not performed. This can ensure that when subsequent memory allocation is performed, the reused storage area can cover all tensors to which memory allocation is not performed. In addition, the target tensor in the first tensor is determined based on the size difference, so that the most suitable target tensor can be found, thereby reducing the waste of memory resources.

[0100] In one embodiment, Figure 8 As shown, the determining of the target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated includes:

[0101] S702, determining a usage cycle difference between a tensor using the reused storage area and a tensor not performing memory allocation according to a usage cycle of the tensor using the reused storage area and a usage cycle of the tensor not performing memory allocation;

[0102] S704, in response to the fact that the usage period of the tensor that produces the smallest usage period difference is different from the input operator and the output operator indicated by the usage period of the tensor that is not memory allocated, determine the target tensor based on the tensor that produces the smallest usage period difference.

[0103] Specifically, the usage cycle of the tensor with the smallest usage cycle difference can be found based on the usage cycle difference between the usage cycle in the reused storage area and the usage cycle of the tensor without memory allocation, and then it is determined whether the usage cycle of the tensor with the smallest usage cycle difference conflicts with the input operator and output operator of the usage cycle of the tensor without memory allocation. If there is no conflict, the target tensor is determined based on the tensor that produces the smallest usage cycle difference.

[0104] S706, in response to the usage cycle of the tensor that produces the smallest usage cycle difference being the same as any one of the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, according to the size of the usage cycle difference, until a target usage cycle is determined whose usage cycle is different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, and the target tensor is determined according to the tensor corresponding to the target usage cycle.

[0105] Specifically, if there is a conflict, it means that the reused storage area corresponding to the smallest usage cycle difference cannot be applied to the current tensor for which no memory allocation is performed. Therefore, it is necessary to determine them in order of usage cycle differences from small to large until a target usage cycle is determined, whose usage cycle is different from the input operator and output operator indicated by the usage cycle of the tensor for which no memory allocation is performed. Then, the target tensor is determined based on the tensor corresponding to the target usage cycle.

[0106] In addition, it should be noted that if the target tensor cannot be determined based on the usage period, the target tensor can be determined jointly using the usage period and size information.

[0107] In this embodiment, the target tensor can be determined more accurately by using the period difference, so as to accurately find the reused storage area and reduce the waste of storage resources.

[0108] In an exemplary embodiment, for example, the usage period of the current tensor is (15, 16) and there is a buffer allocation result as shown in Table 1 below.

[0109] buffer pool The lifetime of the tensor using this buffer buffer_1 (12，14)、(18，20) buffer_2 (9，10)、(19,21) buffer_3 (16，18)

[0110] The minimum difference between the current tensor usage period (15, 16) and the tensor usage period in buffer_1 is 15-14=1, and the minimum difference between the current tensor usage period and the tensor usage period in buffer_2 is 19-16=3, which conflicts with the already allocated tensor usage period in buffer_3 (16 is the same). Therefore, the minimum usage period is buffer_1, and 15 and 16 do not exist in buffer_1, so buffer_1 should be selected for use.

[0111] In one embodiment, Fig. 9 As shown, the embodiment of the present disclosure also provides another memory allocation method, including:

[0112] S802, acquiring a target model, and determining tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage period and size information, and the usage period is used to indicate an input operator and an output operator of the tensor.

[0113] S804, determining operator information of operators contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor.

[0114] S806, sorting the operators according to the operator information, and determining the priority of each tensor according to a tensor matching the operator and the sorting result of the operators.

[0115] S808: Determine, according to the priority of the tensors, whether there are any tensors in the tensors for which memory allocation has not been performed.

[0116] S810: In response to the type of the input operator of the tensor that has not been memory allocated being an operator that supports running inference acceleration, memory is allocated for the tensor that has not been memory allocated according to a preset allocated memory.

[0117] S812: In response to the fact that the type of the input operator of the tensor for which memory allocation is not performed is not an operator that supports running inference acceleration, and the type of the output operator is an operator that supports running inference acceleration, memory is allocated for the tensor for which memory allocation is not performed according to the first input memory of the output operator.

[0118] S814: In response to the input operator not being an operator that supports running inference acceleration and the output operator not being an operator that supports running inference acceleration, determine operator characteristics of the input operator and the output operator.

[0119] S816: In response to the operator characteristic supporting in-place operation, determine a second input memory of the operator supporting in-place operation.

[0120] S818: When there is no conflict in the second input memory, perform memory allocation on the tensor that has not been memory allocated according to the second input memory.

[0121] S820: When there is a conflict in the second input memory and there is a reused memory area in the memory cache pool, determine to use tensor information of tensors in each of the reused storage areas.

[0122] S822, determining a first tensor among the tensors using the reused storage area, whose size information is greater than the size information of the tensor for which memory allocation is not performed.

[0123] S824: Determine a target tensor in the first tensor according to a size difference between the size information of the first tensor and the size information of the tensor that is not memory allocated.

[0124] Alternatively, S826, determining a usage cycle difference between tensors using the reused storage area and tensors without memory allocation based on the usage cycle of tensors using the reused storage area and the usage cycle of tensors without memory allocation.

[0125] S828, in response to the fact that the usage period of the tensor that produces the smallest usage period difference is different from the input operator and the output operator indicated by the usage period of the tensor that is not memory allocated, determine the target tensor based on the tensor that produces the smallest usage period difference.

[0126] S830, in response to the fact that the usage cycle of the tensor that produces the smallest usage cycle difference is the same as any one of the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, according to the size of the usage cycle difference, until a target usage cycle is determined whose usage cycle is different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, and the target tensor is determined according to the tensor corresponding to the target usage cycle.

[0127] S832: Allocate memory for the tensor that has not been memory allocated according to the reused storage area used by the target tensor.

[0128] For the specific implementation and limitation of this embodiment, please refer to the above embodiment, which will not be repeated here.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiment of the present disclosure also provides a memory allocation device for implementing the memory allocation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more memory allocation device embodiments provided below can refer to the limitations on the memory allocation method above, and will not be repeated here.

[0131] In one embodiment, Fig.10 As shown, a memory allocation device 900 is provided, including: a tensor information determination module 902, an operator information determination module 904, a priority determination module 906 and a memory allocation module 908, wherein:

[0132] A tensor information determination module 902, used to obtain a target model, determine tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage period, and the usage period is used to indicate an input operator and an output operator of the tensor;

[0133] An operator information determination module 904 is used to determine the operator information of the operator contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor;

[0134] A priority determination module 906 is used to sort the operators according to the operator information, and determine the priority of each tensor according to the sorting result of the operators and a tensor matching the operators;

[0135] A memory allocation module 908 is used to determine, in response to the priority of the tensor, that there are tensors in the tensor that have not been memory allocated, and to allocate memory for the tensors that have not been memory allocated at least according to the input operators and output operators indicated by the usage cycle of the tensors that have not been memory allocated.

[0136] In one embodiment of the device, the types of the input operator and the output operator include: operators that support running inference acceleration; the memory allocation module 908 includes:

[0137] The first allocation module is used to allocate memory for the tensor that has not been memory allocated according to a preset allocated memory in response to the type of the input operator being an operator that supports running inference acceleration.

[0138] The second allocation module is used to allocate memory for the tensor that has not been memory allocated according to the first input memory of the output operator in response to the type of the input operator not being an operator that supports running inference acceleration and the type of the output operator being an operator that supports running inference acceleration.

[0139] In one embodiment of the device, the memory allocation module 908 further includes:

[0140] The third allocation module is used to determine the operator characteristics of the input operator and the output operator in response to the type of the input operator not being an operator that supports running inference acceleration and the type of the output operator not being an operator that supports running inference acceleration; in response to the operator characteristics supporting in-place operation, determine the second input memory of the operator that supports in-place operation; and when there is no conflict in the second input memory, allocate memory for the tensor that has not been memory allocated according to the second input memory.

[0141] In one embodiment of the device, the memory allocation module 908 further includes:

[0142] The fourth allocation module is used to determine the tensor information of the tensors in each reused storage area to be used when there is a conflict in the second input memory and there is a reused memory area in the memory cache pool; determine the target tensor according to the tensor information of the tensors in each reused storage area and the tensor information of the tensors to which memory allocation is not performed; and perform memory allocation on the tensors to which memory allocation is not performed according to the reused storage area used by the target tensor.

[0143] In one embodiment of the device, the fourth allocation module comprises:

[0144] A first target tensor determination module determines a first tensor among the tensors using the reused storage area, whose size information is greater than the size information of the tensor that is not memory allocated; and determines a target tensor among the first tensor based on a size difference between the size information of the first tensor and the size information of the tensor that is not memory allocated.

[0145] In one embodiment of the device, the fourth allocation module comprises:

[0146] The second target tensor determination module determines the usage cycle difference between the tensor using the reused storage area and the tensor without memory allocation according to the usage cycle of the tensor using the reused storage area and the usage cycle of the tensor without memory allocation; in response to the usage cycle of the tensor that produces the smallest usage cycle difference being different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, determines the target tensor according to the tensor that produces the smallest usage cycle difference; in response to the usage cycle of the tensor that produces the smallest usage cycle difference being the same as either the input operator or the output operator indicated by the usage cycle of the tensor without memory allocation, determines the target tensor according to the size of the usage cycle difference until a target usage cycle is determined whose usage cycle is different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, and determines the target tensor according to the tensor corresponding to the target usage cycle.

[0147] Each module in the above-mentioned memory allocation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0148] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store tensor information and operator information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a memory allocation method is implemented.

[0149] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.12As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a memory allocation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0150] Those skilled in the art will understand that Fig.11 The structure shown in or 12 is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0153] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0155] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A memory allocation method, characterized in that: The method comprises: Acquire a target model, and determine tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage period, and the usage period is used to indicate an input operator and an output operator of the tensor; Determining operator information of operators contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor; According to the operator information, the operators are sorted, and according to the sorting result of the operators, a tensor matching the operators is determined, and the priority of each of the tensors is determined; In response to determining, according to the priority of the tensors, that there are tensors among the tensors for which memory allocation has not been performed, memory allocation is performed on the tensors for which memory allocation has not been performed at least according to input operators and output operators indicated by a usage cycle of the tensors for which memory allocation has not been performed.

2. The method according to claim 1, characterized in that The types of the input operator and the output operator include: operators that support running inference acceleration; the memory allocation for the tensor that is not memory allocated is performed at least according to the input operator and the output operator indicated by the usage cycle of the tensor that is not memory allocated, including: In response to the type of the input operator being an operator that supports running inference acceleration, memory is allocated to the tensor that has not been memory allocated according to a preset allocated memory; In response to the input operator not being an operator that supports running inference acceleration and the output operator being an operator that supports running inference acceleration, memory is allocated for the tensor that has not been memory allocated according to the first input memory of the output operator.

3. The method according to claim 2, characterized in that The performing memory allocation on the tensor that has not been memory allocated at least according to the input operator and the output operator indicated by the usage cycle of the tensor that has not been memory allocated, further includes: In response to the input operator not being an operator that supports running inference acceleration and the output operator not being an operator that supports running inference acceleration, determining operator characteristics of the input operator and the output operator; In response to the operator characteristic supporting in-place operation, determining a second input memory of the operator supporting in-place operation; When there is no conflict in the second input memory, memory allocation is performed on the tensor for which memory allocation has not been performed according to the second input memory.

4. The method according to claim 3, characterized in that The method further comprises: In the case where there is a conflict in the second input memory and there is a reused memory area in the cache pool of the memory, determine to use tensor information of tensors in each of the reused storage areas; Determine a target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated; Memory is allocated for the tensor that has not been memory allocated according to the reused storage area used by the target tensor.

5. The method according to claim 4, characterized in that The tensor information further includes: size information; the determining of the target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated includes: Determine a first tensor among the tensors using the reused storage area, whose size information is greater than the size information of the tensor for which memory allocation is not performed; A target tensor in the first tensor is determined according to a size difference between the size information of the first tensor and the size information of the tensor that is not memory allocated.

6. The method according to claim 4, characterized in that The determining of the target tensor according to the tensor information of the tensors using each of the reused storage areas and the tensor information of the tensors that are not memory allocated includes: Determine a usage cycle difference between a tensor using the reused storage area and a tensor not performing memory allocation according to a usage cycle of the tensor using the reused storage area and a usage cycle of the tensor not performing memory allocation; In response to the use period of the tensor that generates the smallest use period difference being different from the input operator and the output operator indicated by the use period of the tensor that does not perform memory allocation, determining a target tensor according to the tensor that generates the smallest use period difference; In response to the fact that the usage cycle of the tensor that produces the smallest usage cycle difference is the same as any one of the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, according to the size of the usage cycle difference, until a target usage cycle is determined whose usage cycle is different from both the input operator and the output operator indicated by the usage cycle of the tensor without memory allocation, and the target tensor is determined according to the tensor corresponding to the target usage cycle.

7. A memory allocation device, characterized in that: The device comprises: A tensor information determination module, used to obtain a target model, determine tensor information of each tensor generated in a topological structure based on the target model, wherein the tensor information includes a usage cycle, and the usage cycle is used to indicate an input operator and an output operator of the tensor; An operator information determination module, used to determine the operator information of the operator contained in the topological structure according to the tensor information of each tensor and the topological structure of the target model, wherein each operator matches at least one tensor; A priority determination module, used to sort the operators according to the operator information, and determine the priority of each tensor according to a tensor matching the operator and the sorting result of the operators; A memory allocation module is used to determine, in response to the priority of the tensor, that there are tensors among the tensors for which memory allocation has not been performed, and to allocate memory for the tensors for which memory allocation has not been performed at least according to input operators and output operators indicated by the usage cycle of the tensors for which memory allocation has not been performed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images