Memory access optimization method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a memory access optimization method and device, electronic equipment and storage medium. The method comprises: obtaining an operator sequence in an original model; determining at least one operator sub-sequence in the operator sequence according to the category of each operator in the operator sequence, wherein the operator sub-sequence comprises a plurality of operators that can be combined; and combining the plurality of operators in each operator sub-sequence into a corresponding target operator to obtain a target model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a memory access optimization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the gradual development of artificial intelligence technology, deep neural network models are widely used in many industries, especially deep convolutional neural network models, which are widely used due to their efficient optimization capabilities for media such as images and videos. To ensure the performance of deep neural network models, larger-scale networks are usually used, which contain a large number of operators to provide stronger spatial mapping and information extraction capabilities. However, too many operators will lead to excessive memory access during network operation, resulting in longer computation time and lower efficiency. Summary of the Invention

[0003] This disclosure provides a memory access optimization method, apparatus, electronic device, and storage medium to address the deficiencies in related technologies.

[0004] According to a first aspect of the present disclosure, a memory access optimization method is provided, comprising:

[0005] Obtain the operator sequence from the original model;

[0006] Based on the type of each operator in the operator sequence, at least one operator subsequence is determined in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged;

[0007] Multiple operators in each operator subsequence are merged into the corresponding target operator to obtain the target model.

[0008] In one embodiment, the original model includes a neural network model.

[0009] In one embodiment, determining at least one operator subsequence in the operator sequence based on the type of each operator in the operator sequence includes:

[0010] Based on the type of each operator in the operator sequence, determine whether the corresponding operator belongs to the combinable operators;

[0011] A sequence consisting of multiple consecutive mergeable operators is defined as an operator subsequence.

[0012] In one embodiment, determining whether a corresponding operator belongs to a mergeable operator based on the type of each operator in the operator sequence includes:

[0013] If the type of the operator exists in a preset type list, it is determined that the operator belongs to the mergeable operator category;

[0014] If the type of the operator does not exist in the preset type list, it is determined that the operator is not a mergeable operator.

[0015] In one embodiment, the preset category list includes output fusion class, data rearrangement class, data access class, and single element processing class.

[0016] In one embodiment, merging multiple operators in each operator subsequence into a corresponding target operator includes:

[0017] In the case where the plurality of operators are output fusion operators and data rearrangement operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship between the output position of each data element before and after data rearrangement;

[0018] Based on the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly to obtain the target operator.

[0019] In one embodiment, merging multiple operators in each operator subsequence into a corresponding target operator includes:

[0020] When the plurality of operators are output-fusionable operators and data access operators, the output position of at least one accessed element among the plurality of data elements in the output data of the output-fusionable operator is determined according to the data access operator.

[0021] Based on the output position of the at least one accessed element, the output data of the output fusion operator is adjusted to the at least one accessed element to obtain the target operator.

[0022] In one embodiment, merging multiple operators in each operator subsequence into a corresponding target operator includes:

[0023] When the plurality of operators are sequentially arranged output fusion operators, data rearrangement operators, and data access operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement;

[0024] Based on the data access operator, determine the output position of at least one accessed element among multiple data elements in the output data of the output fusion operator;

[0025] According to the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly, and the output data of the output fusion operator is adjusted to the at least one accessed element according to the output position of the at least one accessed element, so as to obtain the target operator.

[0026] In one embodiment, merging multiple operators in each operator subsequence into a corresponding target operator includes:

[0027] In the case where the multiple operators are output-fusionable operators and single-element processing operators, the parameters of the output-fusionable operators are adjusted according to the parameters of the single-element processing operators.

[0028] In one embodiment, it also includes:

[0029] The test data is input into the original model and the target model respectively to obtain the output data of the original model and the output data of the target model;

[0030] If the output data of the original model and the output data of the target model are consistent, the original model is updated to the target model.

[0031] According to a second aspect of the present disclosure, a memory access optimization apparatus is provided, comprising:

[0032] The acquisition module is used to acquire the operator sequence in the original model;

[0033] A filtering module is used to determine at least one operator subsequence in the operator sequence according to the type of each operator in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged;

[0034] The merging module is used to merge multiple operators in each operator subsequence into the corresponding target operator to obtain the target model.

[0035] In one embodiment, the original model includes a neural network model.

[0036] In one embodiment, the filtering module is specifically used for:

[0037] Based on the type of each operator in the operator sequence, determine whether the corresponding operator belongs to the combinable operators;

[0038] A sequence consisting of multiple consecutive mergeable operators is defined as an operator subsequence.

[0039] In one embodiment, when the filtering module determines whether a corresponding operator belongs to a mergeable operator based on the type of each operator in the operator sequence, it is specifically used for:

[0040] If the type of the operator exists in a preset type list, it is determined that the operator belongs to the mergeable operator category;

[0041] If the type of the operator does not exist in the preset type list, it is determined that the operator is not a mergeable operator.

[0042] In one embodiment, the preset category list includes output fusion class, data rearrangement class, data access class, and single element processing class.

[0043] In one embodiment, the merging module is specifically used for:

[0044] In the case where the plurality of operators are output fusion operators and data rearrangement operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship between the output position of each data element before and after data rearrangement;

[0045] Based on the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly to obtain the target operator.

[0046] In one embodiment, the merging module is specifically used for:

[0047] When the plurality of operators are output-fusionable operators and data access operators, the output position of at least one accessed element among the plurality of data elements in the output data of the output-fusionable operator is determined according to the data access operator.

[0048] Based on the output position of the at least one accessed element, the output data of the output fusion operator is adjusted to the at least one accessed element to obtain the target operator.

[0049] In one embodiment, the merging module is specifically used for:

[0050] When the plurality of operators are sequentially arranged output fusion operators, data rearrangement operators, and data access operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement;

[0051] Based on the data access operator, determine the output position of at least one accessed element among multiple data elements in the output data of the output fusion operator;

[0052] According to the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly, and the output data of the output fusion operator is adjusted to the at least one accessed element according to the output position of the at least one accessed element, so as to obtain the target operator.

[0053] In one embodiment, the merging module is specifically used for:

[0054] In the case where the multiple operators are output-fusionable operators and single-element processing operators, the parameters of the output-fusionable operators are adjusted according to the parameters of the single-element processing operators.

[0055] In one embodiment, a test module is also included, for:

[0056] The test data is input into the original model and the target model respectively to obtain the output data of the original model and the output data of the target model;

[0057] If the output data of the original model and the output data of the target model are consistent, the original model is updated to the target model.

[0058] According to a third aspect of the present disclosure, an electronic device is provided, the device including a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to implement the method described in the first aspect when executing the computer instructions.

[0059] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0060] As can be seen from the above embodiments, by obtaining the operator sequence in the original model, at least one operator subsequence can be determined in the operator sequence according to the type of each operator. Since the operator subsequence includes multiple operators that can be merged, multiple operators in each operator subsequence can be merged into the corresponding target operator, thereby completing the optimization process of the original model and obtaining the target model. Operator merging can reduce memory access during model operation, thereby reducing computation time and improving computational efficiency.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0063] Figure 1 This is a flowchart illustrating a memory access optimization method according to an exemplary embodiment of this disclosure;

[0064] Figure 2 This is a schematic diagram illustrating a data rearrangement operator according to an exemplary embodiment of this disclosure;

[0065] Figure 3A This is a schematic diagram illustrating the position indexes of each data element of the original data in an exemplary embodiment of this disclosure;

[0066] Figure 3B This is a schematic diagram illustrating the position index of each data element in the re-photographed data according to an exemplary embodiment of this disclosure;

[0067] Figure 4 This is a schematic diagram illustrating a data access class operator in an exemplary embodiment of this disclosure;

[0068] Figure 5 This is a schematic diagram of the memory access optimization device shown in the embodiments of this disclosure;

[0069] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this disclosure. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0071] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0072] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0073] With the improvement of hardware computing power and software collaboration capabilities, increasingly larger deep learning models are being trained to achieve higher performance and better generalization. However, large models, whether deployed on the edge or the cloud, face significant latency issues due to high-frequency calls, which reduces the model's computation speed and performance.

[0074] Based on this, in a first aspect, at least one embodiment of this disclosure provides a memory access optimization method, please refer to the appendix. Figure 1 It illustrates the process of the method, including steps S101 to S103.

[0075] This method can be used to optimize models to reduce memory access, i.e., optimize memory access. The model targeted by this method can be a neural network model, such as a deep neural network model. Furthermore, this method can be executed by electronic devices such as terminal devices or servers. Terminal devices can be user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistant (PDA) handheld devices, computing devices, in-vehicle devices, wearable devices, etc. This method can be implemented by the processor calling computer-readable instructions stored in memory. Alternatively, this method can be executed by a server, such as a local server or a cloud server.

[0076] In step S101, the operator sequence in the original model is obtained.

[0077] The original model can be the model to be optimized by this method, such as a neural network model, specifically a deep neural network model, a deep convolutional neural network model, etc. The original model includes multiple operators, each used to perform one or more data processing operations. The multiple operators in the original model are sequential, that is, arranged in order to form an operator sequence. After each operator in this operator sequence completes its assigned data processing, it sends the processing result to the next operator for further processing. Therefore, this step can obtain the operator sequence composed of all operators in the original model arranged in the specified order.

[0078] In addition, the type of each operator in the operator sequence can be obtained simultaneously.

[0079] In step S102, at least one operator subsequence is determined in the operator sequence according to the type of each operator in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged.

[0080] Of all operator types, some can be merged, such as output-mergeable operators (e.g., convolution, fully connected operators), data rearrangement operators, and data access operators; others cannot. An operator subsequence can contain at least two operators and at most all operators in the subsequence. Operators within a subsequence can be merged, while operators between subsequences cannot be merged.

[0081] In one possible embodiment, at least one operator subsequence can be determined in the operator sequence in the following manner:

[0082] First, based on the type of each operator in the operator sequence, determine whether the corresponding operator belongs to the category of mergeable operators. For example, the categories of mergeable operators can be pre-counted, and the results can be used to generate a preset category list. This list records all mergeable operator categories; in other words, operators outside this list are non-mergeable operator categories. For example, the preset category list includes output merging, data rearrangement, and data access categories. Then, if the operator's type exists in the preset category list, determine that the operator belongs to the category of mergeable operators; if the operator's type does not exist in the preset category list, determine that the operator does not belong to the category of mergeable operators.

[0083] Next, the sequence of consecutive combinable operators is determined as an operator subsequence. For example, all operators in the operator sequence that are not combinable (hereinafter referred to as non-combinable operators) can be extracted. Then, all operators between adjacent non-combinable operators are determined as operator subsequences. Finally, operator subsequences containing only one operator are deleted, and the remaining ones are the determined operator subsequences. For example, if an original model's operator sequence contains 15 operators, where operators 1, 2, 7, and 12 are non-combinable operators and the rest are combinable operators, then operators 3, 4, 5, and 6 form one operator subsequence, and operators 8, 9, 10, and 11 form another operator subsequence.

[0084] In step S103, multiple operators in each operator subsequence are merged into the corresponding target operator to obtain the target model.

[0085] Combining multiple operators can create a new operator. The input data of the first operator is fed into this new operator, which then outputs the output data of the last operator. In other words, this new operator can effectively perform the data processing that the multiple operators combined could. However, it's understandable that compared to the data processing performed by the multiple operators, this new operator significantly reduces memory access and noticeably improves processing speed.

[0086] By merging multiple operators in each operator subsequence into the corresponding target operator, the number of operators in the entire model is reduced, resulting in optimization. Compared to the original model, the optimized target model has fewer operators, thus significantly reducing memory accesses during data processing and noticeably improving processing efficiency. In a possible scenario, the original model is a neural network model running on a chip such as an AI accelerator chip. Each operator in the original model accesses the chip's memory during computation, leading to low computational efficiency and potentially causing errors due to frequent memory accesses. Optimizing the neural network model within the chip using this method—updating the original model to the target model—reduces the number of operators, thus reducing memory accesses during target model execution. This improves computational efficiency and prevents errors caused by frequent memory accesses.

[0087] It is understandable that memory access optimization cannot alter the functionality of the original model; otherwise, it will damage the model and cause it to lose its original data processing capabilities. Therefore, after obtaining the target model, test data can be input into both the original model and the target model to obtain the output data of the original model and the target model, respectively. Then, if the output data of the original model and the target model are consistent, the original model is updated to the target model. The test data refers to the data targeted by the original model's function. For example, if the original model processes image data, the test data is image data; if the original model processes audio data, the test data is audio data. This testing process determines whether the target model can perform the functions of the original model. If it cannot perform the functions of the original model, the target model can be deleted, and the original model can be re-optimized. If it can perform the functions of the original model, the original model can be deleted and replaced with the target model, thereby reducing the model's memory usage.

[0088] As can be seen from the above embodiments, by obtaining the operator sequence in the original model, at least one operator subsequence can be determined in the operator sequence according to the type of each operator. Since the operator subsequence includes multiple operators that can be merged, multiple operators in each operator subsequence can be merged into the corresponding target operator, thereby completing the optimization of the original model and obtaining the target model. Operator merging can reduce memory access during model operation, thereby reducing computation time and improving computational efficiency.

[0089] In some embodiments of this disclosure, multiple operators in an operator subsequence can be merged according to the type of each operator in the subsequence, using a corresponding merging method. The multiple operators in the operator subsequence may include at least two of the following: output fusionable type, data rearrangement type, data access type, and single-element optimization type. The merging process will be described in detail below using examples of operator subsequences composed of output fusionable type and data rearrangement type, output fusionable type and data access type, output fusionable type, data rearrangement type, and data access type, and output fusionable type and single-element optimization type.

[0090] In one possible embodiment, the multiple operators in the operator subsequence to be merged are output fusionable operators and data rearrangement operators. In this case, a first mapping relationship can be determined first based on the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement. Next, based on the first mapping relationship, the output position of each data element of the output data of the output fusionable operator is adjusted accordingly to obtain the target operator.

[0091] For example, the output fusion-like operator is the convolution operator. The data rearrangement operator is the permute operator, which can rearrange the input data. Please refer to the appendix. Figure 2 It illustrates the data rearrangement process of the Permute operator, i.e., the original data is... Its shape is (2, 3), and after using the permute((1, 0)) operation, the data becomes For example, using row-major storage, the position index corresponding to the original data can be obtained as follows: However, the position index of the data after rearranging using permute((1, 0)) changes. Please refer to the appendix for details. Figure 3A and attached Figure 3BThe diagram shows the position indices corresponding to the original data and the rearranged data, respectively. By comparing the position indices corresponding to the original data and the rearranged data, the following table (i.e., the first mapping relationship) can be obtained to characterize the mapping relationship of the output position of each data element before and after data rearrangement.

[0092]

[0093] The output position of each data element of the output data of the convolution operator can be adjusted according to the above mapping table. That is, the output position of the data element whose output position has changed is updated. In this way, the target operator can directly output each data element in the result of the convolution operation to the position after data rearrangement, thereby completing the data processing of the convolution operator and the data rearrangement operator at the same time.

[0094] In one possible embodiment, the multiple operators in the operator subsequence to be merged are output fusionable operators and data access operators. In this case, the output position of at least one accessed element among the multiple data elements in the output data of the output fusionable operator can be determined first based on the data access operator. Next, based on the output position of the at least one accessed element, the output data of the output fusionable operator is adjusted to the at least one accessed element to obtain the target operator.

[0095] For example, output fusion-like operators are convolution operators. Data access operators also acquire target data without changing the data distribution; for example, data access-like operators are append-only operators. Figure 4 The operator shown acquires data at intervals of 2, reading data from the original data at intervals of 2. If the output of the convolution operator is... Figure 4 The format of the original data shown in the figure allows the output of the convolution operator to be adjusted to the data elements at the first and third element positions of the original output data. This eliminates the need to output all the data before performing a data memory access operation, effectively avoiding the problem of multiple data memory accesses. Furthermore, data that is no longer used is not calculated, which effectively improves the calculation speed.

[0096] In one possible embodiment, the multiple operators in the operator subsequence to be merged are an operator subsequence composed of an output fusionable operator, a data rearrangement operator, and a data access operator. In this case, a first mapping relationship can be determined first based on the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement. Next, the output position of at least one accessed element among the multiple data elements in the output data of the output fusionable operator is determined based on the data access operator. Finally, based on the first mapping relationship, the output position of each data element in the output data of the output fusionable operator is adjusted accordingly, and the output data of the output fusionable operator is adjusted to the at least one accessed element based on the output position of the at least one accessed element, thereby obtaining the target operator. The specific execution methods and examples of the above steps have been described in detail in the previous two embodiments, and will not be repeated here.

[0097] In one possible embodiment, the multiple operators in the operator subsequence to be merged are an output fusionable operator and a single-element processing operator. The parameters of the output kernel fusion operator can then be adjusted using the content of the single-element processing operator.

[0098] For example, the output can be fused using a convolution operator, and the single-element processing operator is a batch normalization (BN) operator. The convolution operator parameters include the weights corresponding to the convolution. And the corresponding bias term b:

[0099]

[0100] in Refers to the input feature map. This is the output of the convolution calculation.

[0101] The BN operator includes four parameters. , representing the scale factor, shift factor, mean, and variance of the input data, respectively. The corresponding calculation formulas are:

[0102]

[0103] in For input to the upper layer, To prevent the denominator from being 0, It is the output of the BN layer calculation.

[0104] We can then merge the above operators:

[0105]

[0106] Simple formula transformations:

[0107]

[0108] make ,but

[0109] As can be seen, merging convolution and BN can be combined into a single convolution operator, which completely reduces the computation and memory access caused by the BN operator and effectively improves the computation speed.

[0110] Understandably, other single-element optimization operators such as ReLU can also be merged with output-fusionable operators such as convolution and fully connected operators in the same way.

[0111] In one embodiment of this disclosure, the original model is a neural network model within a chip for image processing. For example, this model can identify whether a mask is present on a person's face in an image to be detected, or identify person information, etc. In this neural network model, there is a sequence of operators consisting of multiple operators. Each operator is used to process the image to be detected or its intermediate features. Taking the channel compression processing of the intermediate features of the image to be detected as an example, it requires a subsequence of two operators. One operator is a convolution operator, used to perform convolution processing on the intermediate feature. The other operator is a data access operator, used to read data from the output of the convolution operator at intervals of 2. Therefore, the convolution operator needs to read (first access) the intermediate feature (obtained by other operators) from the chip's memory, then perform convolution processing on the intermediate feature, and store the convolution processing result (second access) in the chip's memory. Next, the data access operator reads (third access) the convolution processing result from the chip's memory, performs data reading on the convolution processing result at intervals of 2, and then stores the read result (fourth access) in the chip's memory. When using the method provided by this disclosure... The method merges each operator subsequence in the operator sequence of the original model to obtain the target model. In the target model, the operator subsequence used for channel compression processing of the intermediate features of the image to be detected is merged into the corresponding target operator (the specific merging method has been described in detail in the above embodiments and will not be repeated here). This target operator can simultaneously complete the convolution processing of intermediate features and data reading at intervals of 2. Therefore, the target operator needs to read (first access) the intermediate features in the chip's memory, then perform convolution operations on the intermediate features and read data at intervals of 2, and then store the reading results (second access) in the chip's memory. Comparing the operators before and after merging, it can be seen that the merged target operator reduces the number of memory accesses by two compared to the operator subsequence before merging. Similarly, each operator subsequence of the original model can reduce the number of memory accesses after merging, thereby reducing the chip's memory access during image processing, improving the chip's computational efficiency, and avoiding operational errors caused by frequent memory accesses.

[0112] According to a second aspect of the embodiments of this disclosure, a memory access optimization apparatus is provided. Please refer to the appendix. Figure 5 The device includes:

[0113] The acquisition module 501 is used to acquire the operator sequence in the original model;

[0114] The filtering module 502 is used to determine at least one operator subsequence in the operator sequence according to the type of each operator in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged;

[0115] The merging module 503 is used to merge multiple operators in each operator subsequence into a corresponding target operator to obtain the target model.

[0116] In some embodiments of this disclosure, the original model includes a neural network model.

[0117] In some embodiments of this disclosure, the filtering module is specifically used for:

[0118] Based on the type of each operator in the operator sequence, determine whether the corresponding operator belongs to the combinable operators;

[0119] A sequence consisting of multiple consecutive mergeable operators is defined as an operator subsequence.

[0120] In some embodiments of this disclosure, when the filtering module is used to determine whether a corresponding operator belongs to a mergeable operator based on the type of each operator in the operator sequence, it is specifically used for:

[0121] If the type of the operator exists in a preset type list, it is determined that the operator belongs to the mergeable operator category;

[0122] If the type of the operator does not exist in the preset type list, it is determined that the operator is not a mergeable operator.

[0123] In some embodiments of this disclosure, the preset category list includes output fusion class, data rearrangement class, data access class, and single element processing class.

[0124] In some embodiments of this disclosure, the merging module is specifically used for:

[0125] In the case where the plurality of operators are output fusion operators and data rearrangement operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship between the output position of each data element before and after data rearrangement;

[0126] Based on the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly to obtain the target operator.

[0127] In some embodiments of this disclosure, the merging module is specifically used for:

[0128] When the plurality of operators are output-fusionable operators and data access operators, the output position of at least one accessed element among the plurality of data elements in the output data of the output-fusionable operator is determined according to the data access operator.

[0129] Based on the output position of the at least one accessed element, the output data of the output fusion operator is adjusted to the at least one accessed element to obtain the target operator.

[0130] In some embodiments of this disclosure, the merging module is specifically used for:

[0131] When the plurality of operators are sequentially arranged output fusion operators, data rearrangement operators, and data access operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement;

[0132] Based on the data access operator, determine the output position of at least one accessed element among multiple data elements in the output data of the output fusion operator;

[0133] According to the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly, and the output data of the output fusion operator is adjusted to the at least one accessed element according to the output position of the at least one accessed element, so as to obtain the target operator.

[0134] In some embodiments of this disclosure, the merging module is specifically used for:

[0135] In the case where the multiple operators are output-fusionable operators and single-element processing operators, the parameters of the output-fusionable operators are adjusted according to the parameters of the single-element processing operators.

[0136] In some embodiments of this disclosure, a testing module is also included, for:

[0137] The test data is input into the original model and the target model respectively to obtain the output data of the original model and the output data of the target model;

[0138] If the output data of the original model and the output data of the target model are consistent, the original model is updated to the target model.

[0139] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments of the method in the third aspect, and will not be elaborated upon here.

[0140] Thirdly, at least one embodiment of this disclosure provides an apparatus, please refer to the appendix. Figure 6The diagram illustrates the structure of the device, which includes a memory and a processor. The memory stores computer instructions that can run on the processor, and the processor processes the model based on the method described in any of the first aspects when executing the computer instructions.

[0141] Fourthly, at least one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects.

[0142] In this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0144] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A memory access optimization method, characterized in that, include: Obtain the sequence of operators in the original model, which is used to process image data or sound data; Based on the type of each operator in the operator sequence, at least one operator subsequence is determined in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged; Multiple operators in each operator subsequence are merged into the corresponding target operator to obtain the target model; The step of merging multiple operators in each operator subsequence into a corresponding target operator includes: In the case where the plurality of operators are output fusion operators and data rearrangement operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship between the output position of each data element before and after data rearrangement; Based on the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly to obtain the target operator.

2. The memory access optimization method according to claim 1, characterized in that, The original model includes a neural network model.

3. The memory access optimization method according to claim 1, characterized in that, The step of determining at least one operator subsequence in the operator sequence based on the type of each operator in the operator sequence includes: Based on the type of each operator in the operator sequence, determine whether the corresponding operator belongs to the combinable operators; A sequence consisting of multiple consecutive mergeable operators is defined as an operator subsequence.

4. The memory access optimization method according to claim 3, characterized in that, The step of determining whether a corresponding operator belongs to a mergeable operator based on the type of each operator in the operator sequence includes: If the type of the operator exists in a preset type list, it is determined that the operator belongs to the mergeable operator category; If the type of the operator does not exist in the preset type list, it is determined that the operator is not a mergeable operator.

5. The memory access optimization method according to claim 4, characterized in that, The preset category list includes output fusion class, data rearrangement class, data access class, and single element processing class.

6. The memory access optimization method according to claim 4, characterized in that, The step of merging multiple operators in each operator subsequence into a corresponding target operator includes: When the plurality of operators are output-fusionable operators and data access operators, the output position of at least one accessed element among the plurality of data elements in the output data of the output-fusionable operator is determined according to the data access operator. Based on the output position of the at least one accessed element, the output data of the output fusion operator is adjusted to the at least one accessed element to obtain the target operator.

7. The memory access optimization method according to claim 4, characterized in that, The step of merging multiple operators in each operator subsequence into a corresponding target operator includes: When the plurality of operators are sequentially arranged output fusion operators, data rearrangement operators, and data access operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship of the output position of each data element before and after data rearrangement; Based on the data access operator, determine the output position of at least one accessed element among multiple data elements in the output data of the output fusion operator; According to the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly, and the output data of the output fusion operator is adjusted to the at least one accessed element according to the output position of the at least one accessed element, so as to obtain the target operator.

8. The memory access optimization method according to claim 1, characterized in that, Also includes: The test data is input into the original model and the target model respectively to obtain the output data of the original model and the output data of the target model; If the output data of the original model and the output data of the target model are consistent, the original model is updated to the target model.

9. A memory access optimization device, characterized in that, include: The acquisition module is used to acquire the operator sequence in the original model, which is used to process image data or sound data. A filtering module is used to determine at least one operator subsequence in the operator sequence according to the type of each operator in the operator sequence, wherein the operator subsequence includes multiple operators that can be merged; The merging module is used to merge multiple operators in each operator subsequence into a corresponding target operator to obtain the target model; In the case where the plurality of operators are output fusion operators and data rearrangement operators, a first mapping relationship is determined according to the data rearrangement operator, wherein the first mapping relationship is used to characterize the mapping relationship between the output position of each data element before and after data rearrangement; Based on the first mapping relationship, the output position of each data element of the output data of the output fusion operator is adjusted accordingly to obtain the target operator.

10. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 8 when executing the computer instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 8.

Images