A method, device, equipment and medium for operating a neural network model
By obtaining and accumulating the memory information of each operator in the neural network model and selecting the appropriate running device, the problem that the neural network model cannot run stably due to insufficient memory is solved, and its stable operation in various application scenarios is achieved.
Patent Information
- Application Number
- CN202411831084.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-04
AI Technical Summary
The neural network model cannot run stably when the memory of the running device is insufficient, which affects its application value.
By obtaining multiple neural network operators in the target neural network model, the memory information of each operator is determined, and the information is accumulated to obtain the target memory information, so as to select a suitable operating device to ensure stable operation.
The memory information required for the neural network model is accurately determined and the device that supports its operation is selected, thus ensuring the stable operation of the neural network model.
Smart Images
Figure CN119759561A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of Internet technology, and in particular to a method, device, equipment and medium for operating a neural network model. Background Art
[0002] At present, with the rapid development of science and technology, neural network models have shown extraordinary application value in many fields with their powerful data analysis and processing capabilities. However, if neural network models want to continue to play their advantages in various application scenarios, their stable operation is of utmost importance.
[0003] At present, one factor that affects the stable operation of the neural network model is that the memory of the running device on which the neural network model is deployed may not be sufficient to support the operation of the neural network model, resulting in the instability of the neural network model to run. Summary of the invention
[0004] The embodiments of the present invention provide a method, device, equipment and medium for operating a neural network model, which can ensure the stable operation of the neural network model.
[0005] In a first aspect, an embodiment of the present invention provides a method for operating a neural network model, comprising:
[0006] Acquire multiple neural network operators in the target neural network model; wherein the neural network operator includes at least one of an input operator, at least one intermediate operator and an output operator;
[0007] Determining first memory information required for each neural network operator;
[0008] Accumulating first memory information corresponding to multiple neural network operators to obtain target memory information;
[0009] Based on the target memory information, a target operating device corresponding to the target neural network model is determined, and the target neural network model is operated in the target operating device.
[0010] In a second aspect, an embodiment of the present invention further provides a running device of a neural network model, the device comprising:
[0011] A neural network operator acquisition module, used to acquire multiple neural network operators in a target neural network model; wherein the neural network operator includes an input operator, at least one intermediate operator and an output operator;
[0012] A first memory information determination module, used to determine the first memory information required by each neural network operator;
[0013] A target memory information determination module, used for accumulating first memory information corresponding to a plurality of neural network operators to obtain target memory information;
[0014] A target operation device determination module is used to determine the target operation device corresponding to the target neural network model based on the target memory information, and to operate the target neural network model in the target operation device.
[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for operating the neural network model described in the embodiment of the present invention.
[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operating method of the neural network model described in the embodiment of the present invention when executed.
[0020] The embodiment of the present invention discloses a method, device, equipment and medium for running a neural network model, including: obtaining multiple neural network operators in a target neural network model; wherein the neural network operator includes at least one of an input operator, at least one intermediate operator and an output operator; determining the first memory information required for each neural network operator; accumulating the first memory information corresponding to multiple neural network operators to obtain the target memory information; determining the target running device corresponding to the target neural network model based on the target memory information, and running the target neural network model in the target running device. In the above technical solution, before running the target neural network model, the first memory information required for each neural network operator in the target neural network model is pre-determined, and the first memory information corresponding to each neural network operator is accumulated, so that the target memory information corresponding to the target neural network model can be accurately determined; then, the target running device corresponding to the target neural network model is determined according to the obtained target memory information, and the target running device that supports the running of the target neural network model can be accurately determined, so as to ensure the stable running of the target neural network model. Compared with the prior art, the technical solution provided by this embodiment can accurately determine the target memory information corresponding to the target neural network model, and can accurately determine the target running device that supports the running of the target neural network model, so as to ensure the stable running of the target neural network model. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 is a flow chart of a method for operating a neural network model in Embodiment 1 of the present invention;
[0022] Figure 2 is a flow chart of a method for operating a neural network model in Embodiment 2 of the present invention;
[0023] Figure 3 It is a schematic diagram of the structure of a running device of a neural network model in Embodiment 3 of the present invention;
[0024] Figure 4 It is a structural schematic diagram of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0026] Embodiment 1
[0027] Figure 1This is a flowchart of a method for operating a neural network model provided in the first embodiment of the present invention. This embodiment is applicable to the case of operating a neural network model. The method can be executed by an operating device for a neural network model. The device can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server. Specifically, it includes the following steps:
[0028] S110. Obtain multiple neural network operators in the target neural network model.
[0029] In this embodiment, the target neural network model is the research object of this embodiment, and the memory size required by the target neural network model needs to be determined in the subsequent process. The neural network operator can be understood as an operating element in the target neural network model, and can be regarded as a module that performs specific operations or data processing tasks. Specifically, the neural network operator may include at least one of an input operator, at least one intermediate operator, and an output operator. Among them, the input operator is an operator that receives input data in the target neural network model, the output operator is an operator that outputs the final result in the target neural network model, and the intermediate operator is other operators in the target neural network model except the input operator and the output operator.
[0030] Exemplarily, it is assumed that the target neural network model includes a convolution operator, a pooling operator, a fully connected operator, and an activation function operator connected in sequence, wherein the convolution operator, the pooling operator, the fully connected operator, and the activation function operator are neural network operators. Further, the convolution operator is an input operator, the pooling operator and the fully connected operator are intermediate operators, and the activation function operator is an output operator.
[0031] Continuing with the above description, the target neural network model satisfies the directed acyclic condition and the attribute information of each neural network operator is a static shape. Optionally, the method for determining whether the target neural network model satisfies the directed acyclic condition may be: obtaining a computational graph corresponding to the target neural network model; if the computational graph is a directed acyclic graph, the target neural network model satisfies the directed acyclic condition; if the computational graph is a non-directed acyclic graph, the target neural network model does not satisfy the directed acyclic condition.
[0032] Furthermore, in this embodiment, the target neural network model is required to satisfy the directed acyclic condition, that is, the computational graph corresponding to the target neural network model is a directed acyclic graph. Optionally, a topological sorting method or a depth-first search method can be used to determine whether the computational graph corresponding to the target neural network model is a directed acyclic graph. In this embodiment, the attribute information of each neural network operator is a static shape. Specifically, if the shape of the input data and output data of the neural network operator is fixed, the neural network operator is a static shape operator, that is, the attribute information of the neural network operator is a static shape.
[0033] S120. Determine first memory information required by each neural network operator.
[0034] In this embodiment, the first memory information can be understood as the memory size required by the neural network operator. Furthermore, the method for determining the first memory information required for each neural network operator can be: for each neural network operator, calculate the static memory size and dynamic memory size required by the neural network operator; add the static memory size and the dynamic memory size to obtain the first memory information required by the neural network operator. Among them, static memory can be understood as memory that only needs to be allocated once during the execution of the target neural network model inference and exists throughout the entire inference execution period, and dynamic memory can be understood as memory that is allocated and released as needed during the target neural network model inference process.
[0035] S130. Accumulate the first memory information corresponding to the multiple neural network operators to obtain target memory information.
[0036] In this embodiment, the target memory information can be understood as the memory size required by the target neural network model.
[0037] Exemplarily, assuming that there are n neural network operators, and their corresponding first memory information is a1, a2, ..., an, then the n first memory information are accumulated to obtain the target memory information A, where A = a1+a2+...+an.
[0038] S140. Determine the target operating device corresponding to the target neural network model based on the target memory information, and operate the target neural network model in the target operating device.
[0039] In this embodiment, the target operating device can be understood as a device that can support the operation of the target neural network model. Optionally, the method of determining the target operating device corresponding to the target neural network model based on the target memory information can be: determining the target operating device corresponding to the target neural network model from multiple candidate operating devices based on the target memory information.
[0040] The technical solution of this embodiment is to obtain multiple neural network operators in the target neural network model; determine the first memory information required by each neural network operator; accumulate the first memory information corresponding to the multiple neural network operators to obtain the target memory information; determine the target operation device corresponding to the target neural network model based on the target memory information, and run the target neural network model in the target operation device. In the above technical solution, before running the target neural network model, the first memory information required by each neural network operator in the target neural network model is pre-determined, and the first memory information corresponding to each neural network operator is accumulated, so that the target memory information corresponding to the target neural network model can be accurately determined; then, the target operation device corresponding to the target neural network model is determined according to the obtained target memory information, and the target operation device that supports the operation of the target neural network model can be accurately determined, so as to ensure the stable operation of the target neural network model. The technical solution provided by this embodiment can accurately determine the target memory information corresponding to the target neural network model, and can accurately determine the target operation device that supports the operation of the target neural network model, so as to ensure the stable operation of the target neural network model.
[0041] Embodiment 2
[0042] Figure 2 This is a flowchart of a method for operating a neural network model provided in Embodiment 2 of the present invention. Based on the above embodiment, the method includes the following steps:
[0043] S210. Obtain multiple neural network operators in the target neural network model.
[0044] S220. For each neural network operator, determine the static memory information and dynamic memory information required by the neural network operator.
[0045] In this embodiment, the static memory information can be understood as the static memory size required by the neural network operator; the dynamic memory information can be understood as the dynamic memory size required by the neural network operator.
[0046] In this embodiment, if the neural network operator includes at least one parameter and at least one constant, one implementation method of determining the static memory information required by the neural network operator can be described as the following steps:
[0047] a1) obtaining at least one parameter and at least one constant in a neural network operator;
[0048] In this embodiment, the parameters may be parameters such as weights and biases, and the constants may be understood as constants in the neural network operator, such as certain fixed attribute values. For example, taking the convolution operator as an example, the constants may be strides, kernel shape, and pads. Furthermore, all implementations of neural network operators correspond to clear calculation formulas, such as the calculation formula corresponding to the convolution operator is
[0049] b1) if the neural network operator is an input operator or an intermediate operator, determining static memory information required by the neural network operator based on a data type of at least one parameter and a data type of at least one constant;
[0050] In this embodiment, the data type may be float32 type, float16 type, int8 type, etc.
[0051] In this embodiment, for determining the static memory information required by the neural network operator based on the data type of at least one parameter and the data type of at least one constant, one implementation method can be described as the following steps:
[0052] (b11) for each parameter, determining a first byte number corresponding to the parameter based on a data type of the parameter, and determining the first byte number as second memory information required for the parameter;
[0053] In this embodiment, different data types occupy different numbers of bytes, for example: float32 type occupies 4 bytes; float16 type occupies 2 bytes; int8 type occupies 1 byte. For example, if the data type of a parameter is float32 type, the first byte number corresponding to the parameter is 4 bytes.
[0054] Exemplarily, based on the calculation formula of the above convolution operator, the parameters included in the convolution operator are: (s×k×k×c) weights W and s biases b, and then the first number of bytes corresponding to the parameters is determined according to the data type of the parameters, and the first number of bytes is determined as the second memory information required for the parameters. Assume that the second memory information required for each weight parameter is size_of_elem(W), and the second memory information required for each bias parameter is size_of_elem(b).
[0055] (b12) for each constant, determining a second byte number corresponding to the constant based on a data type of the constant, and determining the second byte number as third memory information required for the constant;
[0056] For example, assume that the convolution operator contains constants: 2 strides, 2 kernel shapes, and 4 pads, then determine the second number of bytes corresponding to the constant according to the data type of the constant, and determine the second number of bytes as the third memory information required by the constant. Assume that the third memory information required for each strides is size_of_elem(strides), the third memory information required for each kernel shape is size_of_elem(kernel shape), and the third memory information required for each pads is size_of_elem(pads).
[0057] (b13) Accumulating the second memory information corresponding to each parameter and the third memory information corresponding to each constant to obtain the static memory information required by the neural network operator.
[0058] Exemplarily, the second memory information corresponding to each parameter and the third memory information corresponding to each constant are accumulated using the following formula to obtain the static memory information J required by the neural network operator: J = s×k×k×c×size_of_elem(W)+s×size_of_elem(b)+2×size_of_elem(strides)+2×size_of_elem(kernelshape)+4×size_of_elem(pads).
[0059] c1) If the neural network operator is an output operator, obtain at least one output data corresponding to the neural network operator, and determine the static memory information required by the neural network operator based on the data type of the at least one output data, the data type of at least one parameter, and the data type of at least one constant.
[0060] In this embodiment, the method for determining the static memory information required by the neural network operator based on the data type of at least one output data, the data type of at least one parameter, and the data type of at least one constant can be: for each parameter, the first byte number corresponding to the parameter is determined based on the data type of the parameter, and the first byte number is determined as the second memory information required for the parameter; for each constant, the second byte number corresponding to the constant is determined based on the data type of the constant, and the second byte number is determined as the third memory information required for the constant; for each output data, the third byte number corresponding to the output data is determined based on the data type of the output data, and the third byte number is determined as the fourth memory information required for the output data; the second memory information corresponding to each parameter, the third memory information corresponding to each constant, and the fourth memory information corresponding to each output data are accumulated to obtain the static memory information required by the neural network operator.
[0061] Exemplarily, if the convolution operator is an output operator, based on the calculation formula of the convolution operator, the convolution operator contains h×w×s output data Y. Assuming that the fourth memory information required for each output data Y is size_of_elem(Y), the following formula can be used to accumulate the second memory information corresponding to each parameter, the third memory information corresponding to each constant, and the fourth memory information corresponding to each output data to obtain the static memory information J required by the neural network operator:
[0062] J=s×k×k×c×size_of_elem(W)+s×size_of_elem(b)+2×size_of_elem(strides)+2×size_of_elem(kernel shape)+4×size_of_elem(pads)+h×w×s×size_of_elem(Y).
[0063] In this embodiment, if the neural network operator contains at least one parameter and does not contain a constant, one implementation method of determining the static memory information required for the neural network operator can be described as: obtaining at least one parameter in the neural network operator; if the neural network operator is an input operator or an intermediate operator, determining the static memory information required for the neural network operator based on the data type of at least one parameter; if the neural network operator is an output operator, obtaining at least one output data corresponding to the neural network operator, and determining the static memory information required for the neural network operator based on the data type of the at least one output data and the data type of at least one parameter.
[0064] Continuing with the above description, the method for determining the static memory information required by the neural network operator based on the data type of at least one parameter can be: for each parameter, determine the first byte number corresponding to the parameter based on the data type of the parameter, and determine the first byte number as the second memory information required for the parameter; accumulate the second memory information corresponding to each parameter to obtain the static memory information required by the neural network operator.
[0065] Continuing with the above description, the method for determining the static memory information required by the neural network operator based on the data type of at least one output data and the data type of at least one parameter can be: for each parameter, determine the first byte number corresponding to the parameter based on the data type of the parameter, and determine the first byte number as the second memory information required for the parameter; for each output data, determine the third byte number corresponding to the output data based on the data type of the output data, and determine the third byte number as the fourth memory information required for the output data; accumulate the second memory information corresponding to each parameter and the fourth memory information corresponding to each output data to obtain the static memory information required by the neural network operator.
[0066] In this embodiment, if the neural network operator contains at least one constant and no parameters, one implementation method of determining the static memory information required for the neural network operator can be described as: obtaining at least one constant in the neural network operator; if the neural network operator is an input operator or an intermediate operator, determining the static memory information required for the neural network operator based on the data type of at least one constant; if the neural network operator is an output operator, obtaining at least one output data corresponding to the neural network operator, and determining the static memory information required for the neural network operator based on the data type of the at least one output data and the data type of at least one constant.
[0067] Continuing with the above description, the method for determining the static memory information required by the neural network operator based on the data type of at least one constant can be: for each constant, determine the second byte number corresponding to the constant based on the data type of the constant, and determine the second byte number as the third memory information required for the constant; accumulate the third memory information corresponding to each constant to obtain the static memory information required by the neural network operator.
[0068] Continuing with the above description, the method for determining the static memory information required by the neural network operator based on the data type of at least one output data and the data type of at least one constant can be: for each constant, determine the second byte number corresponding to the constant based on the data type of the constant, and determine the second byte number as the third memory information required for the constant; for each output data, determine the third byte number corresponding to the output data based on the data type of the output data, and determine the third byte number as the fourth memory information required for the output data; accumulate the third memory information corresponding to each constant and the fourth memory information corresponding to each output data to obtain the static memory information required by the neural network operator.
[0069] In this embodiment, if the neural network operator contains neither constants nor parameters, one implementation method of determining the static memory information required for the neural network operator can be described as: if the neural network operator is an input operator or an intermediate operator, the static memory information required by the neural network operator is 0; if the neural network operator is an output operator, at least one output data corresponding to the neural network operator is obtained, and the static memory information required for the neural network operator is determined based on the data type of the at least one output data.
[0070] Continuing with the above description, the method for determining the static memory information required by the neural network operator based on the data type of at least one output data can be: for each output data, determine the third byte number corresponding to the output data based on the data type of the output data, and determine the third byte number as the fourth memory information required for the output data; accumulate the fourth memory information corresponding to each output data to obtain the static memory information required by the neural network operator.
[0071] In this embodiment, one implementation method of determining the dynamic memory information required by the neural network operator can be described as the following steps:
[0072] a2) obtaining at least one intermediate data in a neural network operator;
[0073] In this embodiment, the intermediate data can be understood as the intermediate result data of the neural network operator in the calculation process, such as W×X in the calculation formula of the convolution operator, where there are a total of h×w×s×k×k×c intermediate data (W×X).
[0074] b2) if the neural network operator is an output operator, determining the dynamic memory information required by the neural network operator based on the data type of at least one intermediate data;
[0075] In this embodiment, the method for determining the dynamic memory information required by the neural network operator based on the data type of at least one intermediate data can be: for each intermediate data, determine the fourth byte number corresponding to the intermediate data based on the data type of the intermediate data, and determine the fourth byte number as the fifth memory information required for the intermediate data; accumulate the fifth memory information corresponding to each intermediate data to obtain the dynamic memory information required by the neural network operator.
[0076] Exemplarily, if the convolution operator is an output operator, assuming that the fifth memory information required for each intermediate data (W×X) is size_of_inner, the fifth memory information corresponding to each intermediate data can be accumulated using the following formula to obtain the dynamic memory information D required by the neural network operator.
[0077] D=h×w×s×k×k×c×size_of_inner
[0078] Following the above description, the data type of the intermediate data can be determined using implicit type conversion rules. For example, if the intermediate data is W×X, the number of bytes corresponding to the data types of W and X is determined; the data type with the largest number of bytes is used as the data type of the intermediate data.
[0079] c2) If the neural network operator is an input operator or an intermediate operator, obtain at least one output data corresponding to the neural network operator, and determine the dynamic memory information required by the neural network operator based on the data type of the at least one output data and the data type of the at least one intermediate data.
[0080] In this embodiment, the method for determining the dynamic memory information required by the neural network operator based on the data type of at least one output data and the data type of at least one intermediate data can be: for each output data, determine the third byte number corresponding to the output data based on the data type of the output data, and determine the third byte number as the fourth memory information required for the output data; for each intermediate data, determine the fourth byte number corresponding to the intermediate data based on the data type of the intermediate data, and determine the fourth byte number as the fifth memory information required for the intermediate data; accumulate the fourth memory information corresponding to each output data and the fifth memory information corresponding to each intermediate data to obtain the dynamic memory information required by the neural network operator.
[0081] Exemplarily, if the convolution operator is an input operator or an intermediate operator, the convolution operator includes h×w×s output data Y. Assuming that the fourth memory information required for each output data Y is size_of_elem(Y), the following formula can be used to accumulate the fourth memory information corresponding to each output data and the fifth memory information corresponding to each intermediate data to obtain the dynamic memory information D required by the neural network operator.
[0082] D=h×w×s×size_of_elem(Y)+h×w×s×k×k×c×size_of_inner
[0083] S230, accumulating the static memory information and the dynamic memory information to obtain the first memory information required by the neural network operator.
[0084] Exemplarily, the above-mentioned static memory information J and dynamic memory information D are added to obtain the first memory information (D+J) required by the neural network operator.
[0085] S240. Accumulate the first memory information corresponding to the multiple neural network operators to obtain target memory information.
[0086] S250: Obtain remaining memory information of multiple candidate running devices.
[0087] In this embodiment, the candidate running device can be understood as an alternative device for running the target neural network model, and each candidate running device has corresponding remaining memory information, that is, the remaining memory size. Exemplarily, the candidate running device can be a desktop computer or server with different memory configurations.
[0088] S260. The candidate operating device whose remaining memory information is greater than the target memory information is used as the target operating device, and the target neural network model is run in the target operating device.
[0089] Exemplarily, assuming that there are m candidate operating devices, a candidate operating device whose remaining memory information is greater than the target memory information is selected from the m candidate operating devices as the target operating device. The target operating device may include one or more candidate operating devices. If the target operating device includes multiple candidate operating devices, a candidate operating device is randomly selected from them to run the target neural network model.
[0090] The technical solution of this embodiment is to obtain multiple neural network operators in the target neural network model; for each neural network operator, determine the static memory information and dynamic memory information required by the neural network operator; accumulate the static memory information and dynamic memory information to obtain the first memory information required by the neural network operator; accumulate the first memory information corresponding to multiple neural network operators to obtain the target memory information; obtain the remaining memory information of multiple candidate operating devices; take the candidate operating device whose remaining memory information is greater than the target memory information as the target operating device, and operate the target neural network model in the target operating device. In the above technical solution, before operating the target neural network model, the static memory information and dynamic memory information required by each neural network operator in the target neural network model are pre-determined, and the static memory information and dynamic memory information are accumulated to accurately determine the first memory information required by the neural network operator; then, the first memory information corresponding to each neural network operator is accumulated, so that the target memory information corresponding to the target neural network model can be accurately determined; finally, the candidate operating device whose remaining memory information is greater than the target memory information is taken as the target operating device, so that the target operating device that supports the operation of the target neural network model can be accurately determined from multiple candidate operating devices, thereby ensuring the stable operation of the target neural network model. The technical solution provided in this embodiment can accurately determine the first memory information required by the neural network operator and the target memory information corresponding to the target neural network model, and can accurately determine the target operating device that supports the operation of the target neural network model from multiple candidate operating devices, thereby ensuring the stable operation of the target neural network model.
[0091] Embodiment 3
[0092] Figure 3 A schematic diagram of the structure of a running device of a neural network model provided in Embodiment 3 of the present invention is shown in FIG. Figure 3 As shown, the device comprises:
[0093] A neural network operator acquisition module 310 is used to acquire multiple neural network operators in a target neural network model; wherein the neural network operator includes at least one of an input operator, at least one intermediate operator and an output operator;
[0094] A first memory information determination module 320, used to determine the first memory information required by each neural network operator;
[0095] A target memory information determination module 330 is used to accumulate first memory information corresponding to multiple neural network operators to obtain target memory information;
[0096] The target operation device determination module 340 is used to determine the target operation device corresponding to the target neural network model based on the target memory information, and to operate the target neural network model in the target operation device.
[0097] Optionally, the target neural network model satisfies the directed acyclic condition and the attribute information of each neural network operator is a static shape.
[0098] Optionally, the first memory information determining module 320 may specifically include:
[0099] A memory information determination unit, used to determine, for each neural network operator, static memory information and dynamic memory information required by the neural network operator;
[0100] The first memory information obtaining unit is used to accumulate the static memory information and the dynamic memory information to obtain the first memory information required by the neural network operator.
[0101] Optionally, the memory information determining unit may specifically include:
[0102] A parameter acquisition subunit, used to acquire at least one parameter and at least one constant in the neural network operator;
[0103] A static memory information determination subunit is used to determine the static memory information required by the neural network operator based on the data type of at least one parameter and the data type of at least one constant if the neural network operator is an input operator or an intermediate operator; if the neural network operator is an output operator, obtain at least one output data corresponding to the neural network operator, and determine the static memory information required by the neural network operator based on the data type of the at least one output data, the data type of at least one parameter, and the data type of at least one constant.
[0104] Optionally, the static memory information determination subunit may be specifically used for:
[0105] For each parameter, determining a first number of bytes corresponding to the parameter based on a data type of the parameter, and determining the first number of bytes as second memory information required by the parameter;
[0106] For each constant, determine a second byte number corresponding to the constant based on a data type of the constant, and determine the second byte number as third memory information required by the constant;
[0107] The second memory information corresponding to each parameter and the third memory information corresponding to each constant are accumulated to obtain static memory information required by the neural network operator.
[0108] Optionally, the memory information determining unit may further include:
[0109] An intermediate data acquisition subunit, used to acquire at least one intermediate data in the neural network operator;
[0110] A dynamic memory information determination subunit is used to determine the dynamic memory information required by the neural network operator based on the data type of at least one intermediate data if the neural network operator is an output operator; if the neural network operator is an input operator or an intermediate operator, obtain at least one output data corresponding to the neural network operator, and determine the dynamic memory information required by the neural network operator based on the data type of at least one output data and the data type of at least one intermediate data.
[0111] Optionally, the target operating device determination module 340 may be specifically used to:
[0112] Remaining memory information of a plurality of candidate operating devices is obtained; and a candidate operating device whose remaining memory information is greater than the target memory information is taken as a target operating device.
[0113] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.
[0114] Embodiment 4
[0115] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0116] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0117] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0118] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the operation method of the neural network model.
[0119] In some embodiments, the method for operating the neural network model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for operating the neural network model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for operating the neural network model in any other appropriate manner (e.g., by means of firmware).
[0120] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0121] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0122] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0123] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0124] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0125] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0126] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a method for operating a neural network model as provided in any embodiment of the present application.
[0127] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for operating a neural network model, characterized in that: include: Acquire multiple neural network operators in the target neural network model; wherein the neural network operator includes at least one of an input operator, at least one intermediate operator and an output operator; Determining first memory information required for each neural network operator; Accumulating first memory information corresponding to multiple neural network operators to obtain target memory information; Based on the target memory information, a target operating device corresponding to the target neural network model is determined, and the target neural network model is operated in the target operating device.
2. The method according to claim 1, characterized in that The target neural network model satisfies the directed acyclic condition and the attribute information of each neural network operator is a static shape.
3. The method according to claim 1, characterized in that Determine the first memory information required for each neural network operator, including: For each neural network operator, determining static memory information and dynamic memory information required by the neural network operator; The static memory information and the dynamic memory information are accumulated to obtain the first memory information required by the neural network operator.
4. The method according to claim 3, characterized in that Determine the static memory information required by the neural network operator, including: Obtaining at least one parameter and at least one constant in the neural network operator; If the neural network operator is an input operator or an intermediate operator, determining static memory information required by the neural network operator based on a data type of at least one parameter and a data type of at least one constant; If the neural network operator is an output operator, at least one output data corresponding to the neural network operator is obtained, and the static memory information required by the neural network operator is determined based on the data type of the at least one output data, the data type of at least one parameter, and the data type of at least one constant.
5. The method according to claim 4, characterized in that Determining static memory information required by the neural network operator based on a data type of at least one parameter and a data type of at least one constant includes: For each parameter, determining a first number of bytes corresponding to the parameter based on a data type of the parameter, and determining the first number of bytes as second memory information required by the parameter; For each constant, determine a second byte number corresponding to the constant based on a data type of the constant, and determine the second byte number as third memory information required by the constant; The second memory information corresponding to each parameter and the third memory information corresponding to each constant are accumulated to obtain static memory information required by the neural network operator.
6. The method according to claim 3, characterized in that Determine the dynamic memory information required by the neural network operator, including: Acquiring at least one intermediate data in the neural network operator; If the neural network operator is an output operator, determining the dynamic memory information required by the neural network operator based on a data type of at least one intermediate data; If the neural network operator is an input operator or an intermediate operator, at least one output data corresponding to the neural network operator is obtained, and the dynamic memory information required by the neural network operator is determined based on the data type of at least one output data and the data type of at least one intermediate data.
7. The method according to claim 1, characterized in that Determining a target operating device corresponding to the target neural network model based on the target memory information includes: Get the remaining memory information of multiple candidate running devices; The candidate operating device whose remaining memory information is greater than the target memory information is used as the target operating device.
8. A device for operating a neural network model, characterized in that: include: A neural network operator acquisition module, used to acquire multiple neural network operators in a target neural network model; wherein the neural network operator includes at least one of an input operator, at least one intermediate operator and an output operator; A first memory information determination module, used to determine the first memory information required by each neural network operator; A target memory information determination module, used for accumulating first memory information corresponding to a plurality of neural network operators to obtain target memory information; A target operation device determination module is used to determine the target operation device corresponding to the target neural network model based on the target memory information, and to operate the target neural network model in the target operation device.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for operating the neural network model according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operating method of the neural network model described in any one of claims 1 to 7 when executed.