Electronic device and controlling method of electronic device

Abstract

An electronic device and a method for controlling the electronic device are disclosed. Specifically, the electronic device according to the present disclosure includes a memory for storing data related to a neural network model and a processor that determines the execution order of the multiple steps by dividing a learning step performed through a plurality of layers of the neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step; based on the determined execution order, obtains first information regarding which of the multiple steps a plurality of tensors used in a plurality of layers is used in according to the determined execution order; integrates the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared; based on the integrated execution order, minimizes the memory area for allocating data corresponding to the multiple tensors and allocates data to the multiple tensors; and trains the neural network model according to the integrated execution order using the multiple tensors and the data allocated to the multiple tensors. Various other embodiments are possible.

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling an electronic device, and specifically to an electronic device capable of training a neural network model and a method for controlling the same. Background Technology

[0003] Recently, as technologies related to artificial intelligence models, machine learning, and deep learning have advanced, various types of neural network models are being implemented within individual user devices to provide various services to users.

[0004] Neural network models can be trained on a server based on large amounts of data and vast resources, and then installed and operated on individual user devices. However, there is a problem in that training solely on the server makes it difficult to personalize the model to suit the specific characteristics of each user. To address this issue, one method involves transmitting users' personal data to a server to retrain the neural network model; however, transmitting such data can be vulnerable to security threats and may lead to privacy infringement. Furthermore, there is the issue that personalizing the neural network for every user on the server incurs significant service costs.

[0005] Therefore, recently, technologies related to efficiently training neural network models on-device within individual users' terminals have been attracting attention. However, training neural network models on-device faces limitations due to the computing resources of individual user terminals and the scarcity of user data; consequently, there is a need for technologies capable of overcoming these limitations. The problem to be solved

[0007] The present disclosure is made in response to the needs described above, and the purpose of the present disclosure is to provide an electronic device and a method for controlling the electronic device that can significantly reduce memory usage during the learning process of a neural network model. means of solving the problem

[0009] According to one embodiment of the present disclosure for achieving the purpose described above, an electronic device includes a memory for storing data related to a neural network model and a processor for determining the execution order of the plurality of steps by dividing a learning step performed through a plurality of layers of the neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step, and based on the determined execution order, obtaining first information regarding which of the plurality of steps a plurality of tensors used in the plurality of layers is used in according to the determined execution order, integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared, and based on the integrated execution order, minimizing the area of ​​the memory for allocating data corresponding to the plurality of tensors and allocating the data to the plurality of tensors, and training the neural network model according to the integrated execution order using the plurality of tensors and the data allocated to the plurality of tensors.

[0010] Here, the first information may be determined based on information regarding the type of step in which the plurality of steps utilize the plurality of tensors.

[0011] Meanwhile, the types of steps in which the plurality of tensors are used may include the forward propagation step, the gradient calculation step, the derivative calculation step, the backpropagation step including the gradient calculation step and the derivative calculation step, the step including the forward propagation step and the backpropagation step, and the types representing each of the overall learning steps of the neural network model.

[0012] Meanwhile, the second information may include a first mode information indicating that the tensor is in a pre-assigned state, a second mode information indicating that the tensor must be newly created, a third mode information indicating that the tensor's data is changed but the tensor can be shared with other tensors in adjacent layers, a fourth mode information indicating that the tensor's data is not changed and therefore the tensor can be shared with other tensors, and a fifth mode information indicating that the tensor can be shared with all tensors.

[0013] Here, the processor may integrate at least a portion of the determined execution order so that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is equal to or earlier than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is first used.

[0014] Here, the processor may integrate at least a portion of the determined execution order so that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is slower than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is first used, and the second information corresponding to the second tensor is the fourth mode information.

[0015] Meanwhile, the processor can minimize the memory area by determining whether to create an additional memory area for allocating data corresponding to the plurality of tensors or to overwrite a previously created memory area based on the integrated execution order.

[0016] According to one embodiment of the present disclosure for achieving the purpose described above, a control method for an electronic device may include: a step of determining the execution order of a plurality of steps by dividing a learning step performed through a plurality of layers of a neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step; a step of obtaining first information regarding which of the plurality of steps according to the determined execution order a plurality of tensors used in the plurality of layers is used in based on the determined execution order; a step of integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared; a step of allocating data to the plurality of tensors by minimizing the memory area for allocating data corresponding to the plurality of tensors based on the integrated execution order; and a step of training the neural network model according to the integrated execution order using the plurality of tensors and the data allocated to the plurality of tensors.

[0017] Here, the first information may be determined based on information regarding the type of step in which the plurality of steps utilize the plurality of tensors.

[0018] Meanwhile, the types of steps in which the plurality of tensors are used may include the forward propagation step, the gradient calculation step, the derivative calculation step, the backpropagation step including the gradient calculation step and the derivative calculation step, the step including the forward propagation step and the backpropagation step, and the types representing each of the overall learning steps of the neural network model.

[0019] Meanwhile, the second information may include a first mode information indicating that the tensor is in a pre-assigned state, a second mode information indicating that the tensor must be newly created, a third mode information indicating that the tensor's data is changed but the tensor can be shared with other tensors in adjacent layers, a fourth mode information indicating that the tensor's data is not changed and therefore the tensor can be shared with other tensors, and a fifth mode information indicating that the tensor can be shared with all tensors.

[0020] Here, the step of integrating the determined execution order may integrate at least a portion of the determined execution order such that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is used last is equal to or earlier than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is used first.

[0021] Here, the step of integrating the determined execution order may integrate at least a portion of the determined execution order so that the first tensor and the second tensor are shared, if the execution order of the step in which the first tensor among the plurality of tensors is used last is slower than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is used first, and the second information corresponding to the second tensor is the fourth mode information.

[0022] Meanwhile, the step of allocating data to the plurality of tensors may include a step of minimizing the memory area by determining whether to additionally create a memory area for allocating data corresponding to the plurality of tensors or to overwrite a previously created memory area based on the integrated execution order.

[0023] According to one embodiment of the present disclosure for achieving the purpose described above, a non-transient computer-readable recording medium comprising a program for executing a method for controlling an electronic device, wherein the method for controlling the electronic device may include: a step of determining the execution order of the plurality of steps by dividing a learning step performed through a plurality of layers of a neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step; a step of obtaining first information regarding which of the plurality of steps according to the determined execution order a plurality of tensors used in the plurality of layers is used in based on the determined execution order; a step of integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared; a step of allocating data to the plurality of tensors by minimizing the memory area for allocating data corresponding to the plurality of tensors based on the integrated execution order; and a step of training the neural network model according to the integrated execution order using the plurality of tensors and the data allocated to the plurality of tensors. Brief explanation of the drawing

[0025] FIG. 1 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure, FIG. 2 is a drawing showing the execution sequence of a plurality of steps according to one embodiment of the present disclosure, FIG. 3 is a diagram showing a type of step in which a plurality of tensors are used according to one embodiment of the present disclosure, FIG. 4 is a drawing for explaining in detail the process of acquiring first information according to the present disclosure, FIG. 5 is a diagram showing mode information for a plurality of tensors according to one embodiment of the present disclosure, FIG. 6 is a diagram for explaining the process of integrating the execution order determined based on first information and second information, FIG. 7 is a diagram illustrating a method of allocating data to a tensor by minimizing the memory area in one embodiment of the present disclosure, FIG. 8 is a diagram illustrating a method of allocating data to a tensor by minimizing the memory area in another embodiment of the present disclosure, FIGS. 9 and FIGS. 10 are drawings for explaining a method for reducing data loading time according to an embodiment of the present disclosure, FIG. 11 is a drawing for explaining a method for adjusting the number of layers used in the computation step of a neural network model according to one embodiment of the present disclosure, FIG. 12 is a block diagram briefly illustrating the configuration of an electronic device according to one embodiment of the present disclosure, and, FIG. 13 is a block diagram showing in detail the configuration of an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0026] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0027] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0028] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0029] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0030] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0031] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0032] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0033] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component).

[0034] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between said certain component and said other component.

[0035] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0036] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0037] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0038] Meanwhile, the various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0039] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0041] FIG. 1 is a flowchart illustrating a control method of an electronic device (100) according to one embodiment of the present disclosure.

[0042] The electronic device (100) according to the present disclosure refers to a device capable of training a neural network model. For example, the electronic device (100) may be a user terminal or server, such as a smartphone, tablet PC, or smart watch. However, there are no special limitations on the type of the electronic device (100) according to the present disclosure. The training of the neural network model according to the present disclosure may be performed in an on-device form within the electronic device (100), but is not limited thereto.

[0043] The neural network model according to the present disclosure refers to an artificial intelligence model comprising an artificial neural network and can be learned by deep learning. Specifically, the neural network model may include at least one artificial neural network selected from a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), and a generative adversarial network (GAN). However, the neural network model according to the present disclosure is not limited to the examples described above.

[0044] Referring to FIG. 1, the electronic device (100) can determine the execution order of the multiple steps by dividing the learning steps performed through multiple layers of a neural network model into multiple steps including a forward propagation step, a gradient calculation step, and a derivative calculation step (S110).

[0045] Specifically, the training phase of a neural network model can be broadly divided into the feedforward (forward propagation) phase and the backpropagation phase. Here, the feedforward phase refers to the stage where input values ​​are passed from the input layer to the output layer to obtain output values, while the backpropagation phase refers to the stage where gradients are passed from the output layer to the input layer to update the weights of each layer.

[0046] The backpropagation process may include a gradient calculation step and a derivative calculation step. The gradient calculation step refers to the step of calculating gradients to use for updating the weights of each layer included in the neural network model, and the derivative calculation step refers to the step of calculating the derivative of the activation function of each layer.

[0047] FIG. 2 is a diagram showing the execution order of a plurality of steps according to an embodiment of the present disclosure. Referring to FIG. 2, the learning of a neural network model may include a model interpretation step (S210) for interpreting a plurality of layers included in the neural network model, a realization step (S220) for realizing a plurality of layers, an execution order determination step (S230) for determining the execution order between a plurality of layers, a model initialization step (S240) for assigning tensors to a plurality of layers, and a learning execution step (S250) for performing learning based on the initialized model.

[0048] At the bottom of FIG. 2, each of the multiple layer-specific learning steps performed during the learning execution step (S250) is shown in order. Here, FC stands for fully connected, FW for forward, BN for batch normalization, AC for activation, CG for compute gradient, and CD for compute derivative. Numbers such as 1, 2, 3, 11, 12, 13, 24, 25, and 26 indicate the execution order; for the sake of convenience, only the execution order of some steps is shown. In particular, at the bottom of FIG. 2, the backpropagation step corresponding to one of the forward propagation steps is shown by dividing it into the gradient calculation step and the derivative calculation step. For example, the steps corresponding to execution order 11 and 12 represent the gradient calculation step and the derivative calculation step corresponding to a single activation.

[0049] As described above with reference to FIG. 2, in the present disclosure, the learning step performed through a plurality of layers of a neural network model is not simply divided into a forward propagation step and a backpropagation step, but can be divided into a gradient calculation step and a derivative calculation step by further subdividing the backpropagation step. Furthermore, the execution order can be determined for each subdivided step. Hereinafter, "a plurality of steps" refers to steps included in the entire learning step of the neural network model as illustrated at the bottom of FIG. 2, and "execution order" refers to the execution order assigned to each of the plurality of steps.

[0050] Meanwhile, the types of layers and execution order shown in FIG. 2 are exemplary, and layers other than those shown in FIG. 2 may be included in the neural network model, and it is obvious that the backpropagation steps may be divided more finely than shown in FIG. 2 and the execution order may be assigned.

[0051] When the execution order of multiple steps is determined, the electronic device (100) can obtain first information regarding which of the multiple steps according to the execution order a plurality of tensors used in multiple layers is used in based on the determined execution order (S120).

[0052] In the present disclosure, the term "tensor" is used to collectively refer to input / output data, weights, gradients, derivatives, etc., used in a neural network model. In particular, a tensor may be distinguished into a specification portion, which includes information about dimensions, information about the execution order according to the present disclosure, information about the type of step in which a plurality of tensors are used, and mode information, and a data portion, which refers to data assigned to the specification of the tensor. The embodiments described below include a process of defining the specification of a tensor and a process of assigning data to the specification of a tensor.

[0053] The first information may be determined based on information regarding the type of step in which a plurality of tensors are used among a plurality of steps. Here, the type of step in which a plurality of tensors are used may include, as illustrated in FIG. 3, a type representing each of a forward step (forward, F), a gradient calculation step (compute gradient, CG) and a derivative calculation step (compute derivative, CD), a backpropagation step (backward, B) including the gradient calculation step and the derivative calculation step, a step including the forward step and the backpropagation step (iteration, I), and an overall learning step of the neural network model (Max, M).

[0054] Step S120 will be described in detail below with reference to FIG. 4. That is, FIG. 4 is a diagram for explaining in detail the process of obtaining first information regarding which of the multiple steps a plurality of tensors is used in according to a determined execution order. For the convenience of explanation, FIG. 4 is described under the premise that the neural network model includes only three layers.

[0055] L in Fig. 4 n Silver layer, X n is the input / output tensor, D n is the derivative, ΔW n is the gradient, W n represents weights, that is, the upper right figure (410) of FIG. 4 shows three layers L0 included in the neural network model, This shows the input / output tensors, derivatives, gradients, and weights corresponding to L1 and L2, respectively.

[0056] Meanwhile, the bottom right figure (420) of FIG. 4 shows the steps performed through each layer and their execution order. Specifically, according to the example of FIG. 4, the learning steps of the neural network model can be performed in the order of the forward propagation step through layer L0 (execution order 0), the forward propagation step through layer L1 (execution order 1), the forward propagation step through layer L2 (execution order 2), the gradient calculation step through layer L2 (execution order 3), the derivative calculation step through layer L2 (execution order 4), the gradient calculation step through layer L1 (execution order 5), the derivative calculation step through layer L1 (execution order 6), and the gradient calculation step through layer L0 (execution order 7). In FIG. 4, a dotted circle is indicated at execution order 8, which indicates that the derivative calculation step through layer L0 is not necessary.

[0057] The left figure (430) of FIG. 4 illustrates a method for obtaining first information regarding which of the multiple steps a plurality of tensors is used in according to the determined execution order when the execution order of multiple steps is determined as in the rear bottom figure (420) of FIG. 4. In the left figure (430) of FIG. 4, the numbers written in the row corresponding to each tensor indicate the execution order, and the information in parentheses indicates information about the type of step in which the plurality of tensors are used (before the slash in parentheses) and mode information corresponding to the tensors (after the slash in parentheses). Here, the mode information corresponding to the tensors will be described later with reference to FIG. 5 and FIG. 6.

[0058] For example, the electronic device (100) can obtain first information indicating that the tensor X0 is used in the steps corresponding to execution order 0 and execution order 7, respectively, based on the fact that the tensor X0 is used in the forward propagation step performed in layer L0 and the gradient calculation step performed in layer L0. That is, the fact that 0 and 7 are written in the row corresponding to the tensor X0 in figure (430) of FIG. 4 indicates that the tensor X0 is used in the steps corresponding to execution order 0 and execution order 7, respectively. The steps in which the tensor X1, tensor X2, and tensor X3 are used can also be determined in the same way as the method for determining the steps in which the tensor X0 is used.

[0059] Additionally, the electronic device (100) may obtain first information indicating that the tensor D3 is used in the step corresponding to execution sequence 2 and execution sequence 3, respectively, based on the fact that the tensor D3 is used in the backpropagation step performed in layer L2. The step in which the tensor D2 and tensor D1 are used may also be determined in the same way as the method for determining the step in which the tensor D3 is used.

[0060] And, the electronic device (100) can obtain first information indicating that the tensor ΔW2 is used in the step corresponding to execution sequence 3 and execution sequence 4, respectively, based on the fact that the tensor ΔW2 is used in the backpropagation step performed in layer L2. The step in which the tensor ΔW1 and tensor ΔW0 are used can also be determined in the same way as the method for determining the step in which the tensor ΔW2 is used.

[0061] Additionally, the electronic device (100) may obtain first information indicating that the tensor W0 is used in the steps corresponding to execution order 0 and execution order 7, respectively, based on the fact that the tensor W0 must be maintained throughout the entire training phase of the neural network model performed in layer L0. The steps in which the tensor W1 and tensor W2 are used may also be determined in the same way as the method for determining the steps in which the tensor W0 is used.

[0062] When the first information is obtained, the electronic device (100) can integrate the execution order determined based on the first information and the second information regarding whether tensors used in adjacent layers among a plurality of layers can be shared (S130).

[0063] Here, the second information may include a plurality of mode information such as a first mode information (place-holder, P) indicating that the tensor is already created, as shown in FIG. 5, a second mode information (create, C) indicating that the tensor needs to be newly created, a third mode information (modify view, MV) indicating that the tensor's data is modified but can be shared with other tensors in adjacent layers, a fourth mode information (read-only view, RV) indicating that the tensor's data is not modified and can be shared with other tensors, and a fifth mode information (extend, E) indicating that the tensor can be shared with all tensors.

[0064] Specifically, the first mode information and the second mode information indicate that the tensor cannot be shared with other tensors, whereas the third mode information, the fourth mode information, and the fifth mode information indicate that the tensor can be shared with other tensors. Which of the first mode information to the fifth mode information corresponds to the mode information corresponding to a specific tensor may be determined by the electronic device (100) or set by a developer or user.

[0065] Step S130 will be described in detail below with reference to FIG. 6. That is, FIG. 6 is a diagram illustrating the process of integrating the execution order determined based on the first information and the second information. In the description of FIG. 6, as with FIG. 4, for the sake of convenience of explanation, the explanation is based on the premise that the neural network model includes three layers.

[0066] As in the case of Fig. 4, L in Fig. 6 n Silver layer, X n is the input / output tensor, D n is the derivative, ΔW n is the gradient, W n represents the weight. That is, the upper right figure (610) of FIG. 6 shows the three layers L0 included in the neural network model, This represents the input / output tensors, derivatives, gradients, and weights corresponding to L1 and L2, respectively (an explanation of MV and RV will be provided later).

[0067] Meanwhile, the bottom right figure (620) of FIG. 6 shows the steps performed through each layer and the execution order thereof. Specifically, according to the example of FIG. 6, the learning steps of the neural network model can be performed in the order of the forward propagation step through layer L0 (execution order 0), the forward propagation step through layer L1 (execution order 1), the forward propagation step through layer L2 (execution order 2), the derivative calculation step through layer L2 (execution order 4), the derivative calculation step through layer L1 (execution order 6), and the gradient calculation step through layer L0 (execution order 7). In FIG. 4, dotted circles are marked at execution order 3, execution order 5, and execution order 8, which indicates that the gradient calculation step through layer L2, the gradient calculation step through layer L1, and the derivative calculation step through layer L0 are not required.

[0068] The left figure (630) of FIG. 6 illustrates a method for obtaining first information regarding which of the multiple steps a multiple tensor is used in according to the determined execution order, and integrating the determined execution order, when the execution order of multiple steps is determined as in the rear bottom figure (620) of FIG. 6. As with FIG. 4, the number written in the row corresponding to each tensor in the left figure (630) of FIG. 6 indicates the execution order, and the information in parentheses indicates information about the type of step in which the multiple tensor is used (before the slash in parentheses) and mode information corresponding to the tensor (after the slash in parentheses).

[0069] For example, as illustrated in FIG. 6, the electronic device (100) can obtain first information indicating that tensor X0 is used in the step corresponding to execution order 0 and execution order 7, respectively, that tensor X1 is used in the step corresponding to execution order 0 and execution order 1, respectively, and that tensor X2 is used in the step corresponding to execution order 1, execution order 2, and execution order 6, respectively. Description of D3, D2, D1, ΔW0, and W0 is omitted.

[0070] In one embodiment, the electronic device (100) may integrate at least a portion of a determined execution order such that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is equal to or earlier than the execution order of the step in which the second tensor of the layer adjacent to the first tensor is first used.

[0071] Referring to the example in FIG. 6, the electronic device (100) can integrate the execution order so that tensor X1 and tensor X2 are shared if the execution order of the step in which tensor X1 is last used is not later than the execution order of the step in which tensor X2 is first used. Specifically, since the step in which tensor X1 is last used is the step corresponding to execution order 1 and the step in which tensor X2 is first used is the step corresponding to execution order 1, it can be said that there is no need to additionally define tensor X2 and X1 can be used as is. Thus, tensor X1 and X2 are shared and tensor X1 can be determined to be used in the step corresponding to execution order 0, execution order 1, execution order 2, and execution order 6, respectively. In Figure 6, the mode information corresponding to tensor X2 is described as third mode information (modify view, MV), which indicates that the data of tensor X1 is modified, but in light of the execution order, tensor X2 can be shared with another tensor X1 of an adjacent layer.

[0072] In one embodiment, the electronic device (100) can integrate at least a portion of the determined execution order so that the first tensor and the second tensor are shared, even if the execution order of the step in which the first tensor among the plurality of tensors is last used is slower than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is first used.

[0073] Referring to the example in FIG. 6, the electronic device (100) can integrate the execution order so that Tensor X2 and Tensor X3 are shared even if the execution order of the step where Tensor X2 is last used is slower than the execution order of the step where Tensor X3 is first used. Specifically, the step where Tensor X2 is last used is the step corresponding to execution order 6, and the step where Tensor X3 is first used is the step corresponding to execution order 2, but the electronic device (100) can integrate the execution order so that Tensor X2 and Tensor X3 are shared because the mode information corresponding to Tensor X3 is the fourth mode information (read-only view, RV), that is, mode information indicating that Tensor X3 can be integrated with another Tensor X2 of an adjacent layer because the data of Tensor X2 is not changed.

[0074] Although only tensors X1, X2, and X3 have been described above, as shown in FIG. 6, the sharing of tensors and integration of execution order can be achieved between tensors D3, D2, and D3 in the same way as described for tensors X1, X2, and X3.

[0075] Meanwhile, although the integration of tensor sharing and execution order has been described above with reference to the embodiment of FIG. 6, the integration of tensor sharing and execution order may not be performed in cases where such integration cannot be achieved (such as in the embodiment of FIG. 4, where the second information corresponding to all tensors is the first mode information or the second mode information). That is, step S130 according to the present disclosure may not be performed depending on the embodiment. Considering this point, the meaning of "the execution order of a plurality of steps is integrated" below includes not only cases where the integration of tensor sharing and execution order according to step S130 is achieved, but also cases where the integration of tensor sharing and execution order was considered but not performed.

[0076] When the execution sequence of multiple steps is integrated, the electronic device (100) can allocate data to multiple tensors by minimizing the memory area for allocating data to multiple tensors based on the integrated execution sequence (S140).

[0077] Specifically, the electronic device (100) can minimize the memory area by determining whether to create an additional memory area for allocating data corresponding to a plurality of tensors based on an integrated execution order, or to overwrite a previously created memory area.

[0078] Hereinafter, a method for allocating data to a tensor with a minimized memory area will be described in detail with reference to FIGS. 7 and FIG. 8. FIG. 7 illustrates a method for allocating data to a tensor with a minimized memory area when the tensors used in multiple layers and the execution order of multiple steps are finally determined according to the embodiment of FIG. 4, and FIG. 8 illustrates a method for allocating data to a tensor with a minimized memory area when the tensors used in multiple layers and the execution order of multiple steps are finally determined according to the embodiment of FIG. 6.

[0079] Figure 7 (710) sequentially shows tensors used in multiple layers in the case of the embodiment of Figure 4. Figures (720), (730), and (740) show a method for minimizing the memory area during the process of allocating data to tensors used in multiple layers.

[0080] Referring to Figure (720), since tensor W0 is used in the step corresponding to execution order 0 and execution order 7, and tensor W1 is used in the step corresponding to execution order 0 and execution order 7, data corresponding to tensor W1 cannot be overwritten in the memory area corresponding to tensor W0. Therefore, the electronic device (100) can additionally allocate a memory area corresponding to tensor W1. That is, since the validity of tensor W0 must be guaranteed not only in the step corresponding to execution order 0 but also in the step corresponding to execution order 7, data corresponding to tensor W1 cannot be overwritten in the memory area corresponding to tensor W0, and thus an additional memory area corresponding to tensor W1 must be allocated. For the same reason, memory areas corresponding to tensor W2, tensor X0, tensor X1, tensor X2, and tensor X3 are additionally allocated.

[0081] In the above, when considering the memory area for allocating data corresponding to tensor W1, only whether the memory area corresponding to W0 can be overwritten was considered; however, when considering the memory area for allocating data corresponding to tensor W2, not only tensor W1 but also whether the memory area corresponding to tensor W0 can be overwritten with data corresponding to tensor W2 can be considered.

[0082] Referring to Figure (730), since Tensor X3 is used in the step corresponding to execution order 2 and Tensor D3 is used in the steps corresponding to execution order 3 and execution order 4, respectively, data corresponding to Tensor D3 can be overwritten in the memory area corresponding to Tensor X3. Therefore, the electronic device (100) can use the memory area corresponding to Tensor X3 without additionally allocating the memory area corresponding to Tensor D3. In Figure (730), the meaning of Tensor X3 being reused is that the memory area corresponding to Tensor X3 can be used to allocate data to Tensor D3.

[0083] Meanwhile, since tensor D3 is used in the steps corresponding to execution sequence 3 and execution sequence 4, respectively, and tensor ΔW2 is also used in the steps corresponding to execution sequence 3 and execution sequence 4, respectively, an additional memory area corresponding to tensor ΔW2 is allocated.

[0084] Referring to Figure (740), since the tensor X2 is only guaranteed to be valid up to the step corresponding to execution sequence 1 and execution sequence 3, respectively, and the tensor D2 is used at the step corresponding to execution sequence 4 and execution sequence 6, respectively, the data corresponding to the tensor D2 can be overwritten in the memory area corresponding to the tensor X2. Therefore, the electronic device (100) can use the memory area corresponding to the tensor X2 without additionally allocating a memory area corresponding to the tensor D3. In Figure (740), the meaning of tensor X2 being reused is that the memory area corresponding to the tensor X2 can be used to allocate data to the tensor D2.

[0085] Meanwhile, the peak memory consumption of FIG. 7 represents the limit of the memory capacity allowed to allocate data to tensors, which may vary depending on the memory specifications and user / developer settings. In FIG. (730), if additional memory areas corresponding to tensor D3 and tensor ΔW2 are allocated, the peak memory consumption according to the example of FIG. 7 is reached. Accordingly, the electronic device (100) can allocate data based on whether the data corresponding to tensor D3 and the data corresponding to tensor D2 can be overwritten in the memory area corresponding to the previously allocated tensor X3 and the memory area corresponding to the previously allocated tensor X2. However, it is of course possible to use the memory area corresponding to the previously allocated tensor even if the peak memory consumption is not reached during the process of allocating data to the tensors.

[0086] Figure 8 (810) sequentially shows tensors used in multiple layers in the case of the embodiment of Figure 6. Figures (820), (830), and (840) show a method for minimizing the memory area during the process of allocating data to tensors used in multiple layers.

[0087] Referring to Figure (820), since tensor W0 is used in the step corresponding to execution order 0 and execution order 7, respectively, and tensor W1 is used in the step corresponding to execution order 0 and execution order 7, the data corresponding to tensor W1 cannot be overwritten in the memory area corresponding to tensor W0. Therefore, the electronic device (100) can additionally allocate a memory area corresponding to tensor W1. For the same reason, memory areas corresponding to tensor W2, tensor X0, tensor X1, and tensor X3 are additionally allocated. Here, tensor X2 is omitted due to the process of sharing tensors and integrating execution orders as described above with reference to Figure 6.

[0088] Referring to Figure (830), since tensor X3 is used in the step corresponding to execution order 2 and tensor D3 is used in the steps corresponding to execution order 3 and execution order 4, respectively, the data corresponding to tensor D3 can be overwritten in the memory area corresponding to tensor X3. Therefore, the electronic device (100) can use the memory area corresponding to tensor X3 without additionally allocating a memory area corresponding to tensor D3. Meanwhile, tensor ΔW2 is used in the steps corresponding to execution order 3 and execution order 4, respectively, and since there is no area among the already allocated memory areas that can be overwritten by tensor ΔW2, a memory area corresponding to tensor ΔW2 is additionally allocated.

[0089] Referring to Figure (840), tensor D2 is used in the steps corresponding to execution sequence 4 and execution sequence 7, respectively. Since there is no area of ​​memory already allocated that can be overwritten by tensor D2, an additional area of ​​memory corresponding to tensor D2 is allocated.

[0090] Referring to Figure (850), tensor ΔW0 is used in the step corresponding to execution order 7. Since tensors T1, T2, D3, and tensor ΔW2 all need to be guaranteed to be valid only up to the step corresponding to execution order 7, tensor ΔW0 can be overwritten in the memory area corresponding to tensor T1, tensor T2, tensor D3, or tensor ΔW2. Therefore, the electronic device (100) can use the memory area corresponding to tensor T1, tensor T2, tensor D3, or tensor ΔW2 without additionally allocating the memory area corresponding to tensor ΔW2. Accordingly, a fragment exists between the memory area corresponding to tensor ΔW2 and the memory area corresponding to tensor D2.

[0091] After steps S110 to S140 as described above are performed, the electronic device (100) can train a neural network model according to an integrated execution order using a plurality of tensors and data assigned to the plurality of tensors (S150).

[0092] Specifically, when a plurality of tensors and data assigned to the plurality of tensors are defined according to the execution order of a plurality of steps as described above, the electronic device (100) can update the weights of each of the plurality of layers of the neural network model by training the neural network model according to the integrated execution order using the plurality of tensors and the data assigned to the plurality of tensors.

[0093] In particular, when a neural network model according to the present disclosure is trained by a server and then trained again by an electronic device (100) according to the present disclosure, the neural network model may be personalized to suit the user of the electronic device (100) according to the training results as described above.

[0094] According to the embodiment described above with reference to FIGS. 1 to 7, the electronic device (100) can minimize memory usage by efficiently defining a plurality of tensors and data assigned to the plurality of tensors according to the execution order of a plurality of steps. Accordingly, the learning of a neural network model for personalization can be efficiently performed without overhead, particularly on a device.

[0096] FIGS. 9 and FIGS. 10 are drawings for explaining a method for reducing data loading time according to one embodiment of the present disclosure.

[0097] In the present disclosure, the term "loading time" refers to the time required to load data necessary for training a neural network model from data stored in non-volatile memory, such as flash memory or eMMC (embedded multimedia card), into volatile memory, such as RAM (random access memory) or a global buffer included in the processor. However, there are no specific restrictions on which type of storage space the data is loaded into which type of storage space.

[0098] Figure 9 illustrates the information obtained when performing the nth iteration in the training of a neural network model, and Figure 10 illustrates the information obtained when performing the n+1th iteration in the training of a neural network model. The following explanation assumes that the nth iteration in Figure 9 is the first iteration and the n+1th iteration is the second iteration.

[0099] In FIGS. 9 and 10, L1 to L9 represent each of the nine layers included in the neural network model. The term "look a head" refers to index information indicating how many steps to consider in advance in each layer. For example, if the look a head of the first layer is 1, only the computation time required in the first layer and the maximum data loading time are considered during the computation of the first layer, and if the look a head of the second layer is 2, not only the computation time required in the second layer and the maximum data loading time are considered during the computation of the second layer, but also the computation time required in the next layer, the third layer, and the maximum data loading time are considered.

[0100] In Figs. 9 and 10, computation (T C ) represents the computation time for each layer, and max load (T L) represents the maximum data loading time for each layer. And, diff (T C -T L ) represents the value obtained by subtracting the maximum data loading time for each layer from the computation time for each layer. In other words, if diff is negative, it means the maximum data loading time is longer than the computation time of the corresponding layer, so loading must be waited for. Conversely, if diff is positive, it means the maximum data loading time is shorter than the computation time of the corresponding layer, so there is no need to wait for loading, and there is actually time available to allocate for additional data loading.

[0101] Referring to FIG. 9, the electronic device (100) first sets look a head to 1 while performing the first iteration, and then each layer-specific operation time (T C ), maximum loading time of data for each layer (T L ), and the value (diff) obtained by subtracting the maximum loading time of data for each layer from the computation time for each layer can be calculated.

[0102] For example, the electronic device (100) can obtain information that, based only on the computation time and maximum data loading time required in layer L1, the computation time of layer L1 is 1 and the maximum loading time is 3, and thus the difference between the computation time and the maximum loading time is -2. This can be said to be a case where data loading must be waited for a time of 2 when performing the computation of layer L1.

[0103] Additionally, the electronic device (100) can obtain information that the computation time of layer L2 is 5 and the maximum loading time of data is 0 (i.e., when already loaded data is used), and therefore the difference between the computation time and the maximum loading time is 5. This can be said to be a case where 5 hours can be allocated to data loading when performing the computation of layer L2.

[0104] Furthermore, the electronic device (100) can calculate the operation time for each layer, the maximum loading time, and the difference between the operation time and the maximum loading time while performing operations for layers L3 through L9 in the same manner as for layers L1 and L2.

[0105] Referring to FIG. 10, the electronic device (100) can update look a head while performing a second iteration to make the difference between the computation time and the maximum loading time zero or positive. Specifically, if there is a layer with a negative diff value resulting from the first iteration, the overall loading delay time can be reduced by performing pre-loading on a layer with a positive diff value among the layers preceding that layer.

[0106] For example, regarding Layer L1, we will discuss it later, but first, let's start with Layer L2. Since the diff value is positive for Layers L2 through L4, there is no need to perform pre-loading in the preceding layers to reduce the loading latency of Layers L2 through L4. This is also true for Layers L6, L7, and L9.

[0107] However, since the diff value of layer L5 is -1, it is desirable to perform pre-loading in the preceding layer for a time corresponding to 1. Accordingly, the electronic device (100) can adjust the look a head of layer L4 to 2 and increase the maximum loading time by 1 to 3. As a result, since the diff value of layer L4 is still greater than or equal to 0, no loading time delay occurs in layer L4, and since the diff value of L5 is adjusted to 0, no loading time delay occurs in L5 either.

[0108] Meanwhile, since the diff value of layer L8 is -2, it is desirable to perform pre-loading in the preceding layer for a time corresponding to 2. Accordingly, the electronic device (100) can adjust the look a head of layer L7 to 1 and increase the maximum loading time by 1 to 1. As a result, since the diff value of layer L7 is still 0 or greater, no loading time delay occurs in layer L7, but the diff value of L5 decreases from -2 to -1, but a loading time delay still occurs. Accordingly, the electronic device (100) can adjust the look a head of layer L6 to 2 and increase the maximum loading time by 1 to 1. As a result, since the diff value of layer L6 is still 0 or greater, no loading time delay occurs in layer L6, and since the diff value of L8 is adjusted to 0, no loading time delay occurs in L8 either.

[0109] Meanwhile, although the description of Layer L1 was omitted above and the description began with Layer L2, since the learning of the neural network model is performed repeatedly, the electronic device (100) can perform the necessary loading for Layer L1 in advance while performing the computation step of Layer L9 in the same way as described above.

[0110] As the number of iterations increases as described above while training the neural network model, the look a head value converges to an optimized value for each layer, and the diff value for each layer can be adjusted to 0 or a positive value.

[0111] According to the embodiment described above with reference to FIGS. 9 and FIGS. 10, the electronic device (100) balances the computation time and data loading time for each of the multiple layers, thereby significantly reducing the data loading time required across all layers, and as a result, the usage of memory can be minimized.

[0113] FIG. 11 is a drawing for explaining a method for adjusting the number of layers used in the computation step of a neural network model according to one embodiment of the present disclosure.

[0114] N, N+1, and N+2 in Fig. 11 represent the number of iterations of the learning process, and layers indicated by solid lines represent layers used in the computation step, while layers indicated by dotted lines represent layers not used in the computation step. Here, the computation step refers to the forward propagation step.

[0115] Specifically, the layers used in the computation phase calculate gradients to update weights; since they utilize the results from the forward propagation phase to calculate gradients, these values ​​must be loaded into memory until the backpropagation phase is performed. On the other hand, layers not used in the computation phase do not update weights and are utilized only to calculate derivatives during the backpropagation phase; therefore, there is no need to load the results from the forward propagation phase.

[0116] Specifically, in the Nth iteration, the electronic device (100) comprises the odd-numbered layers among the total layers (layer0, layer2, layer4, layer7, layer in FIG. 11). 10Only ) can be used in the computation stage. In this case, the even-numbered layers among all layers (layer1, layer3, layer5, layer9 of FIG. 11) are not used in the computation stage as described above, but are used only in the stage of calculating the derivative in the backpropagation stage, so memory usage can be reduced.

[0117] And, since the weights of all layers must be updated in the training of the neural network model, the electronic device (100) uses only the even-numbered layers (layer1, layer3, layer5, layer9 of FIG. 11) among the total layers in the computation step during the N+1th iteration, and the odd-numbered layers (layer0, layer2, layer4, layer7, layer9 of FIG. 11) among the total layers. 10 ) can be used in the step of calculating the derivative during the backpropagation step.

[0118] Furthermore, in the N+2th iteration, the electronic device (100) includes the odd-numbered layers among the total layers (layer0, layer2, layer4, layer7, layer in FIG. 11). 10 Only ) is used in the calculation step, and the even-numbered layers among all layers (layer1, layer3, layer5, layer9 of FIG. 11) can be used in the backpropagation step to calculate the derivative.

[0119] Meanwhile, although an example has been described above in which odd-numbered layers and even-numbered layers among the entire layer are distinguished and used in the calculation step, it goes without saying that the layers used in the calculation step in each iteration may be selected differently from the example of FIG. 11.

[0120] According to the embodiment described above with reference to FIG. 11, the electronic device (100) can train a neural network model while using memory in an effective manner. Meanwhile, if only some of the multiple layers are used in each iteration, there may be a concern that the accuracy of the neural network model may decrease. However, according to the present disclosure, in the next iteration, layers that were not used in the previous iteration can be included in the computation step. Furthermore, the present disclosure may be applied in cases where the model is trained by a server and then retrained by the electronic device (100) for personalization. Ultimately, according to the embodiment described above, the usage of memory can be minimized within a range that does not significantly reduce the accuracy of the neural network model.

[0122] FIG. 12 is a block diagram showing a simplified configuration of an electronic device (100) according to one embodiment of the present disclosure, and FIG. 13 is a block diagram showing a detailed configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0123] As illustrated in FIG. 12, an electronic device (100) according to one embodiment of the present disclosure includes a memory (110) and a processor (120).

[0124] At least one instruction regarding an electronic device (100) may be stored in the memory (110). Additionally, an operating system (O / S) for operating the electronic device (100) may be stored in the memory (110). Furthermore, various software programs or applications for operating the electronic device (100) may be stored in the memory (110) according to various embodiments of the present disclosure. Additionally, the memory (110) may include a semiconductor memory (110), such as a flash memory (110), or a magnetic storage medium, such as a hard disk.

[0125] Specifically, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110), and the processor (120) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (120), and reading, writing, modifying, deleting, updating, etc. of data by the processor (120) may be performed.

[0126] Meanwhile, in the present disclosure, the term memory (110) may be used to include memory (110), ROM (not shown), RAM (not shown) within the processor (120), or a memory (110) card (not shown) mounted in the electronic device (100) (e.g., micro SD card, memory (110) stick).

[0127] In particular, in various embodiments according to the present disclosure, the memory (110) may store data related to a neural network model, specifically, information regarding various parameters including layers and weights of the neural network model. Additionally, the memory (110) may store a plurality of tensors according to the present disclosure, data assigned to the plurality of tensors, etc. Furthermore, the memory (110) may store information regarding the execution order of a plurality of steps determined according to the present disclosure, first information according to the present disclosure, second information, information regarding the type of step in which a plurality of tensors are used, etc.

[0128] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (110), and the information stored in the memory (110) may be updated as it is received from a server or external device or input by a user.

[0129] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to the configuration of the electronic device (100) including a memory (110), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above.

[0130] The processor (120) can be implemented in various ways. For example, the processor (120) can be implemented as at least one of an Application Specific Integrated Circuit (ASIC), an embedded processor (120), a microprocessor (120), hardware control logic, a hardware Finite State Machine (FSM), and a Digital Signal Processor (DSP). Meanwhile, in this disclosure, the term processor (120) may be used to include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a MPU (Main Processing Unit), etc.

[0131] In particular, in various embodiments according to the present disclosure, the processor (120) determines the execution order of the multiple steps by dividing the learning steps performed through multiple layers of a neural network model into multiple steps including a forward propagation step, a gradient calculation step, and a derivative calculation step, and based on the determined execution order, obtains first information regarding which of the multiple steps according to the determined execution order a plurality of tensors used in the multiple layers are used in, integrates the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the multiple layers can be shared, and based on the integrated execution order, minimizes the area of ​​memory (110) for allocating data corresponding to the multiple tensors and allocates data to the multiple tensors, and can train the neural network model according to the integrated execution order using the multiple tensors and the data allocated to the multiple tensors. Here, since the first information and the second information have been described above, a redundant description of the same content is omitted.

[0132] In one embodiment, the processor (120) may integrate at least a portion of the execution order determined so that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is equal to or earlier than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is first used.

[0133] In one embodiment, the processor (120) may integrate at least a portion of the execution order determined so that the first tensor and the second tensor are shared, if the execution order of the step in which the first tensor among the plurality of tensors is last used is slower than the execution order of the step in which the second tensor of the layer adjacent to the layer of the first tensor is first used, and the second information corresponding to the second tensor is fourth mode information.

[0134] In one embodiment, the processor (120) can minimize the area of ​​memory (110) by determining whether to create an additional area of ​​memory (110) for allocating data corresponding to a plurality of tensors based on an integrated execution order, or to overwrite the previously created area of ​​memory (110).

[0135] In addition, various embodiments as described above with reference to FIGS. 1 to 11 can be similarly applied to the control process of the processor (120), so detailed redundant descriptions of the same content are omitted.

[0136] Meanwhile, as illustrated in FIG. 13, the processor (120) may further include a tensor management module (121) and a data allocation module (122).

[0137] As described above, a tensor according to the present disclosure can be distinguished into a specification portion including information about dimensions, information about the execution order according to the present disclosure, information about the type of step in which a plurality of tensors are used, and mode information, and a data portion representing data assigned to the specification of the tensor. Furthermore, the embodiment described with reference to FIGS. 1 to 8 includes a process of defining the specification of the tensor and a process of assigning data to the specification of the tensor.

[0138] Also, the tensor management module (121) refers to a module that controls the process of defining the specifications of a tensor and may be named as a tensor pool. Specifically, the tensor management module (121) can perform operations according to steps S110, S120, and S130 of FIG. 1.

[0139] The data allocation module (122) refers to a module that controls the process of allocating data to the specifications of a tensor, and may be named as a memory pool (memory planner) (110). Specifically, the data allocation module (122) can perform the operation according to step S140 of FIG. 1.

[0140] In the above description, the tensor management module (121) and the data allocation module (122) have been described as examples of modules included in the processor (120), but it is obvious that other modules corresponding to various operations according to the present disclosure may also be implemented in the form of hardware modules or software modules.

[0141] Meanwhile, the functions related to the neural network model described above can be performed through the memory (110) and the processor (120).

[0142] The processor (120) may be composed of one or more processors (120). In this case, the one or more processors (120) may be a general-purpose processor (120) such as a CPU, AP, etc., a graphics-dedicated processor (120) such as a GPU, VPU, etc., or an artificial intelligence-dedicated processor (120) such as an NPU.

[0143] One or more processors (120) control input data to be processed according to predefined operation rules or artificial intelligence models stored in non-volatile memory (110) and volatile memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0144] Here, being created through learning means that a predefined behavioral rule or artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server / system.

[0145] An artificial intelligence model may be composed of multiple neural network layers. Each layer has multiple weight values ​​and performs operations on the layer through the operation of the multiple weights and the operation of the previous layer. Examples of neural networks include CNN (Convolutional Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), GAN (Generative Adversarial Networks), and Deep Q-Networks; however, the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0146] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0147] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0148] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium such as the memory (110) of the manufacturer's server, the application store's server, or the relay server.

[0149] Each component (e.g., module or program) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration.

[0150] Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0151] Meanwhile, the terms “part” or “module” as used in this disclosure include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A “part” or “module” may be a component formed integrally, or a minimum unit or part thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0152] Various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0153] When the above instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0154] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure. Explanation of the symbols

[0156] 100: Electronic device 110: Memory 120: Processor

Claims

Claim 1 An electronic device comprising: a memory for storing data related to a neural network model; and a processor for determining the execution order of the plurality of steps by dividing a learning step performed through a plurality of layers of the neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step, and based on the determined execution order, obtaining first information regarding which of the plurality of steps a plurality of tensors used in the plurality of layers is used in according to the determined execution order, integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared, and based on the integrated execution order, minimizing the area of ​​the memory for allocating data corresponding to the plurality of tensors and allocating the data to the plurality of tensors, and training the neural network model according to the integrated execution order using the plurality of tensors and the data allocated to the plurality of tensors. Claim 2 An electronic device according to claim 1, wherein the first information is determined based on information regarding the type of step in which the plurality of tensors are used among the plurality of steps. Claim 3 my 1 An electronic device comprising, in the term, a type of step in which the plurality of tensors are used, the type representing each of the forward propagation step, the gradient calculation step, the derivative calculation step, the backpropagation step including the gradient calculation step and the derivative calculation step, the step including the forward propagation step and the backpropagation step, and the overall learning step of the neural network model. Claim 4 An electronic device according to claim 1, wherein the second information comprises: first mode information indicating that the tensor is in a pre-assigned state; second mode information indicating that the tensor must be newly created; third mode information indicating that the tensor's data is changed but the tensor can be shared with other tensors in adjacent layers; fourth mode information indicating that the tensor's data is not changed and therefore the tensor can be shared with other tensors; and fifth mode information indicating that the tensor can be shared with all tensors. Claim 5 In claim 4, the processor is an electronic device that integrates at least a portion of the determined execution order such that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is equal to or earlier than the execution order of the step in which the second tensor of a layer adjacent to the layer of the first tensor is first used. Claim 6 In claim 5, the processor is an electronic device that integrates at least a portion of the determined execution order such that the first tensor and the second tensor are shared, provided that the execution order of the step in which the first tensor among the plurality of tensors is last used is slower than the execution order of the step in which the second tensor of a layer adjacent to the layer of the first tensor is first used. Claim 7 An electronic device according to claim 1, wherein the processor minimizes the memory area by determining, based on the integrated execution order, whether to additionally create a memory area for allocating data corresponding to the plurality of tensors or to overwrite a previously created memory area. Claim 8 A method for controlling an electronic device, comprising: a step of determining the execution order of a plurality of steps by dividing a learning step performed through a plurality of layers of a neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step; a step of obtaining first information regarding which of the plurality of steps according to the determined execution order a plurality of tensors used in the plurality of layers is used in based on the determined execution order; a step of integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared; a step of assigning data to the plurality of tensors by minimizing the memory area of ​​the electronic device for assigning data corresponding to the plurality of tensors based on the integrated execution order; and a step of training the neural network model according to the integrated execution order using the plurality of tensors and the data assigned to the plurality of tensors. Claim 9 A method for controlling an electronic device according to claim 8, wherein the first information is determined based on information regarding the type of step in which the plurality of tensors are used among the plurality of steps. Claim 10 A method for controlling an electronic device according to claim 8, wherein the type of step in which the plurality of tensors are used comprises the forward propagation step, the gradient calculation step, the derivative calculation step, the backpropagation step including the gradient calculation step and the derivative calculation step, the step including the forward propagation step and the backpropagation step, and the type representing each of the overall learning step of the neural network model. Claim 11 A method for controlling an electronic device according to claim 8, wherein the second information comprises: first mode information indicating that the tensor is in a pre-assigned state; second mode information indicating that the tensor must be newly created; third mode information indicating that the tensor's data is changed but the tensor can be shared with other tensors in adjacent layers; fourth mode information indicating that the tensor's data is not changed and therefore the tensor can be shared with other tensors; and fifth mode information indicating that the tensor can be shared with all tensors. Claim 12 A method for controlling an electronic device according to claim 11, wherein the step of integrating the determined execution order comprises integrating at least a portion of the determined execution order such that the first tensor and the second tensor are shared if the execution order of the step in which the first tensor among the plurality of tensors is last used is equal to or earlier than the execution order of the step in which the second tensor of a layer adjacent to the layer of the first tensor is first used. Claim 13 A control method for an electronic device according to claim 12, wherein the step of integrating the determined execution order comprises integrating at least a portion of the determined execution order such that the first tensor and the second tensor are shared, if the execution order of the step in which the first tensor among the plurality of tensors is last used is slower than the execution order of the step in which the second tensor of a layer adjacent to the layer of the first tensor is first used, and the second information corresponding to the second tensor is the fourth mode information. Claim 14 A method for controlling an electronic device according to claim 8, wherein the step of allocating data to the plurality of tensors comprises the step of minimizing the memory area by determining whether to additionally create a memory area for allocating data corresponding to the plurality of tensors or to overwrite a previously created memory area based on the integrated execution order. Claim 15 A non-transient computer-readable recording medium comprising a program for executing a method of controlling an electronic device, wherein the method of controlling the electronic device comprises: a step of determining the execution order of the plurality of steps by dividing a learning step performed through a plurality of layers of a neural network model into a plurality of steps including a forward propagation step, a gradient calculation step, and a derivative calculation step; a step of obtaining first information regarding which of the plurality of steps according to the determined execution order a plurality of tensors used in the plurality of layers is used in based on the determined execution order; a step of integrating the determined execution order based on the first information and second information regarding whether tensors used in adjacent layers among the plurality of layers can be shared; a step of assigning the data to the plurality of tensors by minimizing the memory area of ​​the electronic device for assigning data corresponding to the plurality of tensors based on the integrated execution order; and a step of training the neural network model according to the integrated execution order using the plurality of tensors and the data assigned to the plurality of tensors.

Images