Electronic device and control method therefor

Abstract

This electronic device comprises: a memory including one or more storage media configured to store instructions and to store information on a neural network model and information on a plurality of resources for performing distributed learning of the neural network model; and a processor communicatively coupled to the memory and performing a parallelization process including pipeline parallelization, data parallelization, and tensor parallelization on the basis of the information on the neural network model and the information on the plurality of resources. When the instructions are executed by the processor, the electronic device may: when a change in the plurality of resources is detected while performing the distributed learning according to a result of performing the parallelization process, obtain a first computation amount for the case of performing the distributed learning from a time point when the change is detected to a next checkpoint using the plurality of resources before the change; perform the parallelization process again on the basis of information on the changed plurality of resources; obtain a second computation amount for the case of performing the distributed learning from the time point when the change is detected to the next checkpoint using the changed plurality of resources according to a result of the re-performed parallelization process; and perform the distributed learning in a method corresponding to the smaller computation amount between the first computation amount and the second computation amount.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device. More specifically, the present disclosure relates to an electronic device capable of performing a parallel process using multiple resources and performing distributed learning on a neural network model, and a method for controlling the same.

[0002] Recently, technologies related to artificial intelligence have been developing rapidly, and accordingly, technologies that efficiently utilize resources (e.g., GPUs (Graphics Processing Units)) used to train neural network models are attracting attention.

[0003] In particular, various parallelization methods have recently been utilized to distribute training of neural network models using multiple resources. However, limitations have been pointed out in conventional techniques, such as their inability to efficiently perform distributed training in diverse resource environments.

[0004] For example, it has been pointed out that it is difficult to perform efficient distributed learning using conventional techniques in a diverse resource environment, such as when multiple resources do not include the same GPU but include heterogeneous GPUs with different performances, or when the number of multiple resources changes while performing distributed learning using multiple resources.

[0005] The above content is presented as background technology to help understand the present invention, and should not be considered as making any judgment or assertion as to whether the above content corresponds to prior art in relation to the present invention.

[0006] The present disclosure addresses the problems and / or shortcomings of the prior art described above and provides at least the advantages described below. The purpose of the present disclosure is to provide an electronic device capable of efficiently parallelizing and performing distributed learning on a neural network model in response to various resource environments, and a control method thereof.

[0007] Additional aspects are set forth in some of the descriptions below, and some will become apparent from the description or may be understood through practice of the embodiments. In accordance with one or more embodiments of the present disclosure, there is provided an electronic device as described above. An electronic device comprises a memory including one or more storage media configured to store instructions, store information about a neural network model, and information about a plurality of resources for performing distributed learning for the neural network model, and a processor communicatively coupled to the memory, and performing a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on the information about the neural network model and the information about the plurality of resources, wherein when the instruction is executed by the processor, if a change to the plurality of resources is detected while performing the distributed learning according to a result of performing the parallelization process, the electronic device obtains a first amount of computation in the case of performing the distributed learning from the time when the change is detected to the next checkpoint using the plurality of resources before the change, and performs the parallelization process again based on the information about the changed plurality of resources, and obtains a second amount of computation in the case of performing the distributed learning from the time when the change is detected to the next checkpoint using the changed plurality of resources according to a result of the parallelization process performed again, and The distributed learning is performed in a method corresponding to a smaller amount of computation between the first computation amount and the second computation amount.

[0008] Meanwhile, when the instruction is executed by the processor, additionally, the electronic device may calculate a third amount of computation when performing the distributed learning from a checkpoint before the time at which the change is detected to the next checkpoint using the changed plurality of resources according to the result of the parallelization process performed again, and perform the distributed learning in a method corresponding to the smallest amount of computation among the sum of the fourth amount of computation from the previous checkpoint to the time at which the change is detected and the first amount of computation, the sum of the fourth amount of computation and the second amount of computation, and the third amount of computation.

[0009] Meanwhile, when the instruction is executed by the processor, the electronic device additionally performs the pipeline parallelization to identify a plurality of combinations for allocating the plurality of resources to a plurality of stages for dividing layers included in the neural network model, and determines at least one resource for performing the data parallelization and at least one resource for performing the tensor parallelization among the plurality of resources so that the ratio of the data parallelization is maximized, thereby determining a candidate parallelization method for each of the plurality of combinations, and based on the execution time of the distributed learning according to each of the candidate parallelization methods identified for each of the plurality of combinations, identifies an optimal parallelization method among the candidate parallelization methods as a result of performing the parallelization process, and performs distributed learning for the neural network model based on the optimal parallelization method.

[0010] Meanwhile, when the instruction is executed by the processor, additionally, the electronic device may identify whether there is a resource among the plurality of resources whose memory usage is exceeded when performing the distributed learning according to the first parallelization method having the maximum ratio of the data parallelism, and if it is identified that there is no resource whose memory usage is exceeded, the first parallelization method may be determined as the candidate parallelization method.

[0011] Meanwhile, when the instruction is executed by the processor, additionally, if the electronic device identifies that there is a resource whose memory usage is exceeded, the electronic device determines a second parallelization method in which the first parallelization method is changed by reallocating the layers to the plurality of resources so that the memory usage is not exceeded, and determines the second parallelization method as the candidate parallelization method.

[0012] Meanwhile, when the instruction is executed by the processor, additionally, if the electronic device identifies that there is a resource whose memory usage is exceeded, the electronic device may determine a third parallelization method having a ratio of the data parallelism that is next higher than the first parallelization method, and determine the third parallelization method as the candidate parallelization method.

[0013] Meanwhile, when the instruction is executed by the processor, the electronic device may additionally determine the candidate parallelization method among the second parallelization method and the third parallelization method based on the execution time of the distributed learning according to each of the second parallelization method and the third parallelization method.

[0014] Meanwhile, when the instruction is executed by the processor, additionally, if there is a stage among the plurality of stages that includes two or more resources having different performances, the electronic device can allocate the plurality of resources to the plurality of stages based on the performance of the two or more resources.

[0015] Meanwhile, the information about the plurality of resources may include information about the processing performance of each of the plurality of resources, the bandwidth between the plurality of resources, and the bandwidth between the plurality of stages.

[0016] Meanwhile, when the above instruction is executed by the processor, the electronic device can additionally calculate the execution time by performing the distributed learning for each of the candidate parallelization methods for a preset period of time.

[0017] According to one or more embodiments of the present disclosure to achieve the above-described object, a control method of an electronic device is provided. The control method of the electronic device includes: a step of performing a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on information about a neural network model and information about a plurality of resources for performing distributed learning for the neural network model; a step of obtaining a first amount of computation in the case of performing the distributed learning from a time point at which the change is detected to a next checkpoint using the plurality of resources before the change, when a change to the plurality of resources is detected while performing the distributed learning according to a result of performing the parallelization process; a step of performing the parallelization process again based on information about the changed plurality of resources; a step of obtaining a second amount of computation in the case of performing the distributed learning from a time point at which the change is detected to the next checkpoint using the changed plurality of resources, according to a result of the re-performed parallelization process; and a step of performing the distributed learning in a method corresponding to a smaller amount of computation among the first amount of computation and the second amount of computation.

[0018] Meanwhile, the control method of the electronic device may further include a step of calculating a third amount of computation when performing the distributed learning from a checkpoint before the time at which the change is detected to the next checkpoint using the changed plurality of resources according to the result of the parallelization process performed again, and a step of performing the distributed learning in a method corresponding to the smallest amount of computation among a sum of a fourth amount of computation from the previous checkpoint to the time at which the change is detected and the first amount of computation, a sum of the fourth amount of computation and the second amount of computation, and the third amount of computation.

[0019] Meanwhile, the control method of the electronic device may include a step of identifying a plurality of combinations for allocating the plurality of resources to a plurality of stages for dividing layers included in the neural network model by performing the pipeline parallelization, a step of determining at least one resource for performing the data parallelization and at least one resource for performing the tensor parallelization among the plurality of resources so that the ratio of the data parallelization is maximized, thereby determining a candidate parallelization method for each of the plurality of combinations, a step of identifying an optimal parallelization method among the candidate parallelization methods as a result of performing the parallelization process based on an execution time of the distributed learning according to each of the candidate parallelization methods identified for each of the plurality of combinations, and a step of performing distributed learning for the neural network model based on the optimal parallelization method.

[0020] Meanwhile, the step of determining each candidate parallelization method for each of the plurality of combinations may include a step of identifying whether there is a resource among the plurality of resources whose memory usage is exceeded when performing the distributed learning according to the first parallelization method having the maximum ratio of the data parallelism, and a step of determining the first parallelization method as the candidate parallelization method if it is determined that there is no resource whose memory usage is exceeded.

[0021] Meanwhile, the step of determining each candidate parallelization method for each of the plurality of combinations may further include a step of determining a second parallelization method in which the first parallelization method is changed by reallocating the layers to the plurality of resources so that the memory usage is not exceeded, if it is identified that there is a resource whose memory usage is exceeded, and a step of determining the second parallelization method as the candidate parallelization method.

[0022] Meanwhile, the step of determining each candidate parallelization method for each of the plurality of combinations may further include the step of determining a third parallelization method having a data parallelization ratio second to that of the first parallelization method, if it is identified that there is a resource whose memory usage is exceeded, and the step of determining the third parallelization method as the candidate parallelization method.

[0023] Meanwhile, the step of determining each candidate parallelization method for each of the plurality of combinations may further include the step of determining the candidate parallelization method among the second parallelization method and the third parallelization method based on the execution time of the distributed learning according to each of the second parallelization method and the third parallelization method.

[0024] Meanwhile, the step of identifying the plurality of combinations may allocate the plurality of resources to the plurality of stages based on the performance of the two or more resources, if there is a stage among the plurality of stages that includes two or more resources with different performances.

[0025] Meanwhile, the information about the plurality of resources may include information about the processing performance of each of the plurality of resources, the bandwidth between the plurality of resources, and the bandwidth between the plurality of stages.

[0026] Meanwhile, the step of identifying the optimal parallelization method may include a step of calculating the execution time by performing the distributed learning for each of the candidate parallelization methods for a preset period of time.

[0027] According to one or more embodiments of the present disclosure for achieving the above-described object, a non-transitory computer-readable recording medium is provided, having recorded thereon one or more computer programs including computer-executable instructions that, when executed alone or together with one or more processors of an electronic device, cause the electronic device to perform one or more operations. The above operation includes: an operation of performing a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on information about a neural network model and information about a plurality of resources for performing distributed learning for the neural network model; an operation of obtaining a first amount of computation in the case of performing the distributed learning from a time point at which the change is detected to a next checkpoint using the plurality of resources before the change when a change is detected while performing the distributed learning according to a result of performing the parallelization process; an operation of performing the parallelization process again based on information about the changed plurality of resources; an operation of obtaining a second amount of computation in the case of performing the distributed learning from a time point at which the change is detected to the next checkpoint using the changed plurality of resources according to a result of the re-performed parallelization process; and an operation of performing the distributed learning in a method corresponding to a smaller amount of computation among the first amount of computation and the second amount of computation.

[0028] Other aspects, advantages and key features of the present disclosure will become apparent to those skilled in the art by describing in detail various embodiments of the present invention with reference to the accompanying drawings.

[0029] Other aspects, features and advantages of one or more embodiments according to the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0030] FIG. 1 is a block diagram illustrating an electronic device and a plurality of resources according to one or more embodiments.

[0031] FIG. 2 is a block diagram schematically illustrating a configuration of an electronic device according to one or more embodiments.

[0032] FIG. 3 is a diagram illustrating one or more embodiments related to identifying an optimal distributed learning method when changes to multiple resources are detected.

[0033] FIGS. 4 and 5 are diagrams illustrating one or more embodiments related to identifying an optimal parallelization method based on prioritizing data parallelism for each of multiple stages.

[0034] FIG. 6 is a diagram illustrating one or more embodiments related to determining a candidate parallelization method based on whether there is a resource among multiple resources whose memory usage is exceeded.

[0035] Figure 7 is a block diagram detailing the configuration of an electronic device according to one or more embodiments. And,

[0036] FIG. 8 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0037] Identical or similar components, features and structures are depicted with identical reference numerals throughout the drawings.

[0038] The following description, with reference to the accompanying drawings, is provided to provide a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While numerous specific details are included to aid understanding of the present disclosure, these are merely exemplary. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the spirit and scope of the present disclosure. Furthermore, for the sake of clarity and conciseness, descriptions of well-known functions and configurations may be omitted.

[0039] The terms and words used in the following description and claims are not limited to their dictionary meanings, but rather have the meanings defined by the inventors to ensure a clear and consistent understanding of the invention. Accordingly, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is intended solely for illustrative purposes and is not intended to limit the scope of the present disclosure, which is defined by the appended claims and their equivalents.

[0040] Additionally, unless the context clearly dictates otherwise, terms such as "a" and "an" used in the singular should be interpreted to include the plural form. For example, the expression "the surface of a component" should be understood to include one or more such surfaces.

[0041] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0042] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0043] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0044] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0045] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0046] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0047] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0048] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0049] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0050] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0051] In the embodiments of the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that require specific hardware implementation.

[0052] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0053] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0054] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory device, or portions thereof may be distributed and stored across multiple different memory devices.

[0055] Any function or operation described herein may be processed by one or more processors. The one or more processors include circuitry that performs calculations, and may include, for example, an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural network processing unit (NPU, e.g., an artificial intelligence (AI) chipset), a wireless LAN (Wi-Fi) chipset, a Bluetooth™ chipset, a global positioning system (GPS) chipset, a near field communication (NFC) chipset, a connectivity chipset, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chipset, a universal serial bus (USB) controller, a camera controller, an image processing integrated circuit (IC), a microprocessor unit (MPU), a system on a chip (SoC), an integrated circuit (IC), or a similar circuit device.

[0056] FIG. 1 is a block diagram illustrating an electronic device and a plurality of resources according to an embodiment of the present disclosure.

[0057] Referring to FIG. 1, a system according to the present disclosure may include an electronic device (100) and a plurality of resources.

[0058] An "electronic device (100)" refers to a device capable of performing a parallel process using multiple resources and performing distributed learning on a neural network model. For example, the electronic device (100) may be a server, but there are no specific restrictions on the type of electronic device (100).

[0059] A "resource" may include computing resources for performing various functions, such as specific calculations or tasks. For example, a resource may include hardware components such as a Graphics Processing Unit (GPU) or a Neural Processing Unit (NPU).

[0060] In FIG. 1, it is assumed that at least one external device connected to the electronic device (100) includes multiple resources, and the multiple resources are depicted as existing outside the electronic device (100). However, the present disclosure is not limited thereto, and at least some of the multiple resources may be components included in the electronic device (100) (e.g., a GPU included in the electronic device (100).

[0061] There are no specific restrictions on the number and types of multiple resources, and for example, at least one of the multiple resources may have different performance (e.g., computational speed, memory (110) capacity, bandwidth, etc.) than the other resources. In other words, the multiple resources may include heterogeneous resources.

[0062] A 'neural network model' is a model implemented based on the neural network of the human brain, and can refer to the overall model in which artificial neurons that form a network by combining synapses change the binding strength of the synapses through learning to have the ability to solve problems. A neural network model may include multiple layers (or strata), for example, an input layer, an output layer, and multiple hidden layers between them. A neural network model may include an artificial neural network (ANN) model, a deep neural network (DNN) model, etc. However, there are no special restrictions on the type of neural network model.

[0063] 'Distributed learning' is a method of dividing and learning a neural network model or data using multiple resources (or multiple devices including multiple resources, multiple nodes, or multiple machines), and can be performed using various types of parallelization processes.

[0064] "Parallelism" refers to a method of performing distributed training in parallel by dividing a neural network model across multiple resources. The parallelization process can include data parallelism and model parallelism, and model parallelism can include pipeline parallelism and tensor parallelism.

[0065] "Data parallelism" is a technique for processing training data in parallel with a neural network model. In other words, data parallelism trains the neural network model by assigning the entire neural network model to each of multiple resources, dividing the entire training data into batches of a certain size, and assigning them to each resource.

[0066] "Pipeline parallelization" is a technique for dividing a neural network model into layers and processing them in parallel. In other words, pipeline parallelization divides the multiple resources contained in a neural network model into multiple stages, and trains the neural network model by assigning each stage a resource.

[0067] Tensor parallelization is a technique for dividing and parallelizing the layers within a neural network model. In other words, tensor parallelization is a technique for training a neural network model by dividing multiple layers within the model and allocating them to multiple resources. Unlike pipeline parallelization, which divides each layer individually, tensor parallelization divides a single layer and allocates it to multiple resources. This may require a sync operation that combines the outputs of intermediate or final layers.

[0068] In the above, the electronic device (100) and the plurality of resources according to the present disclosure have been described along with the key terms related to the present disclosure. Hereinafter, various embodiments according to the present disclosure will be described with reference to FIGS. 2 to 8.

[0069] FIG. 2 is a block diagram schematically illustrating the configuration of an electronic device according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating one or more embodiments related to identifying an optimal distributed learning method when changes to multiple resources are detected.

[0070] Referring to FIG. 2, an electronic device (100) according to an embodiment of the present disclosure may include a memory (110) and a processor (120).

[0071] At least one instruction regarding the electronic device (100) may be stored in the memory (110). In addition, an O / S (Operating System) for driving the electronic device (100) may be stored in the memory (110). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110). In addition, the memory (110) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0072] Specifically, the memory (110) may store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (120) may execute various software modules stored in the memory (110) to control the operation of the electronic device (100). For example, the memory (110) is accessed by the processor (120), and reading / writing / modifying / deleting / updating, etc. of data in the memory (110) may be performed by the processor (120).

[0073] Meanwhile, in the present disclosure, the term memory (110) may be used to mean a memory (110), a ROM, a RAM in a processor (120), or a memory card (e.g., a micro SD card, a memory stick) mounted in an electronic device (100).

[0074] In one or more embodiments of the present disclosure, the memory (110) may store information about a neural network model and information about a plurality of resources for performing distributed learning on the neural network model. Here, the "information about the neural network model" may include information about layers included in the neural network model, information about parameters including weights, and the like. In addition, the "information about the plurality of resources" may include information about the processing performance of each of the plurality of resources, bandwidth between the plurality of resources, and bandwidth between the plurality of stages.

[0075] In addition, various information such as information on the result of performing the parallelization process according to the present disclosure, information on the first amount of calculation, information on the second amount of calculation, information on the third amount of calculation, information on multiple combinations, information on candidate parallelization methods, information on the amount of memory (110) used according to the candidate parallelization method, etc. may be stored in the memory (110).

[0076] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (110), and the information stored in the memory (110) may be updated as received from an external device or input by a user.

[0077] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to the configuration of the electronic device (100) including the memory (110), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above.

[0078] The processor (120) may be implemented in various ways. For example, the processor (120) may be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP). Meanwhile, the term "processor (120)" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).

[0079] In one or more embodiments of the present disclosure, the processor (120) may perform a parallelization process including pipeline parallelization, data parallelization, and tensor parallelization based on information about a neural network model and information about a plurality of resources.

[0080] Specifically, the processor (120) can allocate data processing and calculations performed by the neural network model to multiple resources by performing a parallelization process so that the multiple resources can perform at their maximum performance while not exceeding the memory usage of the multiple resources.

[0081] A specific description of the parallelization process will be provided later, and below, an example will be described first in which changes occur to multiple resources while distributed learning is performed after the parallelization process is performed.

[0082] The processor (120) can detect changes to multiple resources while performing distributed learning based on the results of performing the parallelization process.

[0083] Specifically, the processor (120) may detect a change in the number of resources based on at least one of the occurrence of an error during distributed learning, an increase in available resources, and a decrease in available resources. For example, the processor (120) may detect that the number of resources changes to 120 during distributed learning based on the results of performing a parallelization process when the number of resources is 100.

[0084] When a change to multiple resources is detected, the processor (120) can obtain a first computational amount for performing distributed learning from the time the change to the multiple resources is detected until the next checkpoint using the multiple resources before the change. For example, even if a change to the multiple resources is detected, the processor (120) can estimate the computational amount for continuing distributed learning based on the results of a parallelization process performed based on information about the multiple resources before the change.

[0085] Referring to CASE 1 of FIG. 3, the processor (120) can obtain the first computational amount when performing distributed learning from the time when a change in the multiple resources is detected to the next checkpoint, assuming that the 100 resources before the change are used, even if the number of resources changes from 100 to 120. For example, the processor (120) can obtain the first computational amount for section 310 of FIG. 3.

[0086] Here, a "checkpoint" refers to the point in time when intermediate computation results are saved during distributed training of a neural network model. This can be set to occur at preset intervals. The checkpoint can be adjusted based on developer or user settings. For example, a checkpoint can be every four hours, and its duration can vary depending on factors such as the size of the neural network model and the amount of data processed by the model.

[0087] The term "computational load" refers to the total amount of computation required during distributed learning. Depending on the outcome of the parallelization process, the computational load can be categorized into first, second, and third computational loads. Since computational load can be converted into "time required" or "computational speed," the term "computational load" can be replaced with "time required" or "computational speed."

[0088] The processor (120) can re-perform the parallelization process based on information about the changed multiple resources. Specifically, when a change to the multiple resources is detected, the processor (120) can allocate data processing and calculations performed by the neural network model to the multiple resources so that the changed multiple resources can achieve maximum performance while not exceeding the memory usage of the multiple resources.

[0089] The processor (120) can obtain a second computational load when performing distributed learning from the time a change to multiple resources is detected until the next checkpoint, based on the results of the parallelization process performed again. For example, when a change to multiple resources is detected, the processor (120) can estimate the computational load when continuing distributed learning based on the results of the parallelization process performed based on information about the multiple resources after the change.

[0090] Referring to CASE 2 of FIG. 3, when the number of resources changes from 100 to 120, the processor (120) can obtain a second amount of computation when performing distributed learning from the time when a change in the multiple resources is detected to the next checkpoint, assuming that the 120 resources after the multiple resources are changed are used, based on the results of the parallelization process performed again. For example, the processor (120) can obtain a second amount of computation for section 320 of FIG. 3.

[0091] When the first and second computational amounts are obtained, the processor (120) can perform distributed learning in a method corresponding to the smaller computational amount among the first and second computational amounts.

[0092] Meanwhile, in one or more embodiments of the present disclosure, the processor (120) may perform distributed learning in a method corresponding to the smallest amount of computation by additionally considering other distributed learning methods as well as the distributed learning methods corresponding to the first and second amounts of computation, respectively.

[0093] Specifically, the processor (120) can calculate the third amount of computation when performing distributed learning from a checkpoint before a change to multiple resources is detected to the next checkpoint, using the changed multiple resources, based on the results of the parallelization process performed again.

[0094] For example, when a change to multiple resources is detected, the processor (120) can estimate the amount of computation in the case of performing distributed learning based on the results of a parallelization process performed based on information about the multiple resources after the change by returning to a checkpoint before the point in time when the multiple resources were changed, rather than the point in time when the multiple resources were changed.

[0095] Referring to CASE 3 of FIG. 3, when the number of resources changes from 100 to 120, the processor (120) can obtain a third amount of computation when performing distributed learning from a checkpoint prior to the point at which the number of resources changes to the next checkpoint, assuming that the 120 resources after the number of resources changes are used, based on the results of the parallelization process performed again. For example, the processor (120) can obtain a third amount of computation for section 330 of FIG. 3.

[0096] When the first, second and third computational amounts are obtained, the processor (120) can perform distributed learning using a parallelization method corresponding to the smallest computational amount among the sum of the fourth computational amount and the first computational amount, the sum of the fourth and second computational amounts and the third computational amount from the previous checkpoint to the point at which changes to multiple resources are detected.

[0097] Specifically, the first and second computational quantities relate to the computational quantities from the point in time when multiple resources are changed to the next checkpoint, and the periods for which the computational quantities are calculated are the same. However, the third computational quantity relates to the computational quantity from the previous checkpoint to the next checkpoint, and the periods for which the computational quantities are calculated are different from the first and second computational quantities. Therefore, the processor (120) can align the periods for which the computational quantities are calculated to the period from the previous checkpoint to the next checkpoint and then compare the respective computational quantities.

[0098] In the example of FIG. 3, the processor (120) compares the sum of the fourth operation amount and the first operation amount, the sum of the fourth operation amount and the second operation amount, and the fourth operation amount for section 340 of FIG. 3, to identify a method corresponding to the smallest operation amount, and perform distributed learning using the identified parallelization method.

[0099] The overhead of searching for a new optimal parallelization process and operating multiple resources by changing the new parallelization process may be included in the second computational load, and the overhead of returning to a previous checkpoint and performing distributed learning again may be included in the third computational load.

[0100] Therefore, even if changes to multiple resources are detected, the first computational load may be the lowest when distributed learning continues based on the results of the parallelization process performed before the change. Conversely, even if the overhead described above is included in the second and third computational loads, if the reduction in time required for performing the new parallelization process is greater than the overhead, the second or third computational load may be the lowest.

[0101] According to the embodiments described above with reference to FIGS. 1 to 3, when a change to multiple resources is detected while performing distributed learning for a neural network model, the electronic device (100) can perform distributed learning by determining an optimal learning method among continuing distributed learning using multiple resources before the change and performing distributed learning in various ways using multiple changed resources.

[0102] Meanwhile, the process of identifying the optimal parallelization method by performing the parallelization process before and after detecting changes to multiple resources has not been specifically described above. Below, various embodiments of the process of identifying the optimal parallelization method will be described.

[0103] FIGS. 4 and 5 are diagrams illustrating one or more embodiments related to identifying an optimal parallelization method based on prioritizing data parallelism for each of multiple stages according to various embodiments of the present disclosure.

[0104] Figures 4 and 5 illustrate the cases where the number of multiple resources is 6 and the number of pipeline parallelisms is 1 and 2. Hereinafter, pipeline parallelism is abbreviated as PP, data parallelism as DP, and tensor parallelism as TP.

[0105] In one or more embodiments of the present disclosure, the processor (120) may perform pipeline parallelization to identify multiple combinations of assigning multiple resources to multiple stages that partition layers included in a neural network model.

[0106] Specifically, the processor (120) can identify all combinations of dividing the plurality of resources into a plurality of stages based on information about the plurality of resources and assigning each of the plurality of stages to the resources.

[0107] Referring to FIG. 4, the processor (120) can treat six resources as one stage (PP=1), and in this case, six resources can be allocated to one stage, so only one combination is possible.

[0108] Referring to FIG. 5, the processor (120) can divide six resources into two stages (PP=2), and in this case, various combinations are possible, but FIG. 5 illustrates a combination in which two resources are allocated to the first stage and four resources are allocated to the second stage.

[0109] The processor (120) can compare the processing performance of each of the multiple resources and the computational load of each layer included in the neural network model, and allocate the layers to the multiple resources. In addition, the processor (120) can identify multiple combinations for allocating multiple resources to the multiple stages based on information about the processing performance of each of the multiple resources, as well as the bandwidth between the multiple resources and the bandwidth between the multiple stages.

[0110] For example, the processor (120) may assign 10 layers out of 30 layers included in the neural network model to the first stage and 20 layers to the second stage based on the fact that the processing performance of the second stage is twice that of the first stage.

[0111] Meanwhile, if there is a stage among multiple stages that includes two or more resources with different performances, the processor (120) can allocate multiple resources to the multiple stages based on the performance of the two or more resources.

[0112] For example, if resources A and B included in the first stage of FIG. 5 are different types of GPUs from resources C, D, E, and F included in the second stage, the processor (120) may allocate multiple resources to the first stage and the second stage based on the processing performance of each type of GPU.

[0113] When multiple combinations are identified, the processor (120) can determine candidate parallelization methods for each of the multiple combinations by determining at least one resource that performs data parallelization and at least one resource that performs tensor parallelization among the multiple resources so as to maximize the ratio of data parallelization.

[0114] Referring to Figure 4, when 6 resources are treated as 1 stage (PP=1), a parallelization method in which DP is 6 and TP is 1, a parallelization method in which DP is 3 and TP is 2, a parallelization method in which DP is 2 and TP is 3, and a parallelization method in which DP is 1 and TP is 6 are possible.

[0115] Determining the parallelization method to maximize the data parallelism ratio could mean prioritizing the parallelization method with a maximum DP of 6 and a TP of 1 among the above parallelization methods. This is because while a higher DP ratio increases memory usage, it also improves processing performance.

[0116] Specifically, the processor (120) preferentially selects a parallelization method with the highest DP value among all possible DP and TP combinations, and when performing distributed learning according to the selected parallelization method, if there is no resource among multiple resources whose memory usage is exceeded, the selected parallelization method can be determined as a candidate parallelization method of the corresponding combination.

[0117] For example, the processor (120) may determine a parallelization method in which PP is 1, DP is 6, and TP is 1 among all cases described in FIG. 4 as a candidate parallelization method for the corresponding combination. In addition, the processor (120) may determine a parallelization method in which PP is 2, DP of the first stage is 2, and TP is 1, and DP of the second stage is 4 and TP is 1 among all cases described in FIG. 4 as a candidate parallelization method for the corresponding combination.

[0118] The processor (120) can identify an optimal parallelization method among the candidate parallelization methods as a result of performing the parallelization process based on the execution time of distributed learning for each of the candidate parallelization methods identified for each of the plurality of combinations. Furthermore, the processor (120) can perform distributed learning for a neural network model based on the optimal parallelization method.

[0119] Specifically, the candidate parallelization method with the shortest execution time for distributed learning for each of the candidate parallelization methods identified for each combination is identified as the optimal parallelization method, and distributed learning for a neural network model can be performed using the optimal parallelization method.

[0120] The processor (120) can calculate the execution time of distributed learning for each candidate parallelization method based on information about layers included in the neural network model, information about multiple resources, etc. In addition, the processor (120) can also calculate the execution time by performing distributed learning for each candidate parallelization method for a preset period of time. That is, the processor (120) can calculate the execution time of distributed learning for each candidate parallelization method using information stored in the memory (110), and can also calculate the execution time by actually directly performing distributed learning for each candidate parallelization method for a predetermined period of time.

[0121] For example, in the example of FIG. 4, if a parallelization method in which PP is 1, DP is 6, and TP is 1 is determined as a first candidate parallelization method, and in the example of FIG. 5, if a parallelization method in which PP is 2, DP of the first stage is 2, TP is 1, and DP of the second stage is 4 and TP is 1 is determined as a second candidate parallelization method, the processor (120) can identify a candidate parallelization method with a shorter execution time among the first candidate parallelization method and the second candidate parallelization method as an optimal parallelization method and perform distributed learning.

[0122] The process of determining a parallelization method to maximize the ratio of data parallelism is briefly described above, and this will be described in more detail below with reference to Figure 6.

[0123] FIG. 6 is a diagram illustrating one or more embodiments related to determining a candidate parallelization method based on whether there is a resource among a plurality of resources whose memory usage is exceeded according to various embodiments of the present disclosure.

[0124] The processor (120) can identify the first parallelization method with the maximum data parallelism ratio in operation S610. Hereinafter, in the example of FIG. 5, a case in which PP is 2, DP of the first stage is 2 and TP is 1, and DP of the second stage is 4 and TP is 1, in which case the parallelization method is the first parallelization method, will be described.

[0125] When performing distributed learning according to the first parallelization method in operation S620, the processor (120) can identify whether any of the multiple resources has excessive memory usage. Specifically, the processor (120) can estimate memory usage based on information about the memory of each of the multiple resources, or monitor memory usage using a sensor included in each of the multiple resources, thereby identifying whether each of the multiple resources is experiencing an Out of Memory (OOM) condition.

[0126] If it is determined that no resource exceeding memory usage exists in operation S620-N, the processor (120) may determine the first parallelization method as a candidate parallelization method in operation S630. For example, if there is no memory shortage for the parallelization method determined to maximize the ratio of data parallelism, the processor (120) may determine that parallelization method as a candidate parallelization method for the corresponding combination.

[0127] If it is identified that there is a resource that exceeds memory usage in operation S620-Y, the processor (120) can determine a second parallelization method that is a modified version of the first parallelization method by reallocating layers to multiple resources so that memory usage is not exceeded in operation S640. In other words, if there is a memory shortage for the parallelization method determined to maximize the ratio of data parallelism, it cannot be said that the optimal parallelization method has been identified in the combination. Therefore, in this case, the processor (120) can determine a different parallelization method by reallocating layers to multiple resources.

[0128] In the above example, if the memory usage of at least one of the resources included in the first stage is exceeded, and the memory usage of the resources included in the second stage is not exceeded, it can be said that it is desirable to allocate some of the layers allocated to the first stage to the second stage.

[0129] In this case, if 10 layers out of 30 layers included in the neural network model are assigned to the first stage and 20 layers are assigned to the second stage, the processor (120) can determine the second parallelization method by assigning 9 layers to the first stage and 21 layers to the second stage.

[0130] If it is identified that there is a resource that exceeds memory usage in operation S620-Y, the processor (120) may determine a third parallelization method that has a higher ratio of data parallelism next to the first parallelization method in operation S650. For example, if there is a resource that exceeds memory usage, the processor (120) may reallocate layers to multiple resources, but may change the parallelization method by maintaining the layers reallocated to multiple layers and reducing the ratio of data parallelism.

[0131] As in the example above, if the memory usage of at least one of the resources included in the first stage is exceeded and the memory usage of the resources included in the second stage is not exceeded, the processor (120) can determine the third parallelization method by changing the parallelization method to a parallelization method in which the DP of the first stage is 1 and the TP is 2.

[0132] The processor (120) may determine a candidate parallelization method among the second and third parallelization methods based on the execution time of distributed learning according to each of the second and third parallelization methods in operation S660. For example, which of the second and third parallelization methods is the optimal parallelization method in the given combination may be determined based on the execution time of distributed learning according to each parallelization method.

[0133] In the above example, among the second parallelization method by reallocating layers to multiple resources and the third parallelization method by reducing the ratio of data parallelism, the parallelization method with a shorter execution time for distributed learning can be determined as the candidate parallelization method of the combination.

[0134] Meanwhile, when performing distributed learning according to the second parallelization method or the third parallelization method, if it is identified that there is a resource that exceeds memory usage, the processor (120) may reallocate layers to multiple resources again as in the above-described embodiment or reduce the ratio of data parallelism, and this process may be repeated until a parallelization method that has the best processing performance and does not suffer from memory shortage is determined.

[0135] According to the embodiments described above with reference to FIGS. 4 and 5, the electronic device (100) can perform distributed learning for a neural network model in an efficient manner by identifying a parallelization method that can maximize the ratio of data parallelism even in various resource environments.

[0136] In particular, when multiple resources include heterogeneous resources, the combinations of pipeline parallelism, data parallelism, and tensor parallelism can be very diverse, and according to the embodiment of the present disclosure described above, the electronic device (100) can identify an optimal and efficient parallelism method among various combinations.

[0137] FIG. 7 is a block diagram showing in detail the configuration of an electronic device according to an embodiment of the present disclosure.

[0138] Referring to FIG. 7, the electronic device (100) may further include a communication unit (130), an input unit (140), and an output unit (150) in addition to a memory (110) and a processor (120). However, it should be understood that, in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIGS. 1 and 7.

[0139] The communication unit (130) includes a circuit and can perform communication with an external device. Specifically, the processor (120) can receive various data or information from an external device connected via the communication unit (130) and can also transmit various data or information to the external device.

[0140] The communication unit (130) may include at least one of a Wi-Fi module, a Bluetooth module, a wireless communication module, an NFC module, and an Ultra-Wide Band (UWB) module. Specifically, the Wi-Fi module and the Bluetooth module may each perform communication in the Wi-Fi or Bluetooth manner. When using a Wi-Fi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.

[0141] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.

[0142] In one or more embodiments of the present disclosure, the processor (120) may receive information about a neural network model, information about multiple resources, information about a parallelization method, and the like through the communication unit (130). In addition, the processor (120) may transmit information about a neural network model on which distributed learning has been performed to an external device.

[0143] The input unit (140) includes a circuit, and the processor (120) can receive a user command to control the operation of the electronic device (100) through the input unit (140). Specifically, the input unit (140) can be configured with components such as a microphone, a camera, and a remote control signal receiving unit. In addition, the input unit (140) can also be implemented in a form included in a display as a touch screen. In particular, the microphone can receive a voice signal and convert the received voice signal into an electrical signal.

[0144] In one or more embodiments of the present disclosure, the processor (120) may receive user input, such as user input to initiate distributed learning, user input to transmit information about a neural network model to an external device, etc.

[0145] The output unit (150) includes a circuit, and the processor (120) can output various functions that the electronic device (100) can perform through the output unit (150). In addition, the output unit (150) can include at least one of a display, a speaker, and an indicator.

[0146] The display can output image data under the control of the processor (120). Specifically, the display can output an image previously stored in the memory (110) under the control of the processor (120). In particular, the display according to one or more embodiments of the present disclosure can also display a user interface stored in the memory (110). The display can be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display can also be implemented as a flexible display, a transparent display, etc. However, the display according to the present disclosure is not limited to a specific type.

[0147] The speaker can output audio data under the control of the processor (120). The indicator can be turned on under the control of the processor (120). Specifically, the indicator can be turned on in various colors under the control of the processor (120). For example, the indicator can be implemented as a light emitting diode (LED), a liquid crystal display panel (LCD), a vacuum fluorescent display (VFD), etc., but is not limited thereto.

[0148] In one or more embodiments of the present disclosure, the processor (120) may control the output unit (150) to output information about the amount of computation when performing distributed learning according to each parallelization method, information about the execution time when performing distributed learning according to each parallelization method, etc.

[0149] FIG. 8 is a flowchart illustrating a control method of an electronic device according to an embodiment of the present disclosure.

[0150] Referring to FIG. 8, the electronic device (100) may perform a parallelization process based on information about the neural network model and information about multiple resources in operation S810. Specifically, by performing the parallelization process, the electronic device (100) may allocate data processing and calculations performed by the neural network model to multiple resources so that the multiple resources can exhibit maximum performance while not exceeding their memory usage.

[0151] The electronic device (100) can detect changes to multiple resources while performing distributed learning in operation S820. Specifically, the electronic device (100) can detect changes to multiple resources based on at least one of the occurrence of an error during distributed learning, an increase in available resources, and a decrease in available resources. For example, the electronic device (100) can detect that the number of resources changes to 120 while performing distributed learning based on the results of performing a parallelization process when the number of resources is 100.

[0152] The electronic device (100) can obtain the first computational amount when performing distributed learning from the time a change is detected to the next checkpoint using multiple resources prior to the change in operation S830. For example, even if a change is detected in multiple resources, the electronic device (100) can estimate the computational amount when continuing distributed learning based on the results of a parallelization process performed based on information about the multiple resources prior to the change.

[0153] The electronic device (100) can re-perform the parallelization process based on information about the multiple resources changed in operation S840. Specifically, when a change to the multiple resources is detected, the electronic device (100) can allocate data processing and calculations performed by the neural network model to the multiple resources so that the changed multiple resources can achieve maximum performance while not exceeding the memory usage of the multiple resources.

[0154] The electronic device (100) can obtain a second computational amount when performing distributed learning from the time a change is detected to the next checkpoint using the changed multiple resources, based on the results of the parallelization process performed again in operation S850. For example, when a change to multiple resources is detected, the electronic device (100) can estimate the computational amount when continuing distributed learning based on the results of the parallelization process performed based on information about the changed multiple resources.

[0155] The electronic device (100) can perform distributed learning in a method corresponding to a smaller amount of computation between the first computation amount and the second computation amount in operation S860.

[0156] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided in a non-transitory computer readable medium.

[0157] Specifically, in a non-transitory computer-readable recording medium including a program for executing a method for controlling an electronic device (100), the method for controlling an electronic device (100) may include: a step of performing a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on information about a neural network model and information about a plurality of resources for performing distributed learning for the neural network model; a step of obtaining a first amount of computation in the case of performing distributed learning from the time when a change is detected to the next checkpoint using the plurality of resources before the change when a change is detected while performing distributed learning according to a result of performing the parallelization process; a step of performing the parallelization process again based on information about the plurality of changed resources; a step of obtaining a second amount of computation in the case of performing distributed learning from the time when a change is detected to the next checkpoint using the plurality of changed resources according to a result of the parallelization process performed again; and a step of performing distributed learning in a method corresponding to a smaller amount of computation among the first amount of computation and the second amount of computation.

[0158] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).

[0159] The artificial intelligence-related function according to the present disclosure is operated through the processor (120) and memory (110) of the electronic device (100).

[0160] The processor (120) may be composed of one or more processors. In this case, the one or more processors (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processor (120) described above.

[0161] The CPU is a general-purpose processor (120) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. The CPU is advantageous in a serial processing method, enabling organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (120) is not limited to the examples described above, except in cases where it is specifically referred to as a CPU.

[0162] A GPU is a processor (120) for large-scale operations such as floating-point operations used in graphics processing, and can perform large-scale operations in parallel by integrating a large number of cores. More specifically, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (120) to supplement the functions of a CPU. The processor (120) for large-scale operations is not limited to the examples described above, except in cases where it is specifically referred to as a GPU.

[0163] An NPU is a processor (120) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). At this time, since the NPU is designed specifically according to the required specifications of the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (120) specialized in artificial intelligence operations, the NPU can be implemented in various forms such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), a Vision Processing Unit (VPU), etc. The artificial intelligence processor (120) is not limited to the above-described examples, except in cases where it is specified as the above-described NPU.

[0164] Additionally, one or more processors (120) may be implemented as a SoC (System on Chip). In this case, the SoC may further include, in addition to one or more processors (120), a memory (110), and a network interface such as a bus for data communication between the processor (120) and the memory (110).

[0165] When a plurality of processors (120) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the processors (120) among the plurality of processors (120). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (120). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (120).

[0166] In addition, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (120). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (120).

[0167] One or more processors (120) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0168] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0169] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0170] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0171] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0172] According to one or more embodiments of the present disclosure, the method according to the various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be at least temporarily stored or temporarily created in a device-readable storage medium, such as a memory (110) of a manufacturer's server, an application store's server, or an intermediary server.

[0173] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.

[0174] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0175] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0176] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions.

[0177] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0178] It will be apparent to those skilled in the art that various embodiments of the present invention may be implemented in the form of hardware, software, or a combination of hardware and software, in accordance with the claims and the description of this specification.

[0179] The software may be stored on a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium may store one or more computer programs (software modules), and the one or more computer programs may include computer-executable instructions that, when executed by one or more processors of the electronic device, cause the electronic device to perform the method of the present invention.

[0180] The software may be stored in a volatile or non-volatile storage means, for example, in the form of a memory such as a storage device such as a Read Only Memory (ROM) (whether erasable or rewritable) or a Random Access Memory (RAM), a memory chip, a device, an integrated circuit, or an optically or magnetically readable medium such as a Compact Disc (CD), a Digital Versatile Disc (DVD), a magnetic disk, a magnetic tape, or the like. The storage device and the storage medium are various embodiments of a non-transitory machine-readable recording medium capable of storing one or more computer programs including commands, which commands, when executed, may be configured to implement various embodiments of the present invention. Accordingly, one embodiment of the present invention provides a program comprising a code for implementing a device or method according to any claim of the present specification, and a non-transitory machine-readable recording medium on which the program is stored.

[0181] While the present invention has been described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

Claims

1. In electronic devices, A memory including one or more storage media configured to store instructions, information about a neural network model, and information about a plurality of resources for performing distributed learning on the neural network model; and A processor that is communicatively coupled to the memory and performs a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on information about the neural network model and information about the plurality of resources; The above instructions, when executed by the processor, cause the electronic device to: When a change to the plurality of resources is detected while performing the distributed learning according to the result of performing the above parallelization process, the first amount of computation is obtained when performing the distributed learning from the time when the change is detected to the next checkpoint using the plurality of resources before the change. The parallelization process is performed again based on information about the above-mentioned changed plurality of resources, According to the result of the parallelization process performed again, the second amount of computation is obtained when the distributed learning is performed from the time the change is detected to the next checkpoint using the changed multiple resources. An electronic device configured to perform the distributed learning in a manner corresponding to a smaller amount of computation between the first amount of computation and the second amount of computation.

2. In paragraph 1, When the above instructions are executed by the processor, the electronic device additionally: According to the result of the parallelization process performed again above, the third amount of computation is calculated when the distributed learning is performed from the checkpoint before the change is detected to the next checkpoint using the changed multiple resources, An electronic device configured to perform the distributed learning in a manner corresponding to the smallest amount of computation among the sum of the fourth computation amount and the first computation amount from the previous checkpoint to the point at which the change is detected, the sum of the fourth computation amount and the second computation amount, and the third computation amount.

3. In paragraph 1, When the above instructions are executed by the processor, the electronic device additionally: By performing the above pipeline parallelization, a plurality of combinations are identified for allocating the plurality of resources to a plurality of stages that divide the layers included in the neural network model, In order to maximize the ratio of the data parallelism, candidate parallelism methods are determined for each of the plurality of combinations by determining at least one resource performing the data parallelism and at least one resource performing the tensor parallelism among the plurality of resources, Based on the execution time of the distributed learning for each of the candidate parallelization methods identified for each of the above combinations, the optimal parallelization method among the candidate parallelization methods is identified as a result of performing the parallelization process. An electronic device configured to perform distributed learning for the neural network model based on the above optimal parallelization method.

4. In paragraph 3, When the above instructions are executed by the processor, the electronic device additionally: When performing the distributed learning according to the first parallelization method with the maximum ratio of the data parallelism, it is identified whether there is a resource among the plurality of resources whose memory usage is exceeded, An electronic device configured to determine the first parallelization method as the candidate parallelization method if it is identified that no resource exceeding the above memory usage exists.

5. In paragraph 4, When the above instructions are executed by the processor, the electronic device additionally: If it is identified that there is a resource that exceeds the above memory usage, a second parallelization method is determined in which the first parallelization method is changed by reallocating the layers to the plurality of resources so that the above memory usage is not exceeded. An electronic device configured to determine the second parallelization method as the candidate parallelization method.

6. In paragraph 5, When the above instructions are executed by the processor, the electronic device additionally: If it is identified that there is a resource that exceeds the above memory usage, a third parallelization method is determined that has a higher ratio of data parallelism than the first parallelization method. An electronic device configured to determine the third parallelization method as the candidate parallelization method.

7. In paragraph 6, When the above instructions are executed by the processor, the electronic device additionally: An electronic device configured to determine the candidate parallelization method among the second parallelization method and the third parallelization method based on the execution time of the distributed learning according to each of the second parallelization method and the third parallelization method.

8. In paragraph 3, When the above instructions are executed by the processor, the electronic device additionally: An electronic device configured to allocate a plurality of resources to the plurality of stages based on the performance of the two or more resources, if there is a stage among the plurality of stages that includes two or more resources having different performances.

9. In paragraph 3, An electronic device wherein the information about the plurality of resources includes information about the processing performance of each of the plurality of resources, bandwidth between the plurality of resources, and bandwidth between the plurality of stages.

10. In paragraph 3, When the above instructions are executed by the processor, the electronic device additionally: An electronic device configured to calculate the execution time by performing the distributed learning for each of the candidate parallelization methods for a preset period of time.

11. In a method for controlling an electronic device, A step of performing a parallelization process including pipeline parallelism, data parallelism, and tensor parallelism based on information about a neural network model and information about a plurality of resources for performing distributed learning for the neural network model; A step of obtaining a first computational amount when performing the distributed learning from the time when the change is detected to the multiple resources until the next checkpoint using the multiple resources before the change, when a change to the multiple resources is detected while performing the distributed learning according to the result of performing the parallelization process; A step of performing the parallelization process again based on information about the plurality of changed resources; A step of obtaining a second amount of computation when performing the distributed learning from the time when the change is detected to the next checkpoint using the changed plurality of resources according to the result of the parallelization process performed again; and A control method of an electronic device, comprising: a step of performing the distributed learning in a manner corresponding to a smaller amount of computation among the first amount of computation and the second amount of computation; 12. In paragraph 11, The method of controlling the above electronic device is as follows: A step of calculating a third amount of computation when performing the distributed learning from a checkpoint before the change is detected to the next checkpoint using the changed plurality of resources according to the result of the parallelization process performed again; and A control method of an electronic device further comprising: performing the distributed learning in a manner corresponding to the smallest amount of computation among the sum of the fourth computation amount and the first computation amount from the previous checkpoint to the point at which the change is detected, the sum of the fourth computation amount and the second computation amount, and the third computation amount; 13. In paragraph 11, The method of controlling the above electronic device is as follows: A step of identifying a plurality of combinations for allocating the plurality of resources to a plurality of stages for dividing the layers included in the neural network model by performing the pipeline parallelization; A step of determining candidate parallelization methods for each of the plurality of combinations by determining at least one resource performing the data parallelization and at least one resource performing the tensor parallelization among the plurality of resources so that the ratio of the data parallelization is maximized; A step of identifying an optimal parallelization method among the candidate parallelization methods as a result of performing the parallelization process based on the execution time of the distributed learning for each of the candidate parallelization methods identified for each of the plurality of combinations; and A method for controlling an electronic device, comprising: performing distributed learning for the neural network model based on the optimal parallelization method; 14. In paragraph 13, The step of determining each candidate parallelization method for each of the above multiple combinations is: A step of identifying whether there is a resource among the plurality of resources whose memory usage is exceeded when performing the distributed learning according to the first parallelization method with the maximum ratio of the data parallelization; and A control method of an electronic device, comprising: determining the first parallelization method as the candidate parallelization method when it is identified that no resource exceeding the memory usage exists; 15. In paragraph 14, The step of determining each candidate parallelization method for each of the above multiple combinations is: If it is identified that there is a resource whose memory usage is exceeded, determining a second parallelization method in which the first parallelization method is changed by reallocating the layers to the plurality of resources so that the memory usage is not exceeded; and A control method of an electronic device further comprising: a step of determining the second parallelization method as the candidate parallelization method;

Images