Network model training method and device, storage medium and electronic equipment

Abstract

The invention relates to a network model training method and device, a storage medium and electronic equipment, and relates to the technical field of distributed computing, and the method comprises the steps: analyzing a model structure of a to-be-trained neural network model, and obtaining a model hierarchical structure included in the to-be-trained neural network model; according to the current calculation task characteristics of the model hierarchical structure, dividing a current sub-hierarchical calculation task corresponding to the model hierarchical structure to obtain a task calculation context; distributing a task execution node for the task calculation context, and executing the task calculation context based on the task execution node to obtain a current task execution result; and synchronizing and combining the current task execution result to obtain a parameter updating result, and performing parameter updating on the to-be-trained neural network model based on the parameter updating result. According to the invention, the utilization rate of distributed computing resources is improved.

Description

Background Art

[0002] In the process of training related distributed large models, there is an unclear task division, which makes it impossible to fully utilize distributed computing resources, resulting in low utilization of distributed computing resources and low overall training efficiency of the model.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a network model training method, a network model training device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent overcoming the problems of low utilization of distributed computing resources and low overall training efficiency of the model caused by the limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, a method for training a network model is provided, comprising:

[0006] Analyzing the model structure of the neural network model to be trained to obtain the model hierarchical structure included in the neural network model to be trained;

[0007] According to the current computing task characteristics of the model hierarchical structure, the current sub-level computing task corresponding to the model hierarchical structure is divided to obtain a task computing context;

[0008] Allocating a task execution node to the task computing context, and executing the task computing context based on the task execution node to obtain a current task execution result;

[0009] The current task execution results are synchronized and merged to obtain parameter update results, and the parameters of the neural network model to be trained are updated based on the parameter update results.

[0010] In an exemplary embodiment of the present disclosure, according to the current computing task characteristics of the model hierarchical structure, the current sub-level computing task corresponding to the model hierarchical structure is divided to obtain a task computing context, including:

[0011] Determine a first current data computing amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to the model hierarchy structure;

[0012] Determining current computing task characteristics of the model hierarchy structure according to the first current data computing amount, the first current data dependency, and the first current computing resource requirement;

[0013] Determining current hierarchical structure characteristics of the model hierarchical structure according to current sub-hierarchical parameters included in the model hierarchical structure;

[0014] The current sub-level computing task is divided according to the current level structure characteristics and the current computing task characteristics to obtain a task computing context.

[0015] In an exemplary embodiment of the present disclosure, determining a first current data computing amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to a model hierarchy structure includes:

[0016] Acquire a task operation type required to execute a current sub-level computing task corresponding to the model hierarchy structure, and determine a first current data computing amount according to the task operation type;

[0017] Acquire a first data transfer path of a first sub-data block and a first dependency relationship between first sub-data blocks required for executing a current sub-level computing task corresponding to the model hierarchy structure;

[0018] A first current data dependency is determined based on the first data transmission path and the first dependency, and the first current computing resource requirement is determined based on the first current data calculation amount and the first current data dependency.

[0019] In an exemplary embodiment of the present disclosure, the current sub-level computing task is divided according to the current hierarchical structure characteristics and the current computing task characteristics to obtain the task computing context, including:

[0020] Acquire current system resources of a model training system that executes a model training task corresponding to the neural network model to be trained;

[0021] Generate a first task feature of the current sub-level computing task according to the current hierarchical structure feature, the current computing task feature and the current system resources;

[0022] The first task feature is input into the trained task division model to obtain the task calculation context.

[0023] In an exemplary embodiment of the present disclosure, allocating a task execution node to the task computing context includes:

[0024] Determine a second current data calculation amount, a second current data dependency, and a second current computing resource requirement of the task computing context according to a first current data calculation amount, a first current data dependency, and a first current computing resource requirement of the model hierarchy structure;

[0025] Acquire an original computing node included in current system resources, and acquire original node computing resources possessed by the original computing node;

[0026] Matching a current computing node from the original computing nodes according to the second current data computing amount, the second current computing resource requirement, and the original node computing resources;

[0027] A task execution node corresponding to the task computing context is matched from the current computing nodes according to the second current data dependency relationship.

[0028] In an exemplary embodiment of the present disclosure, matching a task execution node corresponding to the task computing context from the current computing node according to the second current data dependency relationship includes:

[0029] Determining, according to the second current data dependency, a second sub-data block required for executing the task computing context from the first sub-data blocks required for executing the current sub-level computing task corresponding to the model hierarchy structure;

[0030] The task execution node is matched from the current computing node according to the second data transfer path and the second data size of the second sub-data block and the node bandwidth resources of the current computing node.

[0031] In an exemplary embodiment of the present disclosure, executing the task computing context based on the task execution node to obtain the current task execution result includes:

[0032] Transmitting the task computing context to the task execution node based on a preset data transmission method; wherein the preset data transmission method includes multi-channel parallel transmission;

[0033] After detecting that the task computing context transmission is completed, the task computing context is executed based on the task execution node to obtain a current task execution result.

[0034] In an exemplary embodiment of the present disclosure, synchronizing and merging the current task execution results to obtain parameter update results includes:

[0035] Determining a target node from the task execution nodes according to the node distances between the task execution nodes;

[0036] Delivering the current task execution result to the target node based on other nodes among the task execution nodes except the target node;

[0037] The current task execution results are merged based on the target node to obtain the parameter update result.

[0038] In an exemplary embodiment of the present disclosure, transmitting the current task execution result to the target node includes:

[0039] Determine a data compression algorithm corresponding to the current task execution result according to the data type of the current task execution result;

[0040] The current task execution result is compressed based on the data compression algorithm, and the compressed current task execution result is transmitted to the target node.

[0041] According to one aspect of the present disclosure, a network model training device is provided, comprising:

[0042] A model structure analysis module, used to analyze the model structure of the neural network model to be trained, and obtain the model hierarchical structure included in the neural network model to be trained;

[0043] A task computing context division module is used to divide the current sub-level computing tasks corresponding to the model hierarchical structure according to the current computing task characteristics of the model hierarchical structure to obtain the task computing context;

[0044] A task computing context execution module, used to assign a task execution node to the task computing context, and execute the task computing context based on the task execution node to obtain a current task execution result;

[0045] The model parameter updating module is used to synchronize and merge the current task execution results to obtain parameter updating results, and update the parameters of the neural network model to be trained based on the parameter updating results.

[0046] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the training method of the network model described in any one of the above is implemented.

[0047] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0048] Processor; and

[0049] A memory, configured to store executable instructions of the processor;

[0050] Wherein, the processor is configured to execute any one of the above-mentioned network model training methods by executing the executable instructions.

[0051] A network model training method provided by an embodiment of the present disclosure, on the one hand, by analyzing the model structure of the neural network model to be trained, the model hierarchical structure included in the neural network model to be trained is obtained; then, according to the current computing task characteristics of the model hierarchical structure, the current sub-level computing tasks corresponding to the model hierarchical structure are divided to obtain a task computing context; then, a task execution node is assigned to the task computing context, and the task computing context is executed based on the task execution node to obtain a current task execution result; finally, the current task execution result is synchronized and merged to obtain a parameter update result, and the parameters of the neural network model to be trained are updated based on the parameter update result. Since the model training task can be divided into multiple task computing contexts, each task computing context can be calculated separately based on multiple task execution nodes, thereby improving the utilization rate of distributed computing resources of the task execution nodes; on the other hand, since each task computing context can be calculated separately based on multiple task execution nodes, thereby improving the overall training efficiency of the model.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0054] Figure 1 A flowchart of a network model training method according to an exemplary embodiment of the present disclosure is schematically shown.

[0055] Figure 2 A diagram schematically shows an example structure of a neural network model to be trained according to an example embodiment of the present disclosure.

[0056] Figure 3 A structural example diagram schematically shows a Bert model, in which a neural network model to be trained according to an example embodiment of the present disclosure is used.

[0057] Figure 4 The flowchart schematically shows a specific division process of a task computing context according to an exemplary embodiment of the present disclosure.

[0058] Figure 5 A flowchart schematically illustrates a process of allocating task execution nodes according to an exemplary embodiment of the present disclosure.

[0059] Figure 6An example diagram schematically illustrates a task computing context and an allocation result of a task execution node corresponding to the task computing context according to an example embodiment of the present disclosure.

[0060] Figure 7 A block diagram schematically illustrates a network model training device according to an exemplary embodiment of the present disclosure.

[0061] Figure 8 An electronic device for implementing a training method for a network model according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0062] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0063] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0064] With the rapid development of deep learning technology, artificial intelligence has made significant progress in natural language processing (NLP), computer vision (CV), and speech recognition. However, with the increase in model complexity and data size, the computing resources required to train deep learning models have also increased significantly; therefore, a single computing node is difficult to meet the training needs of large-scale models, and distributed training has become an effective way to meet this challenge.

[0065] In the relevant technical solutions of distributed training, although the computing tasks are assigned to multiple nodes for parallel execution, the training efficiency is often restricted by the communication overhead and the utilization of computing resources, which makes the training efficiency of the model low; among them, the training efficiency is restricted by the communication overhead and the utilization of computing resources in the following specific aspects: on the one hand, the frequent data exchange between nodes will occupy a large amount of bandwidth and prolong the training time; on the other hand, the uneven distribution of tasks or improper communication strategies may lead to the idleness of computing resources, further reducing the overall efficiency. Therefore, optimizing the utilization of computing resources in a distributed environment and reducing communication overhead have become the key to improving the training speed of large models. Furthermore, in the process of related distributed large model training, there are still the following defects: on the one hand, the task division and parallelism are insufficient: that is, in the traditional large model training, the task division is not detailed, and the distributed computing resources cannot be fully utilized, resulting in low overall training efficiency; on the other hand, the communication overhead is high: that is, the communication between the computing nodes becomes a bottleneck, and the existing methods are insufficient in optimizing data transmission, resulting in high bandwidth occupancy and large delay, affecting system performance; on the other hand, the synchronization mechanism is not flexible: the fixed synchronization strategy lacks flexibility and is difficult to adapt to the dynamic changes of task characteristics during training, resulting in insufficient resource utilization.

[0066] Based on this, this exemplary embodiment first provides a network model training method, which can be run on a terminal device, server, server cluster or cloud server where the model training system is located; of course, those skilled in the art can also run the method disclosed in this disclosure on other platforms as required, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the training method of the network model may include the following steps:

[0067] Step S110. Analyze the model structure of the neural network model to be trained to obtain the model hierarchical structure included in the neural network model to be trained;

[0068] Step S120. According to the current computing task characteristics of the model hierarchy structure, the current sub-level computing task corresponding to the model hierarchy structure is divided to obtain a task computing context;

[0069] Step S130. Allocate a task execution node for the task computing context, and execute the task computing context based on the task execution node to obtain a current task execution result;

[0070] Step S140. Synchronize and merge the current task execution results to obtain parameter update results, and update the parameters of the neural network model to be trained based on the parameter update results.

[0071] In the training method of the above network model, on the one hand, by analyzing the model structure of the neural network model to be trained, the model hierarchical structure included in the neural network model to be trained is obtained; then, according to the current computing task characteristics of the model hierarchical structure, the current sub-level computing tasks corresponding to the model hierarchical structure are divided to obtain the task computing context; then, a task execution node is assigned to the task computing context, and the task computing context is executed based on the task execution node to obtain the current task execution result; finally, the current task execution result is synchronized and merged to obtain the parameter update result, and the parameters of the neural network model to be trained are updated based on the parameter update result. Since the model training task can be divided into multiple task computing contexts, and then each task computing context is respectively calculated based on multiple task execution nodes, the utilization rate of the distributed computing resources of the task execution nodes is improved; on the other hand, since each task computing context can be respectively calculated based on multiple task execution nodes, the overall training efficiency of the model is improved.

[0072] Hereinafter, the training method of the network model described in the exemplary embodiment of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0073] First, the application scenarios of the example embodiments of the present disclosure are explained and illustrated. Specifically, the training method of the network model recorded in the example embodiments of the present disclosure can be used to train network models in natural language processing scenarios, network models in computer vision scenarios, and network models in recommendation systems and advertising delivery scenarios. Among them, the network models in the natural language processing scenarios recorded here may include but are not limited to large language models (GPT, Generative Pre-Trained Transformer), multilingual translation models, and question-answering system models, etc.; the network models in the computer vision scenarios recorded here may include but are not limited to image classification models, target detection models, and video understanding models, etc.; the network models in the recommendation systems and advertising delivery scenarios recorded here may include but are not limited to item recommendation models, personalized advertising delivery models, and user portrait analysis models, etc.

[0074] Secondly, the technical implementation principle of the exemplary embodiment of the present disclosure is explained and illustrated. Specifically, the training method of the network model provided by the exemplary embodiment of the present disclosure takes the context parallel processing mechanism as the core, and the computing tasks of the large model are split according to the context and executed in parallel on different computing nodes, thereby improving the overall training efficiency. At the same time, the network model training method recorded in the exemplary embodiment of the present disclosure also has the following advantages: on the one hand, dynamic task division and parallel processing can be performed: that is, the task division strategy based on the context can be used to dynamically adjust the task allocation, improve the utilization of computing resources, and significantly improve the training efficiency; on the other hand, communication optimization and data compression can be achieved: that is, in the process of actual application, dynamic data compression, optimized communication topology and multi-channel transmission can be used to reduce communication overhead and improve system performance; on the other hand, flexible synchronous communication and asynchronous communication can also be achieved: that is, in the process of actual application, flexible synchronization mechanisms and asynchronous communication strategies can be designed, the synchronization frequency can be dynamically adjusted, the utilization of computing resources can be maximized, and the efficiency of parallel computing can be improved.

[0075] Further, the model hierarchical structure included in the neural network model to be trained involved in the exemplary embodiment of the present disclosure is explained and illustrated. Figure 2 As shown, the model hierarchy structure may include the first level, the second level, the third level, ..., the Lth level, etc. The specific value of L can be determined according to the actual situation of the model, and this example does not impose any special restrictions on this. Figure 3 As shown, assuming that the Bert model includes N layers of Transformer, the specific value of L can be N plus an input layer and an output layer, or it can be directly consistent with the number of Transformer layers. This example does not impose any special restrictions on this.

[0076] The following will be combined Figure 2 as well as Figure 3 right Figure 1 The training method of the network model shown in is further explained and illustrated. Specifically:

[0077] In step S110, the model structure of the neural network model to be trained is analyzed to obtain the model hierarchical structure included in the neural network model to be trained.

[0078] In this example embodiment, first, the neural network model to be trained is determined; secondly, the model structure of the neural network model to be trained is analyzed to identify the various modules or layers in the neural network model to be trained; for example, assuming that the neural network model to be trained (M) contains (L) layers, each layer (l) corresponds to a computing task, which can be represented by (T_l); wherein (T_l) recorded here represents the computing task of the (l)th layer model itself, then: M = T_1, T_2, ..., T_L. Further, taking the neural network model to be trained as the Bert model as an example, the model hierarchy structure obtained after the model structure analysis may include but is not limited to the input layer, Transformer1, Transformer2, ..., TransformerN, output layer, etc.; in the specific structural analysis process, it can be implemented according to the specific modules of the source code of the model, or it can also be implemented according to other methods, and this example does not impose any special restrictions on this.

[0079] In step S120, according to the current computing task characteristics of the model hierarchy structure, the current sub-level computing task corresponding to the model hierarchy structure is divided to obtain a task computing context.

[0080] Specifically, refer to Figure 4 As shown, the specific division process of the task computing context may include the following steps:

[0081] Step S410, determining a first current data calculation amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to the model hierarchy structure.

[0082] Specifically, the specific determination process of the first current data calculation amount, the first current data dependency and the first current computing resource requirement can be achieved in the following way: first, obtain the task operation type required to execute the current sub-level computing task corresponding to the model hierarchy structure, and determine the first current data calculation amount according to the task operation type; second, obtain the first data transmission path of the first sub-data block and the first dependency between the first sub-data blocks required to execute the current sub-level computing task corresponding to the model hierarchy structure; then, determine the first current data dependency according to the first data transmission path and the first dependency, and determine the first current computing resource requirement according to the first current data calculation amount and the first current data dependency. That is, in the process of actual application, the specific process of determining the first current data calculation amount first needs to determine the operations involved in each layer, such as matrix multiplication, convolution, etc., and calculate the number of floating-point operations or other basic operations according to the operation type; for example, in matrix multiplication, the calculation amount can be determined by the dimension of the matrix; the determination of the first current data dependency can be determined by analyzing the data transfer and dependency between the layer and the previous and next layers, for example, the data transmission path in the network can be clarified, and key factors that may affect data dependency, such as shared parameters and jump connections, must be paid attention to; further, the specific process of determining the first current computing resource requirement can be determined based on the first current data calculation amount and the first current data dependency; for example, based on the first current data calculation amount and the first current data dependency, the execution efficiency of the layer on specific hardware (such as GPU and / or TPU) can be inferred, and the memory and bandwidth requirements for data storage and transmission can be considered.

[0083] Step S420, determining the current computing task characteristics of the model hierarchical structure according to the first current data computing amount, the first current data dependency and the first current computing resource requirement.

[0084] Specifically, after obtaining the first current data calculation amount, the first current data dependency relationship, and the first current computing resource requirement of each different model hierarchy structure, the current computing task characteristics of the model hierarchy structure can be determined; for example, for each different model hierarchy structure (T_l), its current computing task characteristics (F_l) can be expressed by the following formula (1):

[0085] F_l = {C_l, D_l, R_l}; Formula (1)

[0086] Among them, C_l is the first current data calculation amount, D_l is the first current data dependency, and R_l is the first current computing resource requirement.

[0087] Step S430: determining the current hierarchical structure characteristics of the model hierarchical structure according to the current sub-hierarchical parameters included in the model hierarchical structure.

[0088] Specifically, the current sub-level parameters recorded here refer to all parameters included in the model hierarchy structure, such as bias parameters, linearization parameters, nonlinearization parameters, etc., which can be determined based on the actual situation of the model, and this example does not impose any special restrictions on this.

[0089] Step S440 , dividing the current sub-level computing task according to the current level structure characteristics and the current computing task characteristics to obtain a task computing context.

[0090] Specifically, the specific division process of the task computing context can be determined in the following manner: first, obtain the current system resources of the model training system that executes the model training task corresponding to the neural network model to be trained; second, generate the first task feature of the current sub-level computing task according to the current hierarchical structure characteristics, the current computing task characteristics and the current system resources; then, input the first task feature into the trained task division model to obtain the task computing context. The task computing context recorded here refers to the division of the model's calculation and / or training tasks into multiple contexts according to the current computing task characteristics recorded above during the training process of the neural network model; each task computing context represents a computing task of a certain function or level in the model, such as a convolutional layer that processes a specific input or a module that calculates attention weights; the task computing context can be expressed by the following formula (2):

[0091] Ctx_i = {T_{l_1},T_{l_2}...T_{l_n}}; Formula (2)

[0092] Furthermore, in the process of dividing the task computing context, the division can be carried out according to the complexity of the model structure and the resource conditions of the computing nodes; in the specific application process, a static or dynamic context division strategy can be adopted; among them, static division is suitable for models with relatively fixed structures, while dynamic division can adjust the context division method in real time according to actual computing needs; at the same time, in order to make the division more accurate, real-time monitoring of resource usage can be used, and learning strategies (such as meta-learning) can be combined to dynamically adjust the context division to adapt to the changing computing environment and task characteristics; among them, for dynamic division, it can be implemented based on the following formula (3):

[0093] Ctx_i(t) = f(M,F,{Resources}); Formula (3)

[0094] Among them, the above formula (3) can be used to represent that the context (Ctx_i) at a certain time point (t) is obtained by mapping the model structure, task characteristics and system resources through function (f) (that is, the task division model recorded above); wherein the (f) function recorded here can be adjusted in real time based on the monitoring results of actual computing resources and the model task characteristics.

[0095] In step S130, a task execution node is allocated to the task computing context, and the task computing context is executed based on the task execution node to obtain a current task execution result.

[0096] In this example embodiment, first, a task execution node is assigned to the task computing context; specifically, refer to Figure 5 As shown, the following steps may be included:

[0097] Step S510, determining the second current data calculation amount, the second current data dependency and the second current computing resource requirement of the task computing context according to the first current data calculation amount, the first current data dependency and the first current computing resource requirement of the model hierarchy structure.

[0098] Step S520, obtaining the original computing nodes included in the current system resources, and obtaining the original node computing resources possessed by the original computing nodes.

[0099] Step S530, matching the current computing node from the original computing nodes according to the second current data computing amount, the second current computing resource requirement and the original node computing resources.

[0100] Step S540: Match a task execution node corresponding to the task computing context from the current computing nodes according to the second current data dependency relationship.

[0101] In an exemplary embodiment, matching the task execution node corresponding to the task computing context from the current computing node according to the second current data dependency relationship can be achieved in the following manner: determining the second sub-data block required to execute the task computing context from the first sub-data block required to execute the current sub-level computing task corresponding to the model hierarchy structure according to the second current data dependency relationship; matching the task execution node from the current computing node according to the second data transmission path, the second data size of the second sub-data block and the node bandwidth resources of the current computing node.

[0102] The specific division process of the task execution nodes will be further explained and illustrated below. Specifically, in the process of allocating task execution nodes to the task computing context, a load balancing strategy or an allocation strategy that minimizes communication overhead can be adopted based on the computing requirements of the context (that is, the second current data computing amount, the second current data dependency, and the second current computing resource requirement of the task computing context) and the computing power of the original computing node (the original node computing resources) to ensure that the computing resources of each node are fully utilized. In addition, more refined task allocation can be performed based on the communication delay and bandwidth differences between nodes to further optimize the context allocation strategy; wherein, the specific division process of the task execution nodes can refer to the following formula (4):

[0103] N_j = Allocate{(Ctx_i)}; Formula (4)

[0104] It should be noted that the reason why the second data transmission path, the second data size and the node bandwidth resources need to be taken into account is to reduce the latency on the basis of improving the utilization rate of the node resources of the current computing node, thereby achieving the purpose of improving the model training efficiency; for example, if the data transmission path is long and the data is large, a node with more node bandwidth resources can be allocated, otherwise, a node with less node bandwidth resources can be allocated; among them, the allocation results of each task computing context and the task execution node corresponding to the task computing context can be referred to Figure 6 shown.

[0105] Secondly, the task computing context is executed based on the task execution node to obtain the current task execution result; specifically, it can be achieved in the following way: based on a preset data transmission method, the task computing context is transmitted to the task execution node; wherein the preset data transmission method includes multi-channel parallel transmission; after detecting that the task computing context transmission is completed, the task computing context is executed based on the task execution node to obtain the current task execution result. That is, in the actual application process, the task computing context can be allocated to the corresponding task execution node based on multi-channel parallel transmission; each task execution node starts parallel processing immediately after receiving the task computing context; wherein, the parallel execution of the task computing context can effectively improve the parallelism of model training and reduce the overall training time; in the actual application process, assuming that the time required for the task execution node (N_j) to execute the context (Ctx_i) is (T_{N_j,Ctx_i}), then the time required for the overall training process of the model is: T_total = max{j}{T_{N_j,Ctx_i}}; it should also be noted here that in parallel processing, each node can perform local optimization independently, such as using tensor cores (TensorCore) to accelerate matrix operations or reduce memory access delays through cache mechanisms.

[0106] In step S140, the current task execution results are synchronized and merged to obtain parameter update results, and the parameters of the neural network model to be trained are updated based on the parameter update results.

[0107] In this example embodiment, first, the current task execution results are synchronized and merged; specifically, this can be achieved in the following way: first, based on the node distance between the task execution nodes, the target node is determined from the task execution nodes; secondly, based on the other nodes in the task execution nodes except the target node, the current task execution results are transmitted to the target node; then, based on the target node, the current task execution results are merged to obtain the parameter update result. That is, after completing the calculation of the local task calculation context, each task execution node synchronizes the results; the synchronization method can adopt global synchronization or hierarchical synchronization, depending on the dependency between the contexts; of course, in the process of synchronizing the task execution results, it is necessary to further consider the impact of the synchronization frequency on the performance, and under what circumstances to choose different synchronization mechanisms, such as the application scenario of asynchronous stochastic gradient descent (SGD); further, for model parameters that need to be updated globally, each node merges the results after synchronization to form the final model update; and, in the merging process, methods such as weighted averaging or gradient aggregation can also be used to ensure the accuracy and consistency of the model update; of course, for parameters that do not need to be updated immediately, a delayed merging strategy can be adopted to reduce the communication overhead during the synchronization process.

[0108] In an example embodiment, passing the current task execution result to the target node can be achieved in the following manner: according to the data type of the current task execution result, determining the data compression algorithm corresponding to the current task execution result; compressing the current task execution result based on the data compression algorithm, and passing the compressed current task execution result to the target node. That is, in the training of a large distributed model, communication between task execution nodes often becomes a bottleneck for improving efficiency; for this reason, the example embodiment of the present disclosure also provides a method for optimizing inter-context communication, aiming to reduce communication overhead, improve parallelism and utilization of computing resources; in this scenario, in order to improve the communication efficiency between task execution nodes, the task execution result can be compressed; further, in the compression process, it is first necessary to select a compression algorithm; wherein, the selection of the compression algorithm can follow the following principles:

[0109] On the one hand, according to the type of transmitted data, appropriate data compression algorithms can be selected, such as lossless compression (Huffman coding, LZ77) or lossy compression (quantization, pruning); for model parameters and gradient information, lossy compression can be used to further reduce the amount of data; on the other hand, for different data types, optimized encoding methods (such as RLE encoding, variable length coding) can be used to reduce the bandwidth occupied by data transmission; of course, for sparse matrices, sparse matrix encoding methods can be used to improve transmission efficiency; on the other hand, during the transmission process, the compression level is dynamically adjusted to balance data accuracy and transmission efficiency; for example, in the early stage of training, a higher compression rate is allowed to accelerate training, and in the later stage of training, the compression rate is gradually reduced to improve model accuracy. It is possible to consider adaptively selecting compression and encoding schemes based on the dynamic characteristics of the transmitted data (such as rate of change, sparsity).

[0110] In an exemplary embodiment, in the process of synchronizing the task execution results, an efficient communication topology, such as a tree topology or a ring topology, can also be designed according to the network structure of the computing nodes to minimize communication delays and bandwidth occupancy; the topology design should take into account the distance and bandwidth differences between nodes to optimize the communication path; at the same time, different topologies can be applied in different training stages, and the communication topology can be dynamically adjusted when necessary to further optimize performance. Of course, in a supported network environment, multi-channel parallel transmission can also be enabled; among them, based on multi-channel parallel transmission, the time overhead of a single communication task can be significantly reduced by transmitting data of multiple contexts at the same time, further improving communication efficiency; further, large data blocks can be decomposed into small data packets, and data can be transmitted using packet communication; in this way, the transmission time of a single data packet can be reduced, thereby reducing transmission delays and waiting time.

[0111] In an example embodiment, in the process of synchronizing the task execution results, based on the dependencies between contexts, it is also possible to analyze which data does not need to be synchronized immediately; for data that does not need to be synchronized immediately, a delayed synchronization strategy is adopted to achieve the purpose of reducing frequent communication overhead and avoiding waiting time between nodes; of course, an asynchronous communication mechanism can also be introduced so that the task execution node can continue to execute other tasks while waiting for data transmission, such as preprocessing the next batch of data or performing local model optimization; through asynchronous communication, computing resources can be maximized to avoid idle nodes; further, in asynchronous communication, a priority scheduling mechanism can also be introduced; based on this method, a higher priority is assigned to communication tasks on the critical path to ensure the progress of the overall training.

[0112] So far, the training method of the network model recorded in the example embodiment of the present disclosure has been fully realized. Based on the above-mentioned contents, it can be known that the training method of the network model recorded in the example embodiment of the present disclosure, on the one hand, can shorten the model development cycle, reduce computing resources and energy consumption, and realize efficient training of larger-scale and complex models; at the same time, it can also optimize resource utilization, reduce hardware requirements and bandwidth consumption, and reduce data center operating costs; and it can also improve product performance and user experience, support the development of new AI applications, and enhance the market competitiveness of enterprises; on the other hand, the training method of the network model recorded in the example embodiment of the disclosure can be dynamically adjusted according to the complexity of the model and the resource conditions of the computing nodes. Compared with the traditional fixed task division method, this scheme significantly improves the utilization rate of distributed computing resources and increases The task parallelism is strengthened, thereby greatly improving the overall efficiency of large model training; on the other hand, the training method of the network model recorded in the public example embodiment significantly reduces the communication overhead by adopting technologies such as dynamic data compression, optimized communication topology and multi-channel parallel transmission, and introduces flexible synchronous and asynchronous communication mechanisms, which can adapt to dynamic changes in the training process; compared with traditional fixed communication and synchronization strategies, this scheme can more effectively reduce bandwidth occupancy and latency, and improve system performance; further, the training method of the network model recorded in the public example embodiment not only solves the key bottlenecks in the prior art, but also greatly improves the efficiency and flexibility of distributed large model training, and has significant practical value and technical advantages.

[0113] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0114] The exemplary embodiment of the present disclosure also provides a training device for a network model. Specifically, refer to Figure 7 As shown, the training device of the network model may include a model structure analysis module 710, a task computing context division module 720, a task computing context execution module 730 and a model parameter update module 740. Among them:

[0115] The model structure analysis module 710 can be used to analyze the model structure of the neural network model to be trained to obtain the model hierarchical structure included in the neural network model to be trained;

[0116] The task computing context partitioning module 720 may be used to partition the current sub-level computing tasks corresponding to the model hierarchical structure according to the current computing task characteristics of the model hierarchical structure to obtain the task computing context;

[0117] The task computing context execution module 730 may be used to assign a task execution node to the task computing context, and execute the task computing context based on the task execution node to obtain a current task execution result;

[0118] The model parameter updating module 740 can be used to synchronize and merge the current task execution results to obtain parameter update results, and update the parameters of the neural network model to be trained based on the parameter update results.

[0119] In an exemplary embodiment of the present disclosure, according to the current computing task characteristics of the model hierarchy structure, the current sub-level computing task corresponding to the model hierarchy structure is divided to obtain a task computing context, including: determining a first current data computing amount, a first current data dependency, and a first current computing resource requirement required to execute the current sub-level computing task corresponding to the model hierarchy structure; determining the current computing task characteristics of the model hierarchy structure according to the first current data computing amount, the first current data dependency, and the first current computing resource requirement; determining the current hierarchical structure characteristics of the model hierarchy structure according to the current sub-level parameters included in the model hierarchy structure; dividing the current sub-level computing task according to the current hierarchical structure characteristics and the current computing task characteristics to obtain a task computing context.

[0120] In an exemplary embodiment of the present disclosure, determining a first current data calculation amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to a model hierarchy structure includes: obtaining a task operation type required to execute the current sub-level computing task corresponding to the model hierarchy structure, and determining the first current data calculation amount based on the task operation type; obtaining a first data transfer path of a first sub-data block and a first dependency between first sub-data blocks required to execute the current sub-level computing task corresponding to the model hierarchy structure; determining a first current data dependency based on the first data transfer path and the first dependency, and determining the first current computing resource requirement based on the first current data calculation amount and the first current data dependency.

[0121] In an exemplary embodiment of the present disclosure, the current sub-level computing task is divided according to the current hierarchical structure characteristics and the current computing task characteristics to obtain the task computing context, including: obtaining the current system resources of the model training system that executes the model training task corresponding to the neural network model to be trained; generating the first task feature of the current sub-level computing task according to the current hierarchical structure characteristics, the current computing task characteristics and the current system resources; inputting the first task feature into the trained task division model to obtain the task computing context.

[0122] In an exemplary embodiment of the present disclosure, a task execution node is assigned to the task computing context, including: determining a second current data computing amount, a second current data dependency, and a second current computing resource requirement of the task computing context according to the first current data computing amount, the first current data dependency, and the first current computing resource requirement of the model hierarchical structure; obtaining the original computing nodes included in the current system resources, and obtaining the original node computing resources of the original computing nodes; matching the current computing nodes from the original computing nodes according to the second current data computing amount, the second current computing resource requirement, and the original node computing resources; matching the task execution node corresponding to the task computing context from the current computing nodes according to the second current data dependency.

[0123] In an exemplary embodiment of the present disclosure, matching a task execution node corresponding to the task computing context from the current computing node according to a second current data dependency relationship includes: determining, according to the second current data dependency relationship, a second sub-data block required to execute the task computing context from a first sub-data block required to execute the current sub-level computing task corresponding to the model hierarchy; matching the task execution node from the current computing node according to a second data transmission path, a second data size of the second sub-data block and a node bandwidth resource of the current computing node.

[0124] In an exemplary embodiment of the present disclosure, the task computing context is executed based on the task execution node to obtain the current task execution result, including: transmitting the task computing context to the task execution node based on a preset data transmission method; wherein the preset data transmission method includes multi-channel parallel transmission; after detecting that the task computing context transmission is completed, executing the task computing context based on the task execution node to obtain the current task execution result.

[0125] In an exemplary embodiment of the present disclosure, the current task execution results are synchronized and merged to obtain parameter update results, including: determining a target node from the task execution nodes based on the node distance between the task execution nodes; transmitting the current task execution results to the target node based on other nodes in the task execution nodes except the target node; and merging the current task execution results based on the target node to obtain the parameter update results.

[0126] In an exemplary embodiment of the present disclosure, the current task execution result is passed to the target node, including: determining a data compression algorithm corresponding to the current task execution result based on the data type of the current task execution result; compressing the current task execution result based on the data compression algorithm, and passing the compressed current task execution result to the target node.

[0127] The specific details of each module in the above-mentioned network model training device have been described in detail in the corresponding network model training method, so they will not be repeated here.

[0128] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0129] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0130] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0131] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0132] Refer to the following Figure 8The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0133] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0134] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 1 Step S110 shown in: analyzing the model structure of the neural network model to be trained to obtain the model hierarchical structure included in the neural network model to be trained; step S120: dividing the current sub-level computing tasks corresponding to the model hierarchical structure according to the current computing task characteristics of the model hierarchical structure to obtain a task computing context; step S130: allocating a task execution node to the task computing context, and executing the task computing context based on the task execution node to obtain a current task execution result; step S140: synchronizing and merging the current task execution results to obtain a parameter update result, and updating the parameters of the neural network model to be trained based on the parameter update result.

[0135] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0136] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0137] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0138] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0139] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0140] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0141] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0142] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0145] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0146] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0147] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the inventions invented herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not invented by the present disclosure. The specification and examples are to be considered merely exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A network model training method, characterized in that: include: Analyzing the model structure of the neural network model to be trained to obtain the model hierarchical structure included in the neural network model to be trained; According to the current computing task characteristics of the model hierarchical structure, the current sub-level computing task corresponding to the model hierarchical structure is divided to obtain a task computing context; Allocating a task execution node to the task computing context, and executing the task computing context based on the task execution node to obtain a current task execution result; The current task execution results are synchronized and merged to obtain parameter update results, and the parameters of the neural network model to be trained are updated based on the parameter update results.

2. The network model training method according to claim 1, characterized in that: According to the current computing task characteristics of the model hierarchical structure, the current sub-level computing task corresponding to the model hierarchical structure is divided to obtain a task computing context, including: Determine a first current data computing amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to the model hierarchy structure; Determining current computing task characteristics of the model hierarchy structure according to the first current data computing amount, the first current data dependency, and the first current computing resource requirement; Determining current hierarchical structure characteristics of the model hierarchical structure according to current sub-hierarchical parameters included in the model hierarchical structure; The current sub-level computing task is divided according to the current level structure characteristics and the current computing task characteristics to obtain a task computing context.

3. The network model training method according to claim 2, characterized in that: Determining a first current data computing amount, a first current data dependency, and a first current computing resource requirement required to execute a current sub-level computing task corresponding to the model hierarchy structure includes: Acquire a task operation type required to execute a current sub-level computing task corresponding to the model hierarchy structure, and determine a first current data computing amount according to the task operation type; Acquire a first data transfer path of a first sub-data block and a first dependency relationship between first sub-data blocks required for executing a current sub-level computing task corresponding to the model hierarchy structure; A first current data dependency is determined based on the first data transmission path and the first dependency, and the first current computing resource requirement is determined based on the first current data calculation amount and the first current data dependency.

4. The network model training method according to claim 2, characterized in that: The current sub-level computing task is divided according to the current level structure characteristics and the current computing task characteristics to obtain a task computing context, including: Acquire current system resources of a model training system that executes a model training task corresponding to the neural network model to be trained; Generate a first task feature of the current sub-level computing task according to the current hierarchical structure feature, the current computing task feature and the current system resources; The first task feature is input into the trained task division model to obtain the task calculation context.

5. The network model training method according to claim 1, characterized in that: Allocating a task execution node to the task computing context includes: Determine a second current data calculation amount, a second current data dependency, and a second current computing resource requirement of the task computing context according to a first current data calculation amount, a first current data dependency, and a first current computing resource requirement of the model hierarchy structure; Acquire an original computing node included in current system resources, and acquire original node computing resources possessed by the original computing node; Matching a current computing node from the original computing nodes according to the second current data computing amount, the second current computing resource requirement, and the original node computing resources; A task execution node corresponding to the task computing context is matched from the current computing nodes according to the second current data dependency relationship.

6. The network model training method according to claim 5, characterized in that: Matching a task execution node corresponding to the task computing context from the current computing node according to the second current data dependency relationship includes: Determining, according to the second current data dependency, a second sub-data block required for executing the task computing context from the first sub-data blocks required for executing the current sub-level computing task corresponding to the model hierarchy structure; The task execution node is matched from the current computing node according to the second data transfer path and the second data size of the second sub-data block and the node bandwidth resources of the current computing node.

7. The network model training method according to claim 1, characterized in that: Executing the task computing context based on the task execution node to obtain a current task execution result includes: Transmitting the task computing context to the task execution node based on a preset data transmission method; wherein the preset data transmission method includes multi-channel parallel transmission; After detecting that the task computing context transmission is completed, the task computing context is executed based on the task execution node to obtain a current task execution result.

8. The network model training method according to claim 1, characterized in that: The current task execution results are synchronized and merged to obtain parameter update results, including: Determining a target node from the task execution nodes according to the node distances between the task execution nodes; Delivering the current task execution result to the target node based on other nodes among the task execution nodes except the target node; The current task execution results are merged based on the target node to obtain the parameter update result.

9. The network model training method according to claim 8, characterized in that: The step of transmitting the current task execution result to the target node includes: Determine a data compression algorithm corresponding to the current task execution result according to the data type of the current task execution result; The current task execution result is compressed based on the data compression algorithm, and the compressed current task execution result is transmitted to the target node.

10. A network model training device, characterized in that: include: A model structure analysis module, used to analyze the model structure of the neural network model to be trained, and obtain the model hierarchical structure included in the neural network model to be trained; A task computing context division module is used to divide the current sub-level computing tasks corresponding to the model hierarchical structure according to the current computing task characteristics of the model hierarchical structure to obtain the task computing context; A task computing context execution module, used to assign a task execution node to the task computing context, and execute the task computing context based on the task execution node to obtain a current task execution result; The model parameter updating module is used to synchronize and merge the current task execution results to obtain parameter updating results, and update the parameters of the neural network model to be trained based on the parameter updating results.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of the network model described in any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the training method of the network model described in any one of claims 1-9 by executing the executable instructions.

Images