Process creation method, device, electronic device and storage medium
By employing a single-threaded intermediate process to create child processes using split fork and spawn functions, the method addresses speed and deadlock issues in deep learning frameworks, enhancing training efficiency.
Patent Information
- Application Number
- CN202110720406.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-28
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2041-06-28
AI Technical Summary
Existing deep learning frameworks face issues with slow child process creation speed using the spawn method and deadlock issues when using the fork method due to multi-threaded processes.
Implement an intermediate process that is single-threaded to create child processes using a split fork function, ensuring fast creation speed without deadlock, and use a spawn function to create the intermediate process when necessary.
This approach enhances the efficiency of child process creation in deep learning frameworks by ensuring speed and preventing deadlock, thereby improving model training efficiency.
Smart Images
Figure CN113391909B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer application technology, and in particular to a process creation method, device, electronic device and storage medium. Background Art
[0002] A deep learning framework is a framework or platform used to train deep learning models. When using a deep learning framework to train a deep learning model, the main process of the deep learning framework will create a sub-process for data processing in each training cycle. After the sub-process is created, it can pre-process the sample data required for the training process by calling the data reading and pre-processing functions provided by the user, thereby providing the required input data for the model training process.
[0003] There are generally two ways for a main process to create a child process: split fork method and generate spawn method. Among them, the spawn method has the problem of slow creation speed; the fork method is fast to create a child process, but it may cause the child process to deadlock. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a process creation method, device, electronic device, and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a process creation method, including:
[0006] In response to the child process creation event being triggered, the main process detects whether an intermediate process has been created currently; in response to the intermediate process having been created, a process creation instruction is sent to the intermediate process;
[0007] In response to receiving the process creation instruction, the intermediate process creates a child process in a first manner.
[0008] In this way, the child process is created by using the intermediate process. As long as the intermediate process is ensured to be a single-threaded process, the problem of deadlock in the child process can be reduced while ensuring the creation speed.
[0009] In a possible implementation, the intermediate process creates the child process in a first manner in response to receiving the process creation instruction, including: the intermediate process executes a split fork function based on the process parameters carried in the process creation instruction to create the child process.
[0010] In a possible implementation manner, the process creation method further includes:
[0011] In response to the intermediate process not being created, the main process creates the intermediate process in a second manner, and after the intermediate process is successfully created, sends the process creation instruction to the intermediate process.
[0012] In a possible implementation, in response to the main process not creating the intermediate process, creating the intermediate process in a second manner includes: the main process executes a spawn function to create the intermediate process.
[0013] In this way, the spawn method is used to create the intermediate process. Since the spawn method is used to create the intermediate process, the intermediate process will not have the problem of process deadlock caused by the fork method; the fork method is used to create the child process. Since the intermediate process is a single-threaded process, the creation of the child process can ensure the creation speed without causing the deadlock of the child process.
[0014] In a possible implementation manner, the child process creation event is triggered, including at least one of the following:
[0015] A new training cycle for the deep learning model begins;
[0016] The number of batches of preprocessed sample data required for training the deep learning model is less than a preset number threshold.
[0017] In this way, by setting different process creation events, the timing of sub-process creation can be flexibly controlled, increasing the flexibility of the deep learning framework.
[0018] In a possible implementation manner, the process creation method further includes:
[0019] The intermediate process creates a data communication pipe between the main process and the child process.
[0020] In a possible implementation manner, the process creation method further includes:
[0021] In response to the data communication pipeline being successfully created, the main process sends a data processing instruction to the sub-process through the data communication pipeline; the data processing instruction carries a data identifier;
[0022] In response to receiving the data processing instruction sent by the main process, the sub-process loads the target data corresponding to the data identifier based on the data identifier carried in the data processing instruction, and performs preset processing on the target data.
[0023] In a possible implementation manner, the process creation method further includes:
[0024] After the sub-process performs preset processing on the target data and obtains result data, the sub-process transmits the result data to the main process through the data communication pipeline.
[0025] In a possible implementation manner, the process creation method further includes:
[0026] In response to receiving the result data sent by the child process, the main process sends a process closing instruction to the intermediate process;
[0027] In response to receiving the process closing instruction, the intermediate process closes the child process.
[0028] In a possible implementation manner, the process creation method further includes:
[0029] In response to receiving the result data sent by the child process, the main process uses the result data to train the deep learning model in the current training cycle;
[0030] And / or, using the result data to train the deep learning model in future training cycles.
[0031] In this way, the sample data can be pre-processed to ensure that the samples required for training can be obtained in time in each training cycle, thereby improving the efficiency of model training.
[0032] In a second aspect, the embodiment of the present disclosure further provides a process creation device, comprising: a main process module and an intermediate process module; wherein:
[0033] The main process module is used to detect whether the intermediate process module has been created in response to the child process creation event being triggered; in response to the intermediate process module having been created, send a process creation instruction to the intermediate process module;
[0034] The intermediate process module is used to create a sub-process module in a first manner in response to receiving the process creation instruction.
[0035] In an optional embodiment, the intermediate process module, in response to receiving the process creation instruction, creates the child process module in the first manner, and is used to: execute a split fork function based on the process parameters carried in the process creation instruction to create the child process module.
[0036] In an optional implementation, the main process module is further used to: in response to the intermediate process module not being created, create the intermediate process module using a second method, and after the intermediate process module is successfully created, send the process creation instruction to the intermediate process module.
[0037] In an optional implementation, in response to the intermediate process module not being created and the intermediate process module being created in the second manner, the intermediate process module is used to: execute a spawn function to create the intermediate process module.
[0038] In an optional implementation manner, the child process creation event is triggered, including at least one of the following:
[0039] A new training cycle for the deep learning model begins;
[0040] The number of batches of preprocessed sample data required for training the deep learning model is less than a preset number threshold.
[0041] In an optional implementation, the intermediate process module is also used to create a data communication pipeline between the main process module and the sub-process module.
[0042] In an optional implementation manner, the main process module is further configured to send a data processing instruction to the sub-process module through the data communication pipeline in response to the successful creation of the data communication pipeline; the data processing instruction carries a data identifier;
[0043] It also includes: a sub-process module, which is used to load target data corresponding to the data identifier in response to receiving a data processing instruction sent by the main process, based on the data identifier carried in the data processing instruction, and perform preset processing on the target data.
[0044] In an optional implementation, the sub-process module, after performing preset processing on the target data to obtain result data, is also used to transmit the result data to the main process module through the data communication pipeline.
[0045] In an optional implementation manner, the main process module is further configured to send a process closing instruction to the intermediate process module in response to receiving the result data sent by the sub-process module;
[0046] The intermediate process module is further used to close the sub-process module in response to receiving the process closing instruction.
[0047] In an optional implementation, the main process module is further configured to, in response to receiving the result data sent by the sub-process module, use the result data to train the deep learning model in the current training cycle;
[0048] And / or, using the result data to train the deep learning model in future training cycles.
[0049] In a third aspect, an optional implementation of the present disclosure further provides an electronic device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.
[0050] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.
[0051] For a description of the effects of the above-mentioned process creation device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned process creation method, which will not be repeated here.
[0052] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0054] Figure 1 A flowchart of a process creation method provided by an embodiment of the present disclosure is shown;
[0055] Figure 2 A flowchart showing a specific example of a specific method for performing deep learning model training using a process creation method provided in an embodiment of the present disclosure;
[0056] Figure 3 A schematic diagram of a process creation device provided by an embodiment of the present disclosure is shown;
[0057] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0059] Currently in Linux systems, the main process usually uses the fork method and the spawn method when creating a child process. Among them, the spawn method is that the main process runs a series of spawn functions to create a child process; however, creating a child process through the spawn method is a process of creating a process from scratch, and there is a problem of slow creation speed; the fork method is that the main process executes the fork function to create a child process by "copying the current state of the main process and the main process memory", and the creation speed is faster; in this child process creation method, the copy-on-write (COW) technology is used to copy the memory image of the main process to the child process, and the child process will inherit the state and memory data of the main process; however, in practice, if the main process includes multiple threads, after one of the threads executes the fork function to create a child process, the child process can only inherit the state of the thread and the memory data of the main process; if before the child process is created, other threads of the main process obtain the lock object, because the child process will not inherit the state of other threads, and the lock object can only be released by the owner, the lock object will not be released, and then the child process will be deadlocked.
[0060] For example, the main process has three threads, 0, 1, and 2; thread 0 executes fork to create a child process, and the child process inherits the state of thread 0 and the memory images corresponding to threads 0, 1, and 2. If thread 1 acquires a lock object before the child process is created, after the child process is created, since it only inherits the state of thread 0, thread 1 does not exist for the child process. However, the lock object can only be released by thread 1, so the child thread cannot release the lock object acquired by thread 1, which leads to a deadlock in the child thread.
[0061] Based on the above research, the present disclosure provides a process creation method, which creates an intermediate process and uses the intermediate process to create a child process. As long as the intermediate process is a single-threaded process, the deadlock problem of the child process can be reduced while ensuring the creation speed.
[0062] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0064] To facilitate understanding of the present embodiment, a process creation method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the process creation method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the process creation method can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0065] The process creation method provided in the embodiment of the present disclosure can be used not only in the deep learning framework, but also in other applications. The embodiment of the present disclosure takes the deep learning framework as an example to illustrate the process creation method provided in the embodiment of the present disclosure.
[0066] See also Figure 1 FIG. 1 is a flowchart of a process creation method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S102, wherein:
[0067] S101: In response to a child process creation event being triggered, the main process detects whether an intermediate process has been created currently; in response to the intermediate process having been created, a process creation instruction is sent to the intermediate process;
[0068] S102: In response to receiving the process creation instruction, the intermediate process creates the child process in a first manner.
[0069] In the embodiment of the present disclosure, in response to the child process creation event being triggered, the main process will detect whether the intermediate process has been created. If the intermediate process has been created, the process creation instruction will be sent to the intermediate process; the intermediate process will respond to receiving the process creation instruction and create the child process using the first method, thereby creating the intermediate process and using the intermediate process to create the child process. As long as the intermediate process is ensured to be a single-threaded process, the problem of deadlock in the child process can be reduced while ensuring the speed of child process creation.
[0070] In the disclosed embodiment, the main process is the main part of the deep learning framework; each module and process during the training of the deep learning model is controlled and promoted by the main process, including the creation and termination of the training process, data reading and preprocessing, the execution process of the operator in the deep learning model, the execution of the operator on different devices, the synchronization between devices during distributed training, etc.; the engine thread is the thread used for the decomposition and distribution of the operator computing tasks in the main process, which runs in the background and is invisible to the user, and will always exist during the entire training process. The main process also includes other threads for implementing other functions in the model training process.
[0071] The intermediate process is designed for the creation of child processes and is responsible for the creation and destruction of child processes. For example, at the beginning of a new round of training cycle, when the main process needs to create a child process, it will send a process creation instruction to the intermediate process; after receiving the process creation instruction sent by the main process, the intermediate process creates the child process. When the child process needs to be closed, the main process can send a process closing instruction to the intermediate process; after receiving the process closing instruction sent by the main process, the intermediate process closes the child process.
[0072] Subprocesses include, for example: processes for data reading and preprocessing, or processes with other functions in a deep learning framework; taking a subprocess as a process for data reading and preprocessing, after the subprocess is created, the main process will send a batch of sample data to be processed to the subprocess through a data transmission pipeline between the subprocess and the main process; after receiving the batch of sample data, the subprocess reads the sample data corresponding to the batch from the external memory by calling the data reading and preprocessing functions provided by the user, and preprocesses the read sample data, and then transmits the preprocessed sample data to the main process through the data transmission pipeline between the main process and the main process. After receiving the preprocessed sample data transmitted by the subprocess, the main process uses the preprocessed sample data to perform training for the current training cycle.
[0073] The subprocess creation event includes, for example, starting a new round of training cycle for the deep learning model, or the batch number of preprocessed sample data required for training the deep learning model is less than a preset number threshold.
[0074] At the beginning of each epoch of training, the main process needs to create a subprocess for process data preprocessing; when a new round of training for the deep learning model begins, the main process detects whether an intermediate process has been created. If it is detected that an intermediate process has been created, a process creation instruction is sent to the intermediate process.
[0075] The process creation instruction carries the process parameters required for creating the child process. After receiving the process creation instruction, the intermediate process creates the child process in the first manner according to the process parameters carried in the process creation instruction.
[0076] In the embodiment of the present disclosure, the first method includes executing a split fork function based on a process parameter carried in a process creation instruction. The first method is used to create a child process, including: the intermediate process executes a split fork function based on the process parameter carried in the process creation instruction to create the child process.
[0077] When the intermediate process executes the fork function, the created child process can inherit the status and data of the intermediate process. In order to ensure that when the intermediate process uses the fork method to create a child process, there will be no deadlock of the child process due to the intermediate process including multiple threads, the intermediate process usually only includes one thread, and the functions of the thread include: in response to receiving the process creation instruction sent by the main process, create a child process, and after the child process is successfully created, create a data transmission pipeline between the main process and the child process. In response to receiving the process closing instruction sent by the main process, close the child process.
[0078] In this way, since the intermediate process will always exist during the deep learning model training process, in each training cycle, the intermediate process can use the fork method to create a child process, which can not only improve the efficiency of child process creation, but also ensure that the child process will not be deadlocked due to the creation of multi-threaded processes.
[0079] In another embodiment of the present disclosure, it also includes: in response to the intermediate process not being created, the main process creates the intermediate process in a second manner, and after the intermediate process is successfully created, sends the process creation instruction to the intermediate process.
[0080] In an embodiment of the present disclosure, the second method includes executing a spawn function. In response to the main process not creating the intermediate process, creating the intermediate process in the second method includes: the main process executes a spawn function to create the intermediate process. In this way, since the intermediate process is created in the spawn method, the intermediate process will not have the problem of process deadlock caused by the fork method.
[0081] In another embodiment of the present disclosure, the main process can also create an intermediate process by executing a fork function. Before executing the fork function, the main process detects the number of threads in the main process. When the number of threads is greater than 1, the main process closes other threads except the driver thread and executes the fork function to create an intermediate process. After creating the intermediate process, other closed threads are started, so that when the fork function is executed, the main process only includes one thread, avoiding the deadlock of other processes caused by the main process including multiple threads when creating other processes by fork.
[0082] In another embodiment of the present disclosure, the process creation method provided by the embodiment of the present disclosure further includes: after the intermediate process creates the child process, the intermediate process creates a data communication pipeline between the main process and the child process.
[0083] Exemplarily, after creating a child process, the intermediate process can, for example, feed back the creation result to the main process; the creation result carries the process ID of the child process; after the main process obtains the process ID, for example, it can communicate with the child process based on the process ID; in addition, after creating the child process, the intermediate process can also pass the process ID of the main process to the child process, so that the child process can communicate with the main process based on the process ID of the main process, thereby realizing the creation of a data communication pipeline between the main process and the child process.
[0084] In response to the successful creation of the data communication pipeline, the main process sends a data processing instruction to the sub-process through the data communication pipeline; the data processing instruction carries a data identifier, such as a batch of sample data that needs to be preprocessed.
[0085] In response to receiving the data processing instruction sent by the main process, the sub-process loads the target data corresponding to the data identifier based on the data identifier carried in the data processing instruction, and performs preset processing on the target data.
[0086] In a specific implementation, when a subprocess loads target data based on a data identifier carried in a data processing instruction, for example, it can first load a data page table corresponding to the sample data; the data page table includes index information of the sample data; the subprocess can query the physical address of the storage target data corresponding to the target data and the length of the target data from the page table based on the data identifier, and then read the target data from the external memory based on the read physical address and the length of the target data.
[0087] After the target data is read, the preprocessing function defined by the user is executed to preprocess the target data and obtain result data after the preprocessing of the target data.
[0088] In addition, after the sub-process pre-processes the target data and obtains the result data, it can also transmit the result data to the main process through the data communication pipeline.
[0089] After receiving the result data transmitted by the sub-process, the main process uses the result data to execute the model training process.
[0090] In the disclosed example, after receiving the result data transmitted by the child process, the main process can use the result data to train the deep learning model in the current training cycle, and / or use the result data to train the deep learning model in the future training cycle.
[0091] In another embodiment of the present disclosure, in response to sending the result data to the main process, the child process actively exits; after the next child process creation event is triggered, the main process will again send a process creation instruction to the intermediate process, and after receiving the process creation instruction, the intermediate process will re-create a new child process.
[0092] In addition, after receiving the result data sent by the child process, the main process can also send a process closing instruction to the intermediate process. The intermediate process receives the process closing instruction sent by the main process and controls the child process to close.
[0093] See also Figure 2 As shown, the embodiment of the present disclosure also provides a specific example of using the process creation method provided by the embodiment of the present disclosure to perform deep learning model training, including:
[0094] S201: The deep learning framework is started, and the main process of the deep learning framework starts running.
[0095] S202: The main process is triggered in response to the child process creation event, and detects whether an intermediate process has been created; if not, jump to S203; if so, jump to S204.
[0096] S203: Create the intermediate process in spawn mode. Jump to S204.
[0097] S204: Send a process creation instruction to the intermediate process.
[0098] S205: The intermediate process receives the process creation instruction sent by the main process, creates a child process by fork, and creates a data communication pipeline between the main process and the child process.
[0099] S206: After the child process is successfully created, the data page table corresponding to the sample data is loaded.
[0100] S207: The intermediate process feeds back to the main process information on successful creation of the sub-process and related information of the data communication pipeline.
[0101] S208: The main process sends identification information of the target data, such as a batch index, to the sub-process according to the relevant information of the data communication pipeline.
[0102] S209: The child process receives the identification information of the target data sent by the main process, and obtains the storage address and data length of the target data from the data page table according to the identification information and the loaded data page table.
[0103] S210: The subprocess reads the target data from the sample data stored in the external memory according to the storage address and data length of the target data, and loads the target data from the external memory into the internal memory.
[0104] S211: The subprocess pre-processes the target data by calling the pre-processing function configured by the user to obtain result data.
[0105] S212: The sub-process sends result data to the main process through the data communication pipeline between the sub-process and the main process.
[0106] S213: After the main process receives the result data sent by the child process, it uses the result data to train the deep learning model in this training cycle, and / or uses the result data to train the deep learning model in future training cycles.
[0107] S214: After the main process receives the result data sent by the child process, it will also send a process closing instruction to the intermediate process.
[0108] S215: After receiving the process closing instruction sent by the main process, the intermediate process closes the child process.
[0109] Through the above process, a round of training for the deep learning model is achieved.
[0110] Through the above process, the problem of sub-process deadlock caused by the main process directly using the fork method to create a sub-process can be avoided while ensuring the speed of sub-process creation.
[0111] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0112] Based on the same inventive concept, the embodiment of the present disclosure also provides a process creation device corresponding to the process creation method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned process creation method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0113] Reference Figure 3 FIG. 1 is a schematic diagram of a process creation device provided by an embodiment of the present disclosure, wherein the device comprises: a main process module 31 and an intermediate process module 32; wherein:
[0114] The main process module 31 is used to detect whether an intermediate process module has been created in response to a sub-process creation event being triggered; in response to the intermediate process module having been created, send a process creation instruction to the intermediate process module;
[0115] The intermediate process module 32 is used to create a sub-process module in a first manner in response to receiving the process creation instruction.
[0116] In an optional embodiment, the intermediate process module 32, in response to receiving the process creation instruction, creates the child process module in the first manner, and is used to: execute a split fork function based on the process parameters carried in the process creation instruction to create the child process module.
[0117] In an optional implementation, the main process module 31 is further used to: in response to the intermediate process module not being created, create the intermediate process module using a second method, and after the intermediate process module is successfully created, send the process creation instruction to the intermediate process module.
[0118] In an optional implementation, in response to the intermediate process module not being created and the intermediate process module being created in the second manner, the intermediate process module 32 is configured to: execute a spawn function to create the intermediate process module.
[0119] In an optional implementation manner, the child process creation event is triggered, including at least one of the following:
[0120] A new training cycle for the deep learning model begins;
[0121] The number of batches of preprocessed sample data required for training the deep learning model is less than a preset number threshold.
[0122] In an optional implementation, the intermediate process module 32 is also used to create a data communication pipeline between the main process module and the sub-process module.
[0123] In an optional implementation manner, the main process module 31 is further configured to send a data processing instruction to the sub-process module through the data communication pipeline in response to the successful creation of the data communication pipeline; the data processing instruction carries a data identifier;
[0124] It also includes: a sub-process module 33, which is used to load the target data corresponding to the data identifier in response to receiving the data processing instruction sent by the main process, based on the data identifier carried in the data processing instruction, and perform preset processing on the target data.
[0125] In an optional implementation, the sub-process module 33, after performing preset processing on the target data to obtain result data, is also used to transmit the result data to the main process module through the data communication pipeline.
[0126] In an optional implementation manner, the main process module 31 is further configured to send a process closing instruction to the intermediate process module in response to receiving the result data sent by the sub-process module;
[0127] The intermediate process module 32 is further configured to close the sub-process module in response to receiving the process closing instruction.
[0128] In an optional implementation, the main process module 31 is further configured to, in response to receiving the result data sent by the sub-process module, use the result data to train the deep learning model in the current training cycle;
[0129] And / or, using the result data to train the deep learning model in future training cycles.
[0130] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0131] The present disclosure also provides an electronic device, such as Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, including:
[0132] A processor 41 and a memory 42; the memory 42 stores machine-readable instructions executable by the processor 41, and the processor 41 is used to execute the machine-readable instructions stored in the memory 42. When the machine-readable instructions are executed by the processor 41, the processor 41 performs the following steps:
[0133] In response to the child process creation event being triggered, the main process detects whether an intermediate process has been created currently; in response to the intermediate process having been created, a process creation instruction is sent to the intermediate process;
[0134] In response to receiving the process creation instruction, the intermediate process creates the child process in a first manner.
[0135] The above-mentioned memory 42 includes internal memory 421 and external memory 422; the memory 421 here is also called internal memory, which is used to temporarily store the calculation data in the processor 41 and the data exchanged with the external memory 422 such as the hard disk. The processor 41 exchanges data with the external memory 422 through the internal memory 421.
[0136] The specific execution process of the above instructions can refer to the steps of the process creation method described in the embodiment of the present disclosure, which will not be repeated here.
[0137] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the process creation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0138] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the process creation method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0139] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0140] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0142] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0143] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0144] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A process creation method, characterized in that: include: In response to the child process creation event being triggered, the main process detects whether the intermediate process has been created; In response to the intermediate process being created, a process creation instruction is sent to the intermediate process; the intermediate process is a single-threaded process for creating a sub-process; the main process is a multi-threaded process; the main process is the main part of the deep learning framework, and each module and process during the training of the deep learning model is controlled and promoted by the main process; the sub-process is a process in the deep learning framework for data reading and preprocessing; In response to receiving the process creation instruction, the intermediate process creates a child process in a first manner.
2. The process creation method according to claim 1, characterized in that: In response to receiving the process creation instruction, the intermediate process creates a child process in a first manner, including: The intermediate process executes a split fork function based on the process parameters carried in the process creation instruction to create the child process.
3. The process creation method according to claim 1 or 2, characterized in that: The process creation method also includes: In response to the intermediate process not being created, the main process creates the intermediate process in a second manner, and after the intermediate process is successfully created, sends the process creation instruction to the intermediate process.
4. The process creation method according to claim 3, characterized in that: In response to the main process not creating the intermediate process, creating the intermediate process in a second manner, including: The main process executes the generated spawn function to create the intermediate process.
5. The process creation method according to any one of claims 1 to 4, characterized in that: The child process creation event is triggered, including at least one of the following: A new training cycle for the deep learning model begins; The number of batches of preprocessed sample data required for training the deep learning model is less than a preset number threshold.
6. The process creation method according to any one of claims 1 to 5, characterized in that: The process creation method also includes: The intermediate process creates a data communication pipe between the main process and the child process.
7. The process creation method according to claim 6, characterized in that: The process creation method also includes: In response to the data communication pipeline being successfully created, the main process sends a data processing instruction to the sub-process through the data communication pipeline; the data processing instruction carries a data identifier; In response to receiving the data processing instruction sent by the main process, the sub-process loads the target data corresponding to the data identifier based on the data identifier carried in the data processing instruction, and performs preset processing on the target data.
8. The process creation method according to claim 7, characterized in that: The process creation method also includes: After the sub-process performs preset processing on the target data and obtains result data, the sub-process transmits the result data to the main process through the data communication pipeline.
9. The process creation method according to claim 8, characterized in that: The process creation method also includes: In response to receiving the result data sent by the child process, the main process sends a process closing instruction to the intermediate process; In response to receiving the process closing instruction, the intermediate process closes the child process.
10. The process creation method according to claim 8, characterized in that: The process creation method also includes: In response to receiving the result data sent by the child process, the main process uses the result data to train the deep learning model in the current training cycle; And / or, using the result data to train the deep learning model in future training cycles.
11. A process creation device, characterized in that: include: Main process module, and intermediate process module; among them, The main process module is used to detect whether the intermediate process module has been created in response to the child process creation event being triggered; in response to the intermediate process module having been created, send a process creation instruction to the intermediate process module; the intermediate process is a single-threaded process for sub-process creation; the main process is a multi-threaded process; the main process is the main part of the deep learning framework, and all modules and processes during the training of the deep learning model are controlled and promoted by the main process; the sub-process is a process in the deep learning framework for data reading and preprocessing; The intermediate process module is used to create a child process in a first manner in response to receiving the process creation instruction.
12. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the process creation method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by an electronic device, the electronic device executes the steps of the process creation method according to any one of claims 1 to 10.