PaddlePaddle deep learning framework transplantation method based on domestic SW processor

Abstract

The present invention discloses a method for transplanting a PaddlePaddle deep learning framework based on a domestic SW processor, the method comprising: building a compilation environment, modifying the compilation strategy of the deep learning framework and its third-party dependent libraries according to the compilation system of the SW processor in the compilation environment; modifying the static source code and dynamic source code of the deep learning framework based on the SW processor; compiling the deep learning framework and the third-party dependent libraries separately to generate so libraries, and performing page alignment on the so libraries according to the executable file structure of the SW processor, thereby completing the transplantation of the deep learning framework on the SW processor. This method can transplant the PaddlePaddle deep learning framework to the Shenwei supercomputer based on the domestic SW processor, which can meet the needs of more application scenarios, and the code is open source, has completely independent intellectual property rights, and can be continuously optimized and improved.

Description

Technical Field

[0001] The present invention relates to the field of high-performance computing technology, and in particular to a PaddlePaddle deep learning framework transplantation method based on a domestic SW processor. Background Art

[0002] Artificial Intelligence (AI) is an important driving force for the new round of scientific and technological revolution and industrial transformation, and deep learning has become the most influential key common technology. In the past decade, deep learning has made breakthrough progress in both technological progress and application implementation. Deep learning methods have led to leapfrog development in AI-related technical fields such as speech, computer vision, and natural language processing, and have also achieved great success in many industry applications. At present, technical research related to deep learning continues to be hot, and deep learning is also pushing artificial intelligence into the era of industrial mass production. As the underlying infrastructure of deep learning technology and applications, the importance of deep learning frameworks and platforms has become increasingly prominent, attracting widespread attention from all walks of life.

[0003] The rapid development of deep learning and applications urgently requires improving efficiency through programming frameworks and development platforms. The deep learning framework is connected to the chip, which can shield the underlying hardware differences and provide good execution performance; it is connected to the application, becoming a standard environment for algorithm development and implementation. In recent years, large foreign technology companies have laid out in this field. Google's TensorFlow and Facebook's PyTorch have gained wide influence in the open source community. The earliest and most influential open source in China is PaddlePaddle launched by Baidu, which has now developed into a fully open source, technologically advanced, and fully functional industrial-grade deep learning platform. The PaddlePaddle framework is based on the deep learning computing abstraction design that is more consistent with programming and the corresponding front-end and back-end design, achieving a combination of high efficiency and ease of use. It has an easy-to-learn and easy-to-use front-end programming interface, a unified and efficient internal core architecture, native and complete distributed training support, and a high-performance and scalable inference engine. The PaddlePaddle framework is naturally compatible with both imperative and declarative programming paradigms, and has significant advantages in large-scale distributed training, multi-terminal deployment, and industrial-grade open source model libraries. It is a leading, mature, stable, and large-scale promotion-ready deep learning platform in China.

[0004] Supercomputers are an important indicator of a country's scientific and technological development level and comprehensive national strength. They are characterized by powerful functions, fast computing speed, and large storage capacity. They support the calculation of large and complex application topics and are mostly used in national high-tech fields and cutting-edge technology research. "Shenwei*TaihuLight" is the world's first supercomputer with a performance of more than 100PFlops. It is also my country's first supercomputer with completely independent intellectual property rights. At present, its computing power ranks third in the world in TOP500 (ranked first for three consecutive years from 2016 to 2018) and first in China. Its peak performance is 9.3 petaflops. It is equipped with the domestically produced SW26010 heterogeneous multi-core processor, which uses the 64-bit independent Shenwei instruction set and adopts a master-slave core architecture, such as Figure 1 As shown in the figure, a processor contains 4 core groups, each of which contains 1 master core, 1 8*8 slave core array and 1 memory controller. The peak computing power of a single processor is 3TFlops, and the memory access bandwidth is 130GB / s. The "Sunway TaihuLight" supercomputer contains a total of 40,960 SW26010 heterogeneous many-core processors.

[0005] Deep learning is the key core technology that has driven breakthroughs in artificial intelligence in recent years. It has led to leapfrog development in speech, vision, and natural language related technology fields, and accelerated the industrialization of artificial intelligence. As the underlying infrastructure of deep learning, deep learning frameworks and platforms have played an important role in promoting the rapid development of deep learning technologies and applications. Sunway Supercomputer is a supercomputer with completely independent intellectual property rights in my country. The existing machine learning frameworks of the domestic Sunway Supercomputer include Facebook's Pytorch and Google's TensorFlow. Although these two machine learning frameworks are also used by a large number of people, there are obvious deficiencies in the Sunway Supercomputer:

[0006] 1. The two existing machine learning frameworks of Sunway Supercomputer are outdated. The Sunway Supercomputer version of Pytorch is 1.5.0, while the latest official version of Pytorch is 1.10.1. The Sunway Supercomputer version of TensorFlow is 2.3.0, while its latest official version is 2.7.0. The versions are generally outdated and cannot meet new application requirements;

[0007] 2. The two existing machine learning frameworks of the Sunway supercomputer are not open source, and the key source code has been packaged into library files, making it impossible to continuously optimize and evolve;

[0008] 3. The two existing machine learning frameworks of Sunway Supercomputer are both developed by foreign manufacturers and have no independent intellectual property rights;

[0009] 4. When training with large-scale data, there is no optimization of memory and data sampling for domestic SW processors. Summary of the invention

[0010] In view of the technical defects of the existing machine learning framework version of the above-mentioned Shenwei supercomputer being outdated and unable to be optimized, the present invention proposes a PaddlePaddle deep learning framework transplantation method based on a domestic SW processor, including:

[0011] Compilation strategy modification step: building a compilation environment, and modifying the compilation strategy of the deep learning framework and its third-party dependent libraries according to the compilation system of the SW processor in the compilation environment;

[0012] Source code modification step: modifying the static source code and the dynamic source code of the deep learning framework based on the SW processor;

[0013] Compilation and transplantation steps: compile the deep learning framework and the third-party dependent library separately to generate a so library, and perform page alignment on the so library according to the executable file structure of the SW processor, thereby completing the transplantation of the deep learning framework on the SW processor.

[0014] The above-mentioned PaddlePaddle deep learning framework transplantation method also includes:

[0015] Framework optimization step: After the large-scale data set is read in parallel by the computing nodes in the multi-core processor, the read data is input into the transplanted training system of the deep learning framework for model training.

[0016] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the compilation strategy modification step includes:

[0017] Steps to build the compilation environment: Install the environment software by compiling from source code;

[0018] Compile control FEATURE definition step: define the compile control FEATURE in the compilation file of the deep learning framework;

[0019] The step of modifying the calling path of the environment software: building and modifying the calling path of the environment software when compiling according to the local environment;

[0020] Compilation parameter modification step: in the compilation file of the deep learning framework, modify the compilation options and compilation parameters in the deep learning framework according to the compilation options and compilation parameters of the SW processor;

[0021] Compile control feature closing step: closing the hardware compile control features that are not related to the SW processor and the compile control features that cannot be matched in the compile control features.

[0022] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the compilation strategy modification step also includes:

[0023] A third-party library path definition step: defining a download path and a compilation path of a third-party dependent library in a compilation file of the deep learning framework;

[0024] The third-party library compiled file modification step: creating a new compiled file of the third-party dependent library in the compilation path of the third-party dependent library, and modifying the compiled file of the third-party dependent library based on the compilation system of the SW processor.

[0025] In the above-mentioned PaddlePaddle deep learning framework transplantation method, the third-party library compilation file modification step also includes: modifying the compilation file path of the third-party dependent library, the third-party dependent library path, and the compilation instructions and compilation parameters of the third-party dependent library in the compilation file of the third-party dependent library.

[0026] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the source code modification step includes: compiling the dynamic source code that is strongly related to the hardware on the computing node of the SW processor through the bsub instruction, and dynamically generating dynamic source code that is strongly related to the SW processor.

[0027] In the above-mentioned PaddlePaddle deep learning framework transplantation method, the source code modification step also includes: controlling the source code modification based on the SW processor by defining a source code control FEATURE.

[0028] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the framework optimization step includes:

[0029] Data volume analysis step: obtaining the data volume of the large-scale data set and the number of currently idle computing nodes through analysis of the control node in the many-core processor;

[0030] Data size obtaining step: determining the size of data to be read by each main core according to the data volume of the large-scale data set, the number of currently idle computing nodes, and the number of main cores included in each computing node;

[0031] Model training step: The main core in each computing node reads data from the large-scale data set according to the size of the data that the main core needs to read, and inputs the read data into the training system for model training.

[0032] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the model training step also includes: each main core of each computing node reads the data into the memory cross-segment space, and after the training is completed, the data in the memory cross-segment space is merged and randomly sorted, and the randomly sorted data is distributed to each main core according to the size of the data to be read by the main core for the next training.

[0033] The above-mentioned PaddlePaddle deep learning framework transplantation method, wherein the framework optimization step also includes: when the control node detects that the training times of the main core reaches a preset number of times, re-execute the model training step.

[0034] Compared with the prior art, the advantages and positive effects of the present invention are:

[0035] 1. The present invention successfully transplanted Baidu's PaddlePaddle deep learning framework to the Shenwei supercomputer based on the domestic SW processor, which can meet the needs of more application scenarios, and the code is open source, with completely independent intellectual property rights, sustainable optimization and improvement, and the PaddlePaddle deep learning framework is developed by the domestic manufacturer Baidu, and the core technology is not subject to foreign manufacturers, with completely independent intellectual property rights;

[0036] 2. The PaddlePaddle deep learning framework based on the domestic SW many-core processor proposed in the present invention can carry out large-scale parallel model training and model prediction, accelerate the speed of machine learning model training and prediction, effectively shorten the model training and prediction time, and when performing parallel training on large-scale data, it takes into account memory optimization while considering random sampling of data sets, so that the trained model is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The SW26010 heterogeneous multi-core processor architecture diagram provided by the present invention;

[0038] Figure 2 A schematic diagram of the steps of the PaddlePaddle deep learning framework transplantation method based on a domestic SW processor provided by the present invention;

[0039] Figure 3 The present invention provides Figure 2 Schematic diagram of the process of step S1;

[0040] Figure 4 A flow chart of an embodiment of a PaddlePaddle deep learning framework transplantation method based on a domestic SW processor provided by the present invention;

[0041] Figure 5 A random sampling framework diagram of a large-scale data set based on a domestic SW processor provided by the present invention;

[0042] Figure 6 The PaddlePaddle deep learning framework model training diagram based on the domestic SW processor provided by the present invention;

[0043] Figure 7 A framework diagram of a PaddlePaddle deep learning framework transplantation system based on a domestic SW processor provided by the present invention;

[0044] Figure 8 is a framework diagram of a computer device according to an embodiment of the present application;

[0045] Wherein, the accompanying drawings are marked as follows:

[0046] 11. Compilation strategy modification unit; 111. Compilation environment building module; 112. Compilation control FEATURE definition module; 113. Environment software call path modification module; 114. Compilation parameter modification module; 115. Compilation control FEATURE closing module; 116. Third-party library path definition module; 117. Third-party library compilation file modification module 117; 12. Source code modification unit; 13. Compilation transplantation unit; 14. Framework optimization unit; 81. Processor; 82. Memory; 83. Communication interface; 80. Bus. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0052] Before describing in detail various embodiments of the present invention, the core inventive concept of the present invention is summarized and described in detail through the following several embodiments.

[0053] First, according to the SW processor compilation system of the domestic Sunway supercomputer, the compilation strategy of the Baidu PaddlePaddle deep learning framework was modified, and the compilation environment was built to compile and port the Baidu PaddlePaddle core framework and its dependent third-party associated libraries respectively; secondly, the source code of the PaddlePaddle deep learning framework was modified to solve the static code mismatch problem that occurred during the porting process; thirdly, for the dynamic source code that is strongly related to the hardware, bsub instructions and node compilation strategies were designed to dynamically generate dynamic source code that is strongly related to the domestic Sunway supercomputer SW processor; the porting and construction of the Baidu PaddlePaddle deep learning framework on the domestic Sunway supercomputer was completed; finally, data sampling optimization of the PaddlePaddle deep learning framework during parallel training of large-scale data sets was implemented.

[0054] Embodiment 1:

[0055] Figure 2 The present invention provides a step-by-step diagram of a method for transplanting a PaddlePaddle deep learning framework based on a domestic SW processor. Figure 2 As shown, this embodiment discloses a specific implementation method of the PaddlePaddle deep learning framework transplantation method based on the domestic SW processor (hereinafter referred to as the "method").

[0056] Specifically, the method disclosed in this embodiment mainly includes the following steps:

[0057] Step S1: Building a compilation environment, in which the compilation strategy of the deep learning framework and its third-party dependent libraries is modified according to the compilation system of the SW processor;

[0058] In this embodiment, the SW processor is the SW processor installed in the domestic Sunway supercomputer, and the deep learning framework is the Baidu PaddlePaddle deep learning framework.

[0059] Specifically, if Figure 3 As shown, step S1 specifically includes the following:

[0060] Step S11: installing the environment software by compiling the source code; wherein the environment software includes python3.6, protoc3.19.00, CMake3.19 and patchelf;

[0061] Step S12: defining a compilation control FEATURE in the compilation file of the deep learning framework; wherein the compilation control FEATURE is used to generally control the compatibility between the compilation modification of the adapted SW processor and the original compilation process;

[0062] Step S13: Building and modifying the calling path of the environment software during compilation according to the local environment;

[0063] Step S14: in the compilation file of the deep learning framework, modifying the compilation options and compilation parameters in the deep learning framework according to the compilation options and compilation parameters of the SW processor;

[0064] Since the name and parameters of the domestically produced Sunway supercomputer compiler are different from those of the original PaddlePaddle compiler, the name of the C++ compiler is changed to swg++, the name of the C compiler is changed to swgcc in the compilation file of the deep learning framework, and the compilation parameters are set to the corresponding parameters of the SW compiler.

[0065] Step S15: disabling hardware compile control features that are not related to the SW processor and compile control features that cannot be matched in the compile control features, so as to avoid compile errors due to mismatch on the domestic SW compiler;

[0066] Step S16: defining a download path and a compilation path of a third-party dependent library in a compilation file of the deep learning framework;

[0067] Specifically, due to the network limitations of the Sunway supercomputer, third-party dependency libraries cannot be downloaded online. To avoid compilation failures caused by network reasons, the download path of the third-party dependency library is defined in the compilation file of the deep learning framework. This path is a local path. The downloaded third-party dependency libraries are saved in this path for use in the PaddlePaddle framework compilation. The third-party dependency libraries include boost, pocketfft, xxhash, gloo, etc.

[0068] Step S17: creating a compilation file of the third-party dependency library in the compilation path of the third-party dependency library, and modifying the compilation file of the third-party dependency library based on the compilation system of the SW processor.

[0069] Step S17 also includes: modifying the compilation file path of the third-party dependency library, the third-party dependency library path, and the compilation instructions and compilation parameters of the third-party dependency library in the compilation file of the third-party dependency library.

[0070] Specifically, in the compilation file of the third-party dependent library, modify the compilation file path of the third-party dependent library, where the compilation files of each third-party dependent library are stored in this path. This is conducive to the protection of the original framework of PaddlePaddle and makes the compilation process based on domestic SW processors clearer;

[0071] In the compilation file of the third-party dependency library, change the third-party dependency library path used to store the third-party dependency library obtained from the download path to a separate path for the SW processor;

[0072] In the compilation file of the third-party dependent library, modify the compilation instructions and compilation parameters of the third-party dependent library to the instructions and parameters corresponding to the domestic SW processor.

[0073] Step S2: modifying the static source code and the dynamic source code of the deep learning framework based on the SW processor;

[0074] Specifically, based on the domestic SW processor, modifying the static source code of the PaddlePaddle deep learning framework can solve the static code mismatch problems that occur during the porting process, such as inconsistent function interfaces and inconsistent structure variable definitions due to different compilers. In addition, considering the compatibility of the source code on multiple platforms, define the source code FEATURE and overall control the source code modification based on the SW processor;

[0075] Among them, step S2 also includes: designing bsub instructions according to the queue name, the number of computing nodes, the sharing mode and the number of cores, compiling the dynamic source code strongly related to the hardware on the computing node of the SW processor through the bsub instruction, and dynamically generating dynamic source code strongly related to the SW processor.

[0076] Step S3: compile the deep learning framework and the third-party dependent library separately to generate a so library, and perform page alignment on the so library according to the executable file structure of the SW processor, thereby completing the transplantation of the deep learning framework on the SW processor.

[0077] The latest version of the PaddlePaddle deep learning framework based on the domestic SW multi-core processor can meet the needs of more application scenarios, and the code is open source, with completely independent intellectual property rights, and can be continuously optimized and improved. On the domestic Shenwei supercomputer, large-scale parallel model training and model prediction can be carried out based on this framework, which can speed up the training and prediction of machine learning models and effectively shorten the model training and prediction time.

[0078] Furthermore, in some embodiments, it also includes:

[0079] Step S4: After the large-scale data set is read in parallel by the computing nodes in the multi-core processor, the read data is input into the transplanted training system of the deep learning framework for model training.

[0080] Specifically, step S4 includes the following contents:

[0081] Step S41: Analyzing the data volume of the large-scale data set and the number of currently idle computing nodes through the control node in the many-core processor;

[0082] Step S42: determining the size of data to be read by each main core according to the amount of data in the large-scale data set, the number of currently idle computing nodes, and the number of main cores included in each computing node;

[0083] Step S43: The main core in each computing node reads data from the large-scale data set according to the size of the data that the main core needs to read, and inputs the read data into the training system for model training; in this way, the main core of the computing node reads data of a certain size from the large-scale data set in parallel, avoiding the situation where a single main core reads too much data and encounters insufficient memory.

[0084] Also, after a training is completed, parameter synchronization is required. The control node collects the gradient information of each computing node, performs weighted average calculation on the collected gradient information, and broadcasts the result to each computing node. The main core of each computing node updates the model parameters based on the result. After the main core of the computing node updates the model parameters, the next training starts.

[0085] Among them, step S43 also includes: each main core of each computing node reads data into the memory cross-segment space, and after training, merges the data in the memory cross-segment space, and randomly sorts it, and distributes the randomly sorted data to each main core for the next training according to the size of the data to be read by the main core. Among them, the cross-segment memory space of the main core in each computing node is shared. Considering the randomness of data set sampling, it is necessary to merge and randomly sort the data in the memory cross-segment before the next training, so that the IO overhead can be minimized while taking into account the randomness of data sampling in the node;

[0086] Finally, step S4 further includes: when the control node detects that the training times of the main core reaches a preset number, re-execute the model training step until the training is completed.

[0087] Please refer to the following Figure 4 . Figure 4 A flowchart of an embodiment of a PaddlePaddle deep learning framework transplantation method based on a domestic SW processor provided by the present invention; combined with Figure 4 The specific application process of this method is as follows:

[0088] Step 1: Build the compilation environment and install python3.6, protoc3.19.00, CMake3.19 and patchelf by compiling from source code;

[0089] Step 2: Due to the differences in the hardware architectures of the domestic Sunway supercomputer and X86, it is necessary to modify the compilation strategy of the Baidu PaddlePaddle deep learning framework based on the SW processor compilation system of the domestic Sunway supercomputer, so that the PaddlePaddle deep learning framework can be successfully compiled on the domestic Sunway supercomputer. Considering the cross-platform universality of the compilation strategy, the control FEATURE (BUILD_SW_MAKE) is defined in the existing compilation file CMakeLists.txt of the PaddlePaddle framework to control the overall compilation modification of the adaptation SW processor and the compatibility of the original compilation process;

[0090] Step 3: Build the path according to the local environment and modify the calling path of the environment software during compilation, such as Python software;

[0091] Step 4: Modify the compilation options and parameters: Since the name and parameters of the domestic Shenwei supercomputer compiler are different from those of the original PaddlePaddle compiler, in the CMakeLists.txt file, change the name of the C++ compiler to swg++, change the name of the C compiler to swgcc, and set the compilation parameters to the corresponding parameters of the SW compiler, such as: -mhost: instructs the SW processor to generate the main core code; -mftz: instructs the compiler to write true zeros to the target register when an underflow occurs; -mieee: instructs the compiler to allow any operation that produces precise results; -msimd: instructs the compiler to support the SIMD extended programming interface and generate corresponding SIMD extended instructions; -fPIC: compile to a library file, etc., so that the compilation parameters of the PaddlePaddle framework match those of the domestic Shenwei processor;

[0092] Step 5: Turn off the hardware compilation control FEATUREs that are not related to the SW processor and the compilation control FEATUREs that cannot be matched, to avoid compilation errors caused by mismatches on the domestic Shenwei compiler (Note: FEATURE means the switch that controls the compilation of a certain function);

[0093] Step 6: Due to the network limitations of the Shenwei supercomputer, third-party dependency libraries cannot be downloaded online. To avoid compilation failures caused by network reasons, define the download path of the third-party dependency library as LOCAL_3RD_ROOT in the CMakeLists.txt file. This path is a local path. The downloaded third-party dependency libraries are saved in this path for the PaddlePaddle framework to compile. Third-party dependency libraries include boost, pocketfft, xxhash, gloo, etc.

[0094] Step 7: Compilation and modification of third-party dependent libraries. To adapt to the compilation of domestic Shenwei processors, third-party dependent libraries need to be compiled and modified accordingly. Since there are many dependent libraries involved, in order to protect the original framework process and distinguish them from the original PaddlePaddle framework, a new third-party dependent library compilation path third_party_sw is created, and the compilation path of the third-party dependent library is introduced as third_party_sw in the CMakeLists.txt file;

[0095] Step 8: In the third_party_sw path, create a third-party library compilation file thrid_parth_sw.cmake, and modify the compilation file path of the third-party dependent library to external_sw in this file. The SW compiler compilation files of each third-party dependent library are stored in this path. This is conducive to the protection of the original framework of PaddlePaddle and makes the compilation process based on domestic SW processors clearer;

[0096] Step 9: Based on the domestic SW processor compilation system, modify the compilation files of each third-party dependent library, including: modify the third-party dependent library path, and modify the third-party dependent library path to a separate path for the domestic SW processor; modify the compilation instructions and parameters to the instructions and parameters corresponding to the domestic SW processor, such as -mhost compilation to generate the main core code, etc.;

[0097] Step 10: Based on the domestic SW processor, modify the source code of the PaddlePaddle deep learning framework to solve the static code mismatch problem that occurs during the porting process. For example, due to different compilers, there are inconsistent function interfaces and inconsistent structure variable definitions. In addition, considering the compatibility of the source code on multiple platforms, define the source code control FEATURE (BUILD_SW_CODE) to overall control the source code modification based on the SW processor;

[0098] Step 11: For dynamic source code that is strongly related to hardware, design bsub instructions according to the queue name, number of computing nodes, sharing mode, and number of slave cores, submit the compilation that is strongly related to hardware to the domestic SW processor computing node through the bsub instruction for compilation, and dynamically generate dynamic source code that is strongly related to the domestic SW processor, including Python code and C / C++ code;

[0099] Step 12: Compilation is completed. Since domestic SW processors uniformly perform 8k page alignment, it is necessary to perform page alignment PAGE_ALIGN on the compiled so library, aligning from 4k to 8k to match the executable file structure of domestic SW processors, in order to run successfully;

[0100] Step 13: Successfully implement the transplantation of Baidu PaddlePaddle framework on the domestic Shenwei supercomputer. Finally, test whether the PaddlePaddle deep learning framework based on the domestic SW processor can run successfully. Figure 6 Model training shown.

[0101] On the other hand, this embodiment also provides a data sampling optimization method for the PaddlePaddle deep learning framework during parallel training of large-scale data sets. Figure 5 . Figure 5 This is a random sampling framework diagram of a large-scale data set based on a domestic SW processor provided by the present invention; the data flow can be referenced Figure 5 , the main steps are as follows:

[0102] Step 1: Start the control node of the SW processor. The control node analyzes the data volume of the large-scale data set and the number of resources of the current idle computing nodes, and determines the size m of a large_batch. The size m of the large_batch is related to the number of idle computing nodes and can be obtained by the ratio of the data volume to the number of idle computing nodes.

[0103] Step 2: Each computing node (CPU) of the SW processor contains 4 main cores. The control node broadcasts a large_batch size value m to each computing node, that is, broadcasts the m / 4 value to each main core and defines m / 4 as a batch.

[0104] Step 3: The master core in each computing node receives a batch size n and reads n-sized data in parallel from a large-scale data set, thus avoiding insufficient memory due to a single master core reading too much data.

[0105] Step 4: The main cores in the computing node read their own data into the cross-segment memory space. The cross-segment memory space of the four main cores in each computing node is shared.

[0106] Step 5: Each main core inputs its own data into the training system of the PaddlePaddle deep learning framework to start model training. The size of each input data set in the training system is mini_batch;

[0107] Step 6: After a training is completed, parameter synchronization is required, and the control node is responsible for collecting the gradient information of each computing node;

[0108] Step 7: The control node performs weighted average calculation on the collected gradient information, and then broadcasts the result to each computing node. The main core of each computing node updates the model parameters.

[0109] Step 8: After the main core of the computing node updates the model parameters, the next training begins;

[0110] Step 9: After the training of the four main cores in a computing node is completed, considering the randomness of the batch data set sampling, it is necessary to merge the data in the memory cross-segment, sort the data randomly, and redistribute the data to the four main cores in a computing node for training according to the batch size. This can minimize the IO overhead while taking into account the randomness of the data sampling in the node;

[0111] Step 10: After the control node detects that the main core has been trained a certain number of times, considering the randomness of sampling of the large_batch data set, each computing node can read the large-scale data set in parallel again;

[0112] Step 11: Until the final training is completed.

[0113] The present invention overcomes the shortcomings of the prior art and provides a method for transplanting a PaddlePaddle deep learning framework based on a domestic SW processor. This method successfully transplants the Baidu PaddlePaddle deep learning framework to the Shenwei supercomputer based on a domestic SW processor. The transplanted PaddlePaddle version number is the latest stable version 2.2, which can meet the latest application requirements, and all codes are open source and can be continuously optimized and improved. In addition, PaddlePaddle is developed by the domestic manufacturer Baidu, and its core technology is not subject to foreign manufacturers, and it has completely independent intellectual property rights. The PaddlePaddle deep learning framework based on the domestic SW multi-core processor can carry out large-scale parallel model training and model prediction, speed up machine learning model training and prediction, and effectively shorten model training and prediction time. When large-scale data is trained in parallel, while taking into account memory optimization, random sampling of data sets is considered to make the trained model more accurate.

[0114] Embodiment 2:

[0115] In combination with the PaddlePaddle deep learning framework transplantation method based on a domestic SW processor disclosed in Example 1, this embodiment discloses a specific implementation example of a PaddlePaddle deep learning framework transplantation system based on a domestic SW processor (hereinafter referred to as the "system").

[0116] Reference Figure 7 As shown, the system comprises:

[0117] Compilation strategy modification unit 11: builds a compilation environment, and modifies the compilation strategy of the deep learning framework and its third-party dependent libraries according to the compilation system of the SW processor in the compilation environment;

[0118] Source code modification unit 12: modifying the static source code and dynamic source code of the deep learning framework based on the SW processor;

[0119] Compilation and transplantation unit 13: compile the deep learning framework and the third-party dependent library respectively to generate a so library, and perform page alignment on the so library according to the executable file structure of the SW processor, thereby completing the transplantation of the deep learning framework on the SW processor;

[0120] Framework optimization unit 14: After reading a large-scale data set in parallel through the computing nodes in the multi-core processor, the read data is input into the transplanted training system of the deep learning framework for model training.

[0121] Wherein, the compilation strategy modification unit 11 includes:

[0122] Compilation environment building module 111: installing environment software by compiling source code;

[0123] Compile control FEATURE definition module 112: defines a compile control FEATURE in a compilation file of the deep learning framework;

[0124] The environment software calling path modification module 113 is used to build and modify the calling path of the environment software when compiling according to the local environment;

[0125] Compilation parameter modification module 114: in the compilation file of the deep learning framework, modify the compilation options and compilation parameters in the deep learning framework according to the compilation options and compilation parameters of the SW processor;

[0126] A compile control feature closing module 115 is used to close hardware compile control features that are not related to the SW processor and compile control features that cannot be matched in the compile control features;

[0127] The third-party library path definition module 116 is used to define the download path and the compilation path of the third-party dependent library in the compilation file of the deep learning framework;

[0128] The third-party library compiled file modification module 117 creates a new compiled file of the third-party dependent library in the compilation path of the third-party dependent library, and modifies the compiled file of the third-party dependent library based on the compilation system of the SW processor.

[0129] For the technical solutions of the PaddlePaddle deep learning framework transplantation system disclosed in this embodiment and the other identical parts of the PaddlePaddle deep learning framework transplantation method disclosed in Embodiment 1, please refer to Embodiment 1 and will not be repeated here.

[0130] Embodiment three:

[0131] Combination Figure 8 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0132] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0133] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0134] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0135] The processor 81 implements any one of the PaddlePaddle deep learning framework transplantation methods in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory 82.

[0136] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 8 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0137] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0138] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0139] In addition, in combination with the PaddlePaddle deep learning framework transplantation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the PaddlePaddle deep learning framework transplantation methods in the above embodiment is implemented.

[0140] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A PaddlePaddle deep learning framework transplantation method based on domestic SW processor, characterized in that: include: Compilation strategy modification step: building a compilation environment, and modifying the compilation strategy of the deep learning framework and its third-party dependent libraries according to the compilation system of the SW processor in the compilation environment; Source code modification step: modifying the static source code and the dynamic source code of the deep learning framework based on the SW processor; Compilation and transplantation step: compile the deep learning framework and the third-party dependent library respectively to generate a so library, and perform page alignment on the so library according to the executable file structure of the SW processor, thereby completing the transplantation of the deep learning framework on the SW processor; Also includes: Framework optimization step: after reading the large-scale data set in parallel through the computing nodes in the multi-core processor, the read data is input into the transplanted training system of the deep learning framework for model training; The framework optimization steps include: Data volume analysis step: obtaining the data volume of the large-scale data set and the number of currently idle computing nodes through analysis of the control node in the many-core processor; Data size obtaining step: determining the size of data to be read by each main core according to the data volume of the large-scale data set, the number of currently idle computing nodes, and the number of main cores included in each computing node; Model training step: the main core in each computing node reads data from the large-scale data set according to the size of the data to be read by the main core, and inputs the read data into the training system for model training; The model training step also includes: each main core of each computing node reads data into the memory cross-segment space, and after training, merges the data in the memory cross-segment space and randomly sorts them, and distributes the randomly sorted data to each main core according to the size of the data to be read by the main core for the next training.

2. The PaddlePaddle deep learning framework transplantation method according to claim 1 is characterized in that: The compilation strategy modification step includes: Steps to build the compilation environment: Install the environment software by compiling from source code; Compile control FEATURE definition step: define the compile control FEATURE in the compilation file of the deep learning framework; The step of modifying the calling path of the environment software: building and modifying the calling path of the environment software when compiling according to the local environment; Compilation parameter modification step: in the compilation file of the deep learning framework, modify the compilation options and compilation parameters in the deep learning framework according to the compilation options and compilation parameters of the SW processor; Compile control feature closing step: closing the hardware compile control features that are not related to the SW processor and the compile control features that cannot be matched in the compile control features.

3. The PaddlePaddle deep learning framework transplantation method according to claim 2 is characterized in that: The compilation strategy modification step also includes: A third-party library path definition step: defining a download path and a compilation path of a third-party dependent library in a compilation file of the deep learning framework; The third-party library compiled file modification step: creating a new compiled file of the third-party dependent library in the compilation path of the third-party dependent library, and modifying the compiled file of the third-party dependent library based on the compilation system of the SW processor.

4. The PaddlePaddle deep learning framework transplantation method according to claim 3 is characterized in that: The third-party library compilation file modification step also includes: modifying the compilation file path of the third-party dependency library, the third-party dependency library path, and the compilation instructions and compilation parameters of the third-party dependency library in the compilation file of the third-party dependency library.

5. The PaddlePaddle deep learning framework transplantation method according to claim 1 is characterized in that: The source code modification step includes: compiling the dynamic source code strongly related to the hardware on the computing node of the SW processor through the bsub instruction, and dynamically generating the dynamic source code strongly related to the SW processor.

6. The PaddlePaddle deep learning framework transplantation method according to claim 5 is characterized in that: The source code modification step also includes: controlling the source code modification based on the SW processor by defining a source code control FEATURE.

7. The PaddlePaddle deep learning framework transplantation method according to claim 1 is characterized in that: The framework optimization step also includes: when the control node detects that the training times of the main core reaches a preset number of times, re-execute the model training step.

Images