A distributed compilation method, system, main server and storage medium

By enabling the engine editor and distributed compilation modules to work together and remotely launch the working compiler, the problem of excessive CPU resource consumption by the engine editor compiler is solved, resulting in faster compilation speed.

CN113885875BActive Publication Date: 2024-11-15SHANGHAI MIHA YOUHAIYUANCHENG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111159250.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-30
Publication Date
2024-11-15
Estimated Expiration
2041-09-30

AI Technical Summary

Technical Problem

When packaging or running games based on Unity3D, the engine editor's compiler consumes a large amount of CPU resources, causing the main server to work at full load and the compilation speed to reach a bottleneck.

Method used

By working in collaboration between the engine editor and the distributed compilation module, the working compiler is remotely launched, a suitable working server is selected from the target network, and the startup parameters are passed through, enabling the working compiler to connect to the engine editor for compilation.

Benefits of technology

By effectively utilizing the computing resources of the working server, the compilation speed was improved, and the problem of CPU overload on the main server was avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113885875B_ABST
    Figure CN113885875B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a distributed compilation method, system, main server and storage medium. The method is applied to a main server deployed with an engine editor and a distributed compilation module, and the method includes: after the engine editor receives the compilation task, the startup task and startup parameters corresponding to the compilation task are sent to the distributed compilation module; the distributed compilation module determines the working server from the candidate servers that can communicate with the engine editor in the target network according to the received startup task, and starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; the engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task. The technical solution of the embodiment of the present invention can speed up the compilation speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a distributed compilation method, system, main server and storage medium. Background Art

[0002] Currently, when packaging or running games based on Unity3D, the engine editor will compile shaders. When compiling a large number of shaders, the engine editor can start multiple compilers on its own main server to compile shaders separately.

[0003] In the process of implementing the present invention, the inventors found that the following technical problems exist in the prior art: the compiler will occupy a large amount of central processing unit (CPU) resources during compilation, which causes the CPU of the main server to often work at full capacity, and the compilation speed also reaches a bottleneck at this time. Summary of the invention

[0004] The embodiments of the present invention provide a distributed compilation method, system, main server and storage medium, which solve the problem of slow compilation speed.

[0005] In a first aspect, an embodiment of the present invention provides a distributed compilation method, which is applied to a main server where an engine editor and a distributed compilation module are deployed, and may include:

[0006] After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module;

[0007] The distributed compilation module determines a working server from candidate servers in the target network that can communicate with the engine editor according to the received startup task, starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters;

[0008] The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0009] In a second aspect, an embodiment of the present invention further provides a distributed compilation system, which may include: a distributed compilation module and an engine editor deployed on a main server; wherein:

[0010] The engine editor is used to send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module after receiving the compilation task;

[0011] The distributed compilation module is used to determine the working server from the candidate servers in the target network that can communicate with the engine editor according to the received startup task, start the working compiler deployed on the working server, and transparently transmit the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters;

[0012] The engine editor is also used to send the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0013] In a third aspect, an embodiment of the present invention further provides a main server, which may include:

[0014] one or more processors;

[0015] A memory for storing one or more programs;

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the distributed compilation method provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed compilation method provided by any embodiment of the present invention.

[0018] According to the technical solution of the embodiment of the present invention, after receiving the compilation task, the engine editor deployed on the main server can send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module deployed on the main server, so that the distributed compilation module can complete the startup work of the remote working compiler; the distributed compilation module determines the working server that can complete the compilation task from the candidate servers that can communicate with the engine editor in the target network according to the received startup task, and starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; thus, the engine editor can send the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task. The above technical solution starts the remote filtered working compiler through the distributed compilation module, and when the working compiler is successfully connected to the engine editor, it can be compiled on the working compiler. Compared with the method of compiling only on the main server (i.e., locally), the remote distributed compilation makes better use of the computing resources on the working server, thereby achieving the effect of fast compilation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1is a flow chart of a distributed compilation method in Embodiment 1 of the present invention;

[0020] Figure 2 is a flow chart of a distributed compilation method in Embodiment 2 of the present invention;

[0021] Figure 3 is a flow chart of a distributed compilation method in Embodiment 3 of the present invention;

[0022] Figure 4 is a flow chart of a distributed compilation method in Embodiment 4 of the present invention;

[0023] Figure 5 is a flowchart of an optional example of a distributed compilation method in Embodiment 4 of the present invention;

[0024] Figure 6a This is a first schematic diagram of a distributed compilation method in Embodiment 4 of the present invention when compiling a Unity Shader;

[0025] Figure 6b This is a second schematic diagram of a distributed compilation method in Embodiment 4 of the present invention when compiling a Unity Shader;

[0026] Figure 7 is a structural block diagram of a distributed compilation system in Embodiment 5 of the present invention;

[0027] Figure 8 It is a structural diagram of a main server in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It is also necessary to explain that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0029] Embodiment 1

[0030] Figure 1 1 is a flow chart of a distributed compilation method provided in the first embodiment of the present invention. This embodiment can be applied to the case of distributed compilation, and is particularly applicable to the case of distributed compilation through a working compiler in a target network (i.e., remote). The method can be executed by a distributed compilation system provided in an embodiment of the present invention, which includes an engine editor and a distributed compilation module integrated on a main server.

[0031] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:

[0032] S110. After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module.

[0033] The engine editor may be an editor in a certain engine that has the functions of editing, managing and organizing various resources, as well as performance analysis, packaging, deployment and version distribution. The compilation task may be a task for compiling at least one program (i.e., code). Compilation is a link in packaging, which is performed by a compiler. The compiler can be understood as a program that translates "one language (usually a high-level language)" into "another language (usually a low-level language)".

[0034] Normally, after receiving the compilation task, the engine editor starts the local compiler on the main server (i.e., local) where the engine editor is located for compilation. However, the computing resources of a main server are limited. In order to speed up the compilation speed, the compilation can be based on the working compiler deployed on each working server that can communicate with the engine editor in the target network. Relative to the local compiler, the working compiler can be understood as a remote compiler, where the target network can be a local area network. Therefore, after receiving the compilation task, the engine editor can send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module, so that the distributed compilation module can remotely start the working compiler. Among them, the startup task can be a task related to starting the working compiler, and the startup parameters can be parameters required to complete the startup and / or compilation process of the working compiler, both of which can be specifically determined according to the compilation task; the distributed compilation module can be a module that can remotely start the working compiler.

[0035] It should be noted that the engine editor and / or distributed compilation module can be a framework directly written according to the distributed compilation process proposed in the embodiment of the present invention, or a framework directly modified on the basis of an existing framework, which is not specifically limited here. Exemplarily, the distributed compilation module can be a rewritten distributed compilation tool FastBuild, IncrediBuild, distcc, etc., all of which have the function of communicating with the work server and screening out the work server suitable for completing the compilation task.

[0036] S120. The distributed compilation module determines a working server from candidate servers in the target network that can communicate with the engine editor according to the received startup task, starts a working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters.

[0037] Among them, a candidate compiler that can communicate with the engine editor is deployed on the candidate server. The candidate compiler is a compiler in the target network that can be used to complete the compilation task. The number of candidate compilers on each candidate server can be one, two or more, which is not specifically limited here. In order to complete the compilation task, it is not necessary to involve every candidate compiler here (this may be due to the large number of candidate compilers and the small number of programs that need to be compiled), or not every candidate compiler can immediately perform the compilation task at the current moment (this may be due to the candidate compiler being in the current moment. The remaining compilation tasks are caused), so the distributed compilation module can determine the working server from each candidate server, and the working compiler deployed on the working server can be a candidate compiler that can quickly complete the compilation task of this time, wherein the number of working compilers can be one, two or more, which is not specifically limited here. In actual applications, optionally, if the engine editor is understood as a server that plays a controlling role, then the working compiler can be understood as a client controlled by the server for the service corresponding to the server. The above-mentioned main server can be considered as a host, and the working server can be considered as a working machine.

[0038] After the distributed compilation module determines the working compiler, it can send various tools required to complete the compilation task, such as the executable program for compiling itself, and files such as dynamic libraries needed in the compilation process to the working compiler. Then, the distributed compilation module can remotely start the working compiler deployed on the working server, and can attach the received startup parameters during the startup process, that is, transparently pass the startup parameters to the working compiler. Since the Internet Protocol Address (IP address) where the engine editor is located can be included in the startup parameters, the working compiler can actively connect to the engine editor according to the startup parameters so as to be able to communicate with the engine editor. In other words, for each working compiler, it has the ability to communicate with the engine editor, but the premise of its implementation is to successfully connect to the engine editor. In this way, whether the compiler is local or remote, this is completely transparent and does not need to be concerned about the engine editor. The engine editor only needs to control the compiler connected to itself to compile. In practical applications, optionally, when the number of working compilers deployed on the working server is at least two, the distributed compilation module can start these working compilers one by one, and the upper limit of the number of working compilers that can be started on each working server can be determined according to the number of logical cores of the CPU on the working server.

[0039] S130: The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0040] Among them, after the engine editor and the working compiler are successfully connected, the engine editor can send the compilation task to the working compiler. At this time, the compilation task can be a program that the working compiler needs to compile. Therefore, the working compiler can compile according to the received compilation task, and then return the compilation result to the engine editor. In actual applications, optionally, since the program that needs to be compiled may be multi-segment, in order to ensure the compilation speed, these multiple program segments may need to be compiled by at least two working compilers together. Therefore, when sending the compilation task to the working compiler, for each working compiler, the corresponding number of programs in the compilation task can be sent according to its acceptance capacity.

[0041] According to the technical solution of the embodiment of the present invention, after receiving the compilation task, the engine editor deployed on the main server can send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module deployed on the main server, so that the distributed compilation module can complete the startup work of the remote working compiler; the distributed compilation module determines the working server that can complete the compilation task from the candidate servers that can communicate with the engine editor in the target network according to the received startup task, and starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; thus, the engine editor can send the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task. The above technical solution starts the remote filtered working compiler through the distributed compilation module, and when the working compiler is successfully connected to the engine editor, it can be compiled on the working compiler. Compared with the method of compiling only on the main server (i.e., locally), the remote distributed compilation makes better use of the computing resources on the working server, thereby achieving the effect of fast compilation.

[0042] An optional technical solution, sending the startup task and startup parameters corresponding to the compilation task to the distributed compilation module, may include: determining the compilation quantity of the working compiler that needs to be compiled according to the received compilation task; if the compilation quantity is greater than a preset startup threshold, splitting the compilation task into a first compilation task and a second compilation task corresponding to the preset startup threshold, and sending the startup task and startup parameters corresponding to the second compilation task to the distributed compilation module; accordingly, the engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task, may include: the engine editor sends the second compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received second compilation task; the engine editor sends the first compilation task to the working compiler that has been successfully connected and has completed the second compilation task, so that the working compiler compiles according to the received first compilation task.

[0043] Among them, the engine editor can determine the first program number of the program that needs to be compiled according to the received compilation task, and then combine the second program number of the program that each working compiler can compile, and can determine the compilation number of the working compiler required for this compilation task. Exemplarily, in order to ensure the compilation speed, only one program can be assigned to each working compiler at the same time, that is, the second program number is 1, and the compilation number is the first program number. The engine editor is pre-set with an upper limit (i.e., a preset startup threshold) for indicating the startup number of the working compiler that can be started at the same time, so when the compilation number is greater than the preset startup threshold, the compilation task can be split into a first compilation task and a second compilation task, wherein the number of programs that need to be compiled in the second compilation task is related to the preset startup threshold, or the preset startup threshold and the second program number. Then, the startup task and startup parameters corresponding to the second compilation task are sent to the distributed compilation module, so that the distributed compilation module starts the working compiler of the preset startup threshold to complete the second compilation task, and the first compilation task is in a waiting state. Furthermore, when a working compiler completes the compilation work and is in an idle state, the engine editor can send the program that needs to be compiled in the first compilation task to the working compiler to continue compiling, thereby ensuring the effective compilation of each program that needs to be compiled in the compilation task.

[0044] Another optional technical solution, after the engine editor receives the compilation task, the above-mentioned distributed compilation method may also include: determining the compilation number of the working compilers that need to be compiled and the idle number of the working compilers that have been started and are not compiling according to the received compilation task; if the compilation number is greater than the idle number, generating a startup task according to the difference between the compilation number and the idle number, or the compilation number. Among them, the working mode of some working compilers is that they can repeatedly process the compilation tasks sent by the engine editor during the life cycle, that is, they can successively process multiple compilation tasks after one startup, so after the engine editor determines the compilation number according to the compilation task, it can first determine the idle number of the working compilers that have been started and are not compiling (that is, in an idle state); when the compilation number is less than or equal to the idle number, then these working compilers that have been started and are in an idle state can complete this compilation task completely, without starting a new working compiler; otherwise, the startup task can be generated according to the difference between the compilation number and the idle number, that is, how many working compilers are started according to the difference, or the startup task can also be generated according to the compilation number, that is, directly starting as many working compilers as can complete this compilation task, thereby ensuring the effective completion of the compilation task.

[0045] Embodiment 2

[0046] Figure 2 It is a flow chart of a distributed compilation method provided in the second embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, determining a working server from candidate servers in the target network that can communicate with the engine editor can include: determining the number of startups of working compilers that need to be started, and the available computing resources of candidate servers in the target network that can communicate with the engine editor; determining a working server from the candidate servers based on the number of startups and the available computing resources. Among them, the explanations of terms that are the same as or corresponding to the above-mentioned embodiments are not repeated here.

[0047] See also Figure 2 The method of this embodiment may specifically include the following steps:

[0048] S210: After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module.

[0049] S220. The distributed compilation module determines the number of working compilers that need to be started and the available computing resources of each candidate server that can communicate with the engine editor in the target network based on the received startup task, and determines the working server from each candidate server based on the startup number and the available computing resources.

[0050] Among them, the available computing resources of a candidate server can indicate how many computing resources in the candidate server can still be used, or whether the candidate server is in an idle state. It can indirectly reflect how many candidate compilers deployed in the candidate server can perform compilation at the current moment. Therefore, according to the startup number and the available computing resources of each candidate server, the working server used to complete the compilation task can be determined from each candidate server, thereby achieving the effect of automatically screening out idle working servers that can match the startup task from the target network.

[0051] S230: The distributed compilation module starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters.

[0052] S240: The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0053] According to the technical solution of the embodiment of the present invention, the distributed compilation module determines the number of working compilers that need to be started and the available computing resources of each candidate server in the target network that can communicate with the engine editor through the received startup tasks. Since the available computing resources can indicate whether the corresponding candidate server is in an idle state, that is, whether the candidate compiler deployed on the candidate server can complete the compilation task at the current moment, the working server that is in an idle state and can match the startup task can be determined from each candidate server according to the startup number and the available computing resources, thereby achieving the effect of accurately determining the working server.

[0054] Embodiment 3

[0055] Figure 3 It is a flow chart of a distributed compilation method provided in the third embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the working compiler and the engine editor are connected to each other through a socket interface for communication; the engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task, which may include: the engine editor sends the compilation task to the working compiler that has been successfully connected through the socket interface, so that the working compiler compiles according to the received compilation task. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0056] See also Figure 3 The method of this embodiment may specifically include the following steps:

[0057] S310: After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module.

[0058] S320. The distributed compilation module determines a working server from the candidate servers in the target network that can communicate with the engine editor through a socket interface based on the received startup task, starts the working compiler deployed on the working server, and passes the received startup parameters to the working compiler so that the working compiler is connected to the engine editor based on the startup parameters.

[0059] Among them, the socket interface is an existing interface for data transmission in the engine editor and the candidate compiler, that is, there is a socket connection line between the main server and the candidate server. Therefore, in terms of communication (that is, data transmission), there is no need to modify the engine editor and the candidate compiler, and data transmission can be carried out directly through the existing socket interface. For example, the engine editor sends the compilation task to the candidate server through the socket interface, and the candidate server returns the compilation result to the engine editor through the socket interface, etc., thereby speeding up the research and development while ensuring effective communication.

[0060] S330: The engine editor sends the compilation task to the successfully connected working compiler through the socket interface, so that the working compiler compiles according to the received compilation task.

[0061] According to the technical solution of the embodiment of the present invention, since a socket connection line already exists between the engine editor and the candidate compiler, no modification is required in terms of data transmission. The existing socket interface on the engine editor and the candidate compiler can be directly used for data transmission, thereby achieving the effect of accelerating the research and development speed.

[0062] An optional technical solution, the above-mentioned distributed compilation method may also include: when the engine editor receives a compilation file request sent by the working compiler through the socket interface, the engine editor reads the compilation file corresponding to the compilation file request, and sends the read compilation file to the working compiler through the socket interface; correspondingly, so that the working compiler compiles according to the received compilation task, it may include: so that the working compiler compiles according to the received compilation task and compilation file.

[0063] Among them, since the engine editor and the compiled file are both located on the main server, and the working compiler is located on the candidate server, when the working compiler and the compiled file are located on different servers, the working compiler cannot directly read the compiled file through the file path of the compiled file on the main server. To this end, in order to ensure that the working compiler can work normally in a distributed situation, it can send a compiled file request for requesting the compiled file to the engine editor through the socket interface, so that the engine editor can determine the compiled file to be requested according to the received compiled file request, and read the compiled file according to the file path of the compiled file in the main server, and send the reading result to the working compiler through the socket interface. The above-mentioned compiled file can be the code that the working compiler needs to reference when completing the compilation task. Thus, the working compiler can obtain the compiled file by request, and then can continue the compilation task according to the compiled file. In addition, optionally, the working compiler can also cache the received compiled file in the memory of the candidate server where it is located, so that when the working compiler also needs to apply the cached compiled file when performing the remaining compilation tasks, it does not need to obtain the compiled file by request, but can directly obtain it from the candidate server, thereby achieving the effect of saving bandwidth and time.

[0064] Another optional technical solution, the above-mentioned distributed compilation method may also include: the engine editor starts the socket interface; correspondingly, so that the working compiler is connected to the engine editor according to the startup parameters, it may include: so that the working compiler finds the socket interface according to the startup parameters, and connects to the engine editor through the socket interface. Among them, the engine editor starts a socket interface (that is, starts a socket process), and after the working compiler is started, it will also start a socket process. Since the startup parameters include the IP address of the engine editor, the working compiler can find the socket process started by the engine editor through the startup parameters, and then realize the communication connection between itself and the engine editor through the socket process started by itself and the socket process started by the engine editor.

[0065] Embodiment 4

[0066] Figure 4It is a flow chart of a distributed compilation method provided in the fourth embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, after the working compiler is connected to the engine editor according to the startup parameters, the above-mentioned distributed compilation method may also include: if the engine editor determines that it has not been successfully connected to the working compiler, then starting the main compiler deployed on the main server, and sending the compilation task to the main compiler, so that the main compiler compiles according to the received compilation task. Among them, the explanation of the terms that are the same or corresponding to the above-mentioned embodiments will not be repeated here.

[0067] See also Figure 4 The method of this embodiment may specifically include the following steps:

[0068] S410. After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module.

[0069] S420. The distributed compilation module determines a working server from candidate servers in the target network that can communicate with the engine editor according to the received startup task, starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler so that the working compiler is connected to the engine editor according to the startup parameters.

[0070] S430 , the engine editor determines whether it has been successfully connected to the working compiler, and if so, executes S440 , otherwise, executes S450 .

[0071] Among them, due to various unstable factors in the target network, there may be a connection failure between the engine editor and the working compiler. Therefore, the engine editor can first determine whether it is successfully connected to the working compiler. If so, the working compiler will complete the compilation task. Otherwise, the main compiler (i.e., the local compiler) deployed on the main server will complete the compilation task, thereby ensuring that the compilation task can be effectively completed under any circumstances.

[0072] S440: The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0073] S450: The engine editor starts the main compiler deployed on the main server and sends the compilation task to the main compiler, so that the main compiler performs compilation according to the received compilation task.

[0074] When the engine editor fails to connect to the working compiler, the engine editor can start the local compiler, and after the engine editor successfully connects to the started local compiler, the compilation task is sent to the local compiler so that the local compiler completes the compilation task.

[0075] For example, see Figure 5 After receiving the compilation task, the engine editor first determines whether the online compilation (i.e., distributed compilation) has been turned on. If not, the local compiler is started to complete the compilation task by the local compiler. Otherwise, the startup task corresponding to the compilation task is sent to the distributed compilation module, and subsequent steps are executed; the distributed compilation module starts the corresponding working compiler according to the received startup task, and the started working compiler actively connects to the engine editor; if the working compiler and the engine editor are successfully connected, the working compiler starts compiling, otherwise the engine editor starts the local compiler to compile.

[0076] According to the technical solution of an embodiment of the present invention, when the engine editor fails to successfully connect to the working compiler, the engine editor starts the main compiler deployed on the main server and sends the compilation task to the main compiler, so that the main compiler compiles according to the received compilation task, thereby ensuring the effective completion of the compilation task under any circumstances.

[0077] In order to better understand the above technical solutions, combined with the application scenarios that may be involved in the embodiments of the present invention, the program to be compiled is Unity Shader, the working compiler is the modified Unity Shader Compiler, and the distributed compilation module is the modified FastBuild. Specifically,

[0078] The original Unity Shader Compiler is different from the common working mode of compilers. It does not directly use input, output and other parameters as the startup parameters of the compiler to start working. The process ends when the compilation is completed. It is actually a resident process that can repeatedly process the compilation tasks sent by the engine editor during its life cycle. The original FastBuild can only support the local distributed compilation of compilers with common working modes, and cannot directly support the distributed compilation of Unity Shader Compiler, nor can it support remote distributed compilation. On this basis, in order to realize the remote distributed compilation of Unity Shader Compiler, an optional solution is to adapt FastBuild completely by modifying Unity Shader Compiler without modifying FastBuild, that is, to change Unity Shader Compiler to the above-mentioned common one-time compilation working mode—exit the process when the compilation is completed. However, by combing the code, it is found that the workload of such modification is very large. The reason is that the interaction between the engine editor and the Unity Shader Compiler is completely based on the socket interface for communication, which basically meets the needs of remote compilation. Then the remaining thing to do is to start the compiler remotely. However, if Unity Shader Compiler is adapted to FastBuild, it means that the communication process between Unity Shader Compiler and the engine editor needs to be completely re-implemented, which is a lot of work. To this end, the embodiments of the present invention adopt another optional solution, which modifies FastBuild and adds a working mode that only starts Unity Shader Compiler remotely without waiting for Unity Shader Compiler to exit based on its original framework. This is because Unity Shader Compiler itself has basically supported the function of remote compilation, so the communication work in the compilation process can be handed over to Unity Shader Compiler itself, thereby achieving the effect of distributed compilation by adapting Unity Shader Compiler.

[0079] On this basis, the specific process of distributed compilation of Unity Shader Compiler can be found in Figure 6a and Figure 6bAfter receiving the compilation task, the Unity Editor deployed on the host, if online compilation is enabled, selects the working machine through FastBuild and starts the Unity Shader Compiler deployed on the working machine; then, the Unity Shader Compiler connects to the Unity Editor, and the two transmit data through the socket interface, for example, the Unity Editor sends the relevant code required in the shader compilation process to the UnityShader Compiler through the socket interface, and the Unity Shader Compiler sends the compilation result to the UnityEditor through the socket interface. In other words, the data flow can be summarized as: Unity3D packages / runs the game - Unity Editor gives the startup task to FastBuild (adding the socket module) - FastBuild automatically hosts (only starts but does not end); UnityShader Compiler starts - the socket process starts - Unity Editor connects to the UnityShader Compiler through the socket.

[0080] Embodiment 5

[0081] Figure 7 This is a block diagram of the structure of the distributed compilation system provided in Embodiment 5 of the present invention, which can be used to execute the distributed compilation method provided in any of the above embodiments. The system and the distributed compilation method in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the system, please refer to the embodiments of the above distributed compilation method. Figure 7 The system may include: an engine editor 510 and a distributed compilation module 520 deployed on a main server; wherein,

[0082] The engine editor 510 is used to send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module 520 after receiving the compilation task;

[0083] The distributed compilation module 520 is used to determine a working server from candidate servers in the target network that can communicate with the engine editor 510 according to the received startup task, start the working compiler deployed on the working server, and transparently transmit the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor 510 according to the startup parameters;

[0084] The engine editor 510 is further used to send the compilation task to the successfully connected working compiler, so that the working compiler compiles according to the received compilation task.

[0085] Optionally, determining a working server from candidate servers in the target network that can be communicatively connected to the engine editor 510 may include:

[0086] Determine the number of working compilers that need to be started and the available computing resources of each candidate server in the target network that can communicate with the engine editor 510;

[0087] According to the startup quantity and available computing resources, a working server is determined from the candidate servers.

[0088] Optionally, the working compiler and the engine editor 510 are connected for communication via a socket interface;

[0089] The engine editor 510 sends the compilation task to the successfully connected working compiler, so that the working compiler compiles according to the received compilation task, which may include:

[0090] The engine editor 510 sends the compilation task to the working compiler that has been successfully connected through the socket interface, so that the working compiler compiles according to the received compilation task.

[0091] On this basis, optionally, the engine editor 510 may also be used to: upon receiving a compilation file request sent by the working compiler through the socket interface, read the compilation file corresponding to the compilation file request, and send the read compilation file to the working compiler through the socket interface;

[0092] Accordingly, so that the working compiler compiles according to the received compilation task, it may include:

[0093] So that the working compiler can compile according to the received compilation tasks and compilation files.

[0094] Optionally, the engine editor 510 may also be used to: start a socket interface;

[0095] Accordingly, the process of connecting the working compiler to the engine editor 510 according to the startup parameters may include:

[0096] So that the working compiler finds the socket interface according to the startup parameters and connects to the engine editor 510 through the socket interface.

[0097] Optionally, the engine editor 510 may also be used to:

[0098] If it is determined that the connection to the working compiler is not successful, the main compiler deployed on the main server is started, and the compilation task is sent to the main compiler, so that the main compiler compiles according to the received compilation task.

[0099] Optionally, sending the startup task and startup parameters corresponding to the compilation task to the distributed compilation module 520 may include:

[0100] Determine the number of compilations of the working compilers required to compile according to the received compilation tasks;

[0101] If the number of compilations is greater than the preset startup threshold, the compilation task is split into a first compilation task and a second compilation task corresponding to the preset startup threshold, and the startup task and startup parameters corresponding to the second compilation task are sent to the distributed compilation module 520;

[0102] Accordingly, the engine editor 510 can be specifically used for:

[0103] Sending the second compile task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received second compile task;

[0104] The first compiling task is sent to the working compiler that has been successfully connected and has completed the second compiling task, so that the working compiler compiles according to the received first compiling task.

[0105] The distributed compilation system provided in the fifth embodiment of the present invention may include: a distributed compilation module and an engine editor deployed on a main server, wherein the engine editor may send the startup task and startup parameters corresponding to the compilation task to the distributed compilation module after receiving the compilation task, so that the distributed compilation module can complete the startup work of the remote working compiler; the distributed compilation module determines the working server that can complete the compilation task from the candidate servers that can communicate with the engine editor in the target network according to the received startup task, and starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; thus, the engine editor can send the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task. The above technical solution starts the remote screened working compiler through the distributed compilation module, and when the working compiler is successfully connected to the engine editor, it can be compiled on the working compiler. Compared with the method of compiling only on the main server (i.e., locally), the remote distributed compilation makes better use of the computing resources on the working server, thereby achieving the effect of fast compilation.

[0106] The distributed compilation system provided in the embodiment of the present invention can execute the distributed compilation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0107] It is worth noting that in the above-mentioned embodiment of the distributed compilation system, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0108] Embodiment 6

[0109] Figure 8 A schematic diagram of the structure of a main server provided in Embodiment 6 of the present invention is shown in FIG. Figure 8 As shown, the main server includes a memory 610, a processor 620, an input system 630, and an output system 640. The number of processors 620 in the main server can be one or more. Figure 8 A processor 620 is taken as an example; the memory 610, the processor 620, the input system 630 and the output system 640 in the main server can be connected via a bus or other means, Figure 8 The connection via bus 650 is taken as an example.

[0110] The memory 610, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the distributed compilation method in the embodiment of the present invention (for example, the engine editor 510 and the distributed compilation module 520 in the distributed compilation system). The processor 620 executes various functional applications and data processing of the main server by running the software programs, instructions and modules stored in the memory 610, that is, implementing the above-mentioned distributed compilation method.

[0111] The memory 610 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the main server, etc. In addition, the memory 610 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 610 may further include a memory remotely arranged relative to the processor 620, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0112] The input system 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the system. The output system 640 may include display devices such as display screens.

[0113] Embodiment 7

[0114] Embodiment 7 of the present invention provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a distributed compilation method when executed by a computer processor. The method can be applied to a main server deployed with an engine editor and a distributed compilation module, including:

[0115] After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module;

[0116] The distributed compilation module determines a working server from candidate servers in the target network that can communicate with the engine editor according to the received startup task, starts the working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters;

[0117] The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task.

[0118] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the distributed compilation method provided by any embodiment of the present invention.

[0119] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0120] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A distributed compilation method, It is characterized in that Applied to a main server, on which an engine editor and a distributed compilation module are deployed, the method includes: After receiving the compilation task, the engine editor sends the startup task and startup parameters corresponding to the compilation task to the distributed compilation module; The distributed compilation module determines a working server from candidate servers in a target network that can be communicatively connected with the engine editor according to the received startup task, starts a working compiler deployed on the working server, and transparently transmits the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; wherein the working compiler and the engine editor are communicatively connected via a socket interface; The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task; Wherein, the method further comprises: When the engine editor receives the compilation file request sent by the working compiler through the socket interface, it reads the compilation file corresponding to the compilation file request, and sends the read compilation file to the working compiler through the socket interface; The step of causing the working compiler to compile according to the received compiling task comprises: So that the working compiler performs compilation according to the received compilation task and the compilation file; wherein the working compiler also caches the received compilation file in the memory of the candidate server where the working compiler is located.

2. The method according to claim 1, It is characterized in that The step of determining a working server from candidate servers in the target network that can be communicatively connected with the engine editor includes: Determine the number of startups of the working compilers that need to be started and the available computing resources of each candidate server in the target network that can be communicatively connected with the engine editor; A working server is determined from each of the candidate servers according to the startup quantity and each of the available computing resources.

3. The method according to claim 1, It is characterized in that The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task, including: The engine editor sends the compilation task to the working compiler that has been successfully connected through the socket interface, so that the working compiler compiles according to the received compilation task.

4. The method according to claim 3, It is characterized in that Also includes: The engine editor starts the socket interface; The step of connecting the working compiler to the engine editor according to the startup parameters comprises: So that the working compiler finds the socket interface according to the startup parameters and connects to the engine editor through the socket interface.

5. The method according to claim 1, It is characterized in that After the step of connecting the working compiler to the engine editor according to the startup parameters, the step further includes: If the engine editor determines that it has not successfully connected to the working compiler, the main compiler deployed on the main server is started, and the compilation task is sent to the main compiler, so that the main compiler compiles according to the received compilation task.

6. The method according to claim 1, It is characterized in that The sending the startup task and startup parameters corresponding to the compilation task to the distributed compilation module includes: Determining the number of compilations of the working compiler that needs to be compiled according to the received compilation task; If the compilation quantity is greater than a preset startup threshold, splitting the compilation task into a first compilation task and a second compilation task corresponding to the preset startup threshold, and sending the startup task and startup parameters corresponding to the second compilation task to the distributed compilation module; The engine editor sends the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task, including: The engine editor sends the second compiling task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received second compiling task; The engine editor sends the first compilation task to the working compiler that has been successfully connected and has completed the second compilation task, so that the working compiler compiles according to the received first compilation task.

7. A distributed compilation system, It is characterized in that include: Distributed compilation modules and engine editors deployed on the main server; The engine editor is configured to send a startup task and startup parameters corresponding to the compilation task to the distributed compilation module after receiving the compilation task; The distributed compilation module is used to determine a working server from candidate servers in a target network that can be communicatively connected with the engine editor according to the received startup task, and start a working compiler deployed on the working server, and transparently transmit the received startup parameters to the working compiler, so that the working compiler is connected to the engine editor according to the startup parameters; wherein the working compiler and the engine editor are communicatively connected via a socket interface; The engine editor is further used to send the compilation task to the working compiler that has been successfully connected, so that the working compiler compiles according to the received compilation task; The engine editor is also used to: Upon receiving a compilation file request sent by the working compiler through the socket interface, reading a compilation file corresponding to the compilation file request, and sending the read compilation file to the working compiler through the socket interface; The step of causing the working compiler to compile according to the received compiling task comprises: So that the working compiler performs compilation according to the received compilation task and the compilation file; wherein the working compiler also caches the received compilation file in the memory of the candidate server where the working compiler is located.

8. A master server, It is characterized in that include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed compilation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the distributed compilation method according to any one of claims 1 to 6 is implemented.