Distributed multi-task packaging method and system, electronic equipment and storage medium

Abstract

The invention provides a distributed multi-task packaging method, system and device and a storage medium, and relates to the technical field of electronic commerce. The method is applied to a distributed multi-task packaging system, each packaging server receives a packaging request and stores task data required by the packaging request to a redis cache task queue; if the number of tasks in packaging of one packaging server is smaller than a preset packaging number threshold value, the packaging server takes out target task data in a waiting state in the redis cache task queue based on a preset rule, creates a packaging sub-process and transmits the target task data to the packaging sub-process; and the packaging sub-process packages the target task data to obtain a packaging result, and uploads the packaging result to the cloud server. According to the method and the device, the packaged task data can be stored persistently, the task queue data cannot be lost, all the task data are managed by using a global redis cache task queue, and the packaging efficiency is improved.

Description

Background Art

[0002] In recent years, in modern e-commerce systems, store decoration systems have become an indispensable tool for merchants to operate stores. The core business of store decoration is that merchants configure store pages through the decoration system. After the page configuration is completed and published, the page content configured by the merchant can be displayed on the merchant store client (referred to as the C-end) page, thereby realizing the configuration and operation of the page content by the merchant. The opening of the store decoration function allows third-party development organizations to provide store merchants with richer and more personalized store templates.

[0003] Based on the store development system, it is necessary to build a floor packaging system to verify the performance and safety of the locally developed floor source code, and finally package and generate floor static resources that can be rendered in the C-end store browsing system.

[0004] In the related art, packaging tasks are stored in local memory variables of the packaging server. However, the above method may cause uneven distribution of the number of packaging tasks on different packaging servers and low overall packaging efficiency. When the packaging server crashes or restarts, the packaging task data is lost, causing packaging interruption.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a distributed multi-task packaging, system, device and storage medium, which at least to a certain extent overcome the problems of uneven task distribution and low packaging efficiency in the multi-task packaging method in the related art.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a distributed multi-task packaging method is provided, which is applied to a distributed multi-task packaging system, wherein the distributed multi-task packaging system includes a load balancer and multiple packaging servers, and the method includes: the load balancer receives a packaging request and distributes the packaging request to the multiple packaging servers; each packaging server saves the task data required for the packaging request to a distributed database redis cache task queue; if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, the packaging server takes out the target task data in the waiting state in the redis cache task queue based on a preset rule, creates a packaging subprocess, passes the target task data to the packaging subprocess, the packaging subprocess executes packaging on the target task data, obtains a packaging result, and uploads the packaging result to a cloud server.

[0009] In one embodiment of the present disclosure, the redis cache data is operated in a distributed lock manner.

[0010] In one embodiment of the present disclosure, each packaging server saves the task data corresponding to the packaging request to the distributed database redis cache task queue, including: adding a first distributed lock; adding the task data to the end of the redis cache task queue; and releasing the first distributed lock.

[0011] In one embodiment of the present disclosure, before the one packaging server takes out the target task data in the waiting state in the redis cache task queue based on preset rules, the method further includes: adding a second distributed lock; after the one packaging server takes out the target task data in the waiting state in the redis cache task queue based on preset rules, the method further includes: adding the target task data to the task list of the one packaging server, and the task list of the one packaging server is maintained in the redis cache; after the one packaging server creates a packaging sub-process, the method further includes: releasing the second distributed lock.

[0012] In one embodiment of the present disclosure, each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue, including: each packaging server uploads the floor source code file of the packaging request to the cloud server, so that the packaging sub-process performs performance safety detection on the floor source code file.

[0013] In one embodiment of the present disclosure, the method further includes: if a packaging exception, verification failure, packaging timeout or normal packaging completion occurs during the packaging of the target task data, then the one packaging server ends the packaging sub-process, wherein an exception log is generated for the packaging exception.

[0014] In one embodiment of the present disclosure, after the one packaging server ends the packaging sub-process, the method further includes: the one packaging server checks whether there is any unexecuted task data in the redis cache task queue, and if so, executes the operation of taking out the target task data in the waiting state in the redis cache task queue based on a preset rule if the number of tasks in the packaging of a packaging server is less than a preset packaging number threshold, until all task data in the redis cache task queue are processed.

[0015] In another aspect of the present disclosure, a distributed multi-task packaging system is also provided, wherein the distributed multi-task packaging system includes a load balancer and multiple packaging servers, wherein the load balancer is used to receive packaging requests and distribute the packaging requests to the multiple packaging servers; each packaging server is used to save the task data required for the packaging request to the distributed database redis cache task queue; wherein, if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, then the packaging server takes out the target task data in the waiting state in the redis cache task queue based on preset rules, creates a packaging sub-process, passes the target task data to the packaging sub-process, and the packaging sub-process executes packaging on the target task data to obtain a packaging result, and uploads the packaging result to the cloud server.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned distributed multi-task packaging method by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the distributed multi-task packaging method described above is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer program or computer instructions, the computer program or the computer instructions being loaded and executed by a processor so that a computer implements the above-mentioned distributed multi-task packaging method.

[0019] In the disclosed embodiment, the load balancer receives a packaging request and distributes the packaging request to multiple packaging servers. Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue; if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, the above-mentioned packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rule, creates a packaging sub-process, and passes the target task data to the packaging sub-process; the packaging sub-process executes packaging on the target task data to obtain a packaging result, and uploads the packaging result to the cloud server, so that the packaging task data is persistently stored. When the packaging server crashes or restarts, the task queue data will not be lost. All task data are managed using a global redis cache task queue, which can be evenly distributed to all packaging server resources, thereby improving packaging efficiency.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 The following is a diagram showing the architecture of a distributed multi-task packaging system provided by an embodiment of the present disclosure;

[0023] Figure 2 A flow chart of a distributed multi-task packaging method provided by an embodiment of the present disclosure is shown;

[0024] Figure 3 A flow chart of a method for operating a redis cache task queue provided by an embodiment of the present disclosure is shown;

[0025] Figure 4 Another flow chart of a distributed multi-task packaging method provided by an embodiment of the present disclosure is shown;

[0026] Figure 5 A flow chart of another distributed multi-task packaging method provided by an embodiment of the present disclosure is shown;

[0027] Figure 6 A flowchart showing an example of a distributed multi-task packaging method provided by an embodiment of the present disclosure;

[0028] Figure 7A schematic diagram showing the structure of a distributed multi-task packaging system provided by an embodiment of the present disclosure is shown;

[0029] Figure 8 A structural block diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0033] ISV, the abbreviation of Independent Software Vendors, refers to independent software developers, that is, third-party development organizations.

[0034] A store refers to an online store or online shop opened by a merchant or seller on an e-commerce platform.

[0035] Floor is a module or component in the store page, such as product floor, carousel floor, etc.

[0036] Local development tools, a set of tools built on the local computer used by ISV designers to provide local development of ISV floor templates.

[0037] Store template, a set of display effects and interactive codes for store floors.

[0038] Packaging is the process in which after the ISV designer has developed the floor using local development tools, he calls the packaging system to parse and build it into the corresponding front-end static resources.

[0039] Distributed lock is a synchronization mechanism that supports coordinated work among multiple nodes in a distributed system. It can ensure that only one node accesses shared resources at the same time, thus avoiding data inconsistency.

[0040] Figure 1 An exemplary system architecture of a distributed multi-task packaging method or a distributed multi-task packaging system to which the embodiments of the present disclosure can be applied is shown.

[0041] like Figure 1 As shown, the system architecture may include a terminal device and a packaging system.

[0042] The packaging system includes a load balancer 101 , a packaging server 102 , a redis cache task queue 103 , and a cloud server 104 .

[0043] Users can interact with the packaging system through the network using terminal devices to receive or send messages.

[0044] The terminal device can be any electronic device with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, display screens, and desktop computers.

[0045] The client of the application that can be installed on the terminal device is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0046] Exemplarily, the terminal device is installed with client applications of various software development tools, such as Nodejs technology stack related applications, etc., and a packaging request can be initiated to the packaging system through the communication client application.

[0047] The load balancer 101 is used to load balance the packaging requests initiated by users and distribute them to various packaging servers 102.

[0048] The packaging server 102 may include multiple, for example Figure 1 In the embodiment, the packaging system includes two packaging servers 102 .

[0049] The packaging server 102 may be a server that provides packaging services. The packaging server 102 may analyze and process received request data (such as packaging requests, etc.), create a packaging subprocess to perform packaging on the packaging data, obtain a packaging result, and upload the packaging result to the cloud server 104.

[0050] Optionally, the packaging server 102 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0051] The cloud server 104 may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0052] The distributed database (Remote Dictionary Server, Redis) cache task queue 103 is used to store the task data required based on the packaging request, so as to store the packaging task data persistently. When the packaging server 102 crashes or restarts, the data in the redis cache task queue 103 will not be lost. All packaging tasks are managed by a global redis cache task queue 103, which can evenly distribute resources to all packaging servers 102 and improve the packaging efficiency of the packaging system.

[0053] Those skilled in the art will know that Figure 1 The number of terminal devices, packaging servers and cloud servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0054] Based on this, the present disclosure designs a distributed multi-task packaging system and related methods to solve at least part of the distributed multi-task packaging problem mentioned above.

[0055] The distributed multi-task packaging method provided by the present invention comprises the following steps: a load balancer receives a packaging request, distributes the packaging request to multiple packaging servers, and each packaging server saves the task data required for the packaging request to a distributed database redis cache task queue; if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, the above-mentioned packaging server takes out the target task data in a waiting state in the redis cache task queue based on a preset rule, creates a packaging subprocess, and passes the target task data to the packaging subprocess; the packaging subprocess executes packaging on the target task data to obtain a packaging result, and uploads the packaging result to a cloud server, so that the packaging task data is persistently stored. When the packaging server crashes or restarts, the task queue data will not be lost. All task data are managed using a global redis cache task queue, which can be evenly distributed to all packaging server resources, thereby improving packaging efficiency.

[0056] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0057] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0058] First, a distributed multi-task packaging method is provided in the embodiments of the present disclosure, and the method can be executed by any system with computing processing capabilities. In some embodiments, the distributed multi-task packaging method provided in the embodiments of the present disclosure can be executed by a server; in other embodiments, the distributed multi-task packaging method provided in the embodiments of the present disclosure can be implemented by a terminal device and a server through interaction.

[0059] Figure 2 A flow chart of a distributed multi-task packaging method provided in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the distributed multi-task packaging method provided in the embodiment of the present disclosure is applied to a distributed multi-task packaging system, which includes a load balancer and multiple packaging servers. The method includes the following steps:

[0060] S202: The load balancer receives the packaging request and distributes the packaging request to multiple packaging servers.

[0061] In one embodiment, the packaging request is initiated by a user or an ISV designer through a packaging button on a terminal device display interface, which is used to request a packaging interface to send a packaging request to a packaging server. It should be noted that the user can send multiple packaging requests at the same time or send one packaging request. In this disclosure, multiple packaging requests are used as an example.

[0062] The user may determine a packaging request based on the amount of data to be packaged, or may divide a piece of data to be packaged into multiple pieces of task data and create multiple packaging requests, which is not specifically limited in the present disclosure.

[0063] The system can also allocate multiple packaging requests through a load balancer so that the number of packaging requests received by different packaging servers is close, thereby improving the overall efficiency of the packaging system.

[0064] Exemplarily, the load balancer may distribute multiple packaging requests to corresponding packaging servers by recording the number of packaging requests. In this case, the load balancer may be a counting module in the packaging system.

[0065] Exemplarily, the load balancer may also determine a corresponding packaging server according to the resource situation of the packaging server occupied by the packaging request.

[0066] S204: Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue.

[0067] In one embodiment, each packaging server of the above S202 saves the task data required for the packaging request to the distributed database redis cache task queue, including: uploading the floor source code file of the packaging request to the cloud server, so that the packaging sub-process performs performance safety detection on the floor source code file.

[0068] For the above packaging request, one packaging request corresponds to one task data, and each task data includes at least one of the following information when stored in the redis cache: task number ID, task status, download address of the source code compressed package of the module to be packaged, the space name stored in the cloud storage, the file name of the source code file, the key corresponding to the cloud storage storage, the module name and other related information, the builder number assigned during packaging, the full-page template ID, the packaging subprocess pid, the start time of packaging, the task creation time, the waiting timeout, the module type, the account pin during packaging, etc. Among them, the task status may include waiting state, packaging, packaging completed, etc.

[0069] It should be noted that redis cache data is operated in a distributed lock manner to ensure data consistency.

[0070] S206. If there is a packaging server whose number of tasks in packaging is less than the preset packaging number threshold, the above packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rules, creates a packaging sub-process, passes the target task data to the packaging sub-process, and the packaging sub-process executes packaging on the target task data to obtain the packaging result, and uploads the packaging result to the cloud server.

[0071] The above-mentioned preset packing quantity threshold is used to determine the maximum packing quantity maxPackinNum that a packing server can perform on task data. The preset packing quantity threshold can be any positive integer and can be pre-configured in the packing server.

[0072] Exemplarily, the preset packaging quantity threshold may be configured as a fixed value, such as 50, 100, etc.

[0073] Exemplarily, the preset packaging quantity threshold may also be a dynamic value that can be dynamically configured, and the present disclosure does not make any specific limitation thereto.

[0074] When the number of tasks in the packaging of a packaging server is less than the preset packaging number threshold, it indicates that the packaging server still has the ability to perform packaging on the task data, and the target task data in the waiting state in the redis cache task queue is allocated to the packaging server.

[0075] When the number of tasks in the packaging of a packaging server is greater than or equal to the preset packaging number threshold, it indicates that the packaging server has reached the upper limit of its ability to execute packaging. At this time, the packaging server is waiting to execute the assigned target task data.

[0076] The above preset rule can be to retrieve the task data in the order in which they are added to the redis cache task queue, or to retrieve the task data according to the priority order of the task data, which is not specifically limited in the present disclosure.

[0077] In one embodiment, each packaging server can create at least one packaging subprocess. For example, Figure 1 In the example, packaging server 1 can create two packaging sub-processes, namely sub-process 1 and sub-process 2; packaging server 2 can also create two packaging sub-processes, namely sub-process 3 and sub-process 4.

[0078] The present disclosure uses Nodejs technology to create a packaging child process. When creating the packaging child process, a configurable maximum timeout packTimeOut can be set, and the task data is transferred to the packaging child process through the parent-child process communication technology.

[0079] In one embodiment, after the packaging subprocess receives the target task data, the packaging subprocess is started to start the performance safety detection of the floor source code file and packaging-related work.

[0080] The packaging result may include a successful packaging result and a failed packaging result, wherein the successful packaging result may include relevant information of the task data and the final packaging file, and the failed packaging result may include relevant information of the task data and the reason for the packaging failure, etc.

[0081] In the disclosed embodiment, the load balancer receives a packaging request and distributes the packaging request to multiple packaging servers. Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue; if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, the above-mentioned packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rule, creates a packaging sub-process, and passes the target task data to the packaging sub-process; the packaging sub-process executes packaging on the target task data to obtain a packaging result, and uploads the packaging result to the cloud server, so that the packaging task data is persistently stored. When the packaging server crashes or restarts, the task queue data will not be lost. All task data are managed using a global redis cache task queue, which can be evenly distributed to all packaging server resources, thereby improving packaging efficiency.

[0082] Figure 3A flowchart of a method for operating a redis cache task queue provided in an embodiment of the present disclosure is shown. Figure 3 As shown, each packaging server in the above S202 saves the task data corresponding to the packaging request to the distributed database redis cache task queue, including:

[0083] S302, adding a first distributed lock;

[0084] S304, adding the task data to the end of the redis cache task queue;

[0085] S306: Release the first distributed lock.

[0086] In one embodiment, a distributed lock is a synchronization mechanism that supports coordinated work among multiple nodes in a distributed system, and can ensure that only one node accesses a shared resource at a given time, thereby avoiding data inconsistency.

[0087] In the disclosed embodiment, by adding a first distributed lock before adding task data to the end of the redis cache task queue, and releasing the first distributed lock after the task data is added, multiple nodes are prevented from accessing and modifying the redis cache task queue at the same time, thereby ensuring data consistency.

[0088] In one embodiment, the distributed multi-task packaging method provided by the embodiment of the present disclosure also includes: adding a second distributed lock, taking out the target task data in a waiting state in the redis cache task queue based on preset rules; adding the target task data to the task list of the above-mentioned one packaging server, and the task list of the above-mentioned one packaging server is maintained in the redis cache; after the above-mentioned one packaging server creates a packaging sub-process, releasing the second distributed lock to ensure data consistency.

[0089] In one embodiment, the task data is added to the end of the redis cache task queue, and the task data at the head of the redis cache task queue is taken out, so as to facilitate the management of the task data.

[0090] The task list of the above-mentioned packaging server is used to record the task data assigned to the above-mentioned packaging server and the execution status of the task data. The content of the task list of the packaging server can be consistent with the content of the task data stored in the redis cache task queue, or it can be part of the content of the task data stored in the redis cache task queue. This disclosure does not make specific limitations.

[0091] Figure 4 Another distributed multi-task packaging method flow chart provided by the embodiment of the present disclosure is shown. Figure 2On the basis of the embodiment, after S206, S208 is added to limit the situation of monitoring the packaging process of the packaging subprocess. Figure 4 As shown, in one embodiment, the distributed multi-task packaging method provided by the embodiment of the present disclosure includes S202 to S208. The method also includes:

[0092] S208. If during the process of packaging the target task data, a packaging exception occurs, verification fails, packaging times out, or packaging is completed normally, then the above-mentioned one packaging server ends the packaging sub-process, wherein, for the packaging exception, an exception log is generated.

[0093] It should be noted that the specific implementation of S202 to S206 in this embodiment is the same as the specific implementation of S202 to S206 in the aforementioned embodiment, and will not be repeated here.

[0094] In one embodiment, the process of packaging the target task data is as follows:

[0095] Prepare the packaging project, download the source code files of the floors to be packaged and the packaging templates stored locally on the packaging server from the cloud server, and merge the source code files of the floors to be packaged and the packaging models into a complete packaging project.

[0096] Syntax compliance detection checks whether the code written by ISV designers is compliant.

[0097] Execute packaging, enter the packaging project and execute the packaging command to generate the corresponding static resource dist directory.

[0098] Performance and security verification: Generate a preview link based on the packaged product to perform performance and security verification.

[0099] Upload packaged products: upload the core bundle js files in the dist directory to the cloud server for storage.

[0100] Save the packaging result and call the backend interface to save the entire packaging result.

[0101] Exit the packaging subprocess. If any of the above steps is abnormal or fails to pass the verification, the packaging times out, or the packaging is completed normally, the packaging subprocess will be exited. For abnormal situations, the backend interface will be called to record the abnormal log.

[0102] The above-mentioned packing timeout means that the time for the packing sub-thread to execute task data is greater than the maximum timeout time packTimeOut configured for the packing sub-thread.

[0103] It should be noted that there are currently multiple packaging sub-processes in the packaging server. As long as the number of ongoing packaging sub-processes does not exceed the configured maximum number of packaging sub-processes maxPackingNum, new packaging sub-processes can continue to be created to process task data. After each packaging sub-process completes the task data, it is destroyed and will not be reused.

[0104] In the disclosed embodiment, abnormal situations and normal situations in the process of packaging target task data are set to ensure smooth execution of the packaging operation.

[0105] Figure 5 FIG. 2 shows another flow chart of a distributed multi-task packaging method provided by an embodiment of the present disclosure. Figure 5 As shown, after the above-mentioned one packaging server in the above-mentioned S208 ends the packaging sub-process, the distributed multi-task packaging method provided by the embodiment of the present disclosure further includes:

[0106] S210. The packaging server checks whether there is any unexecuted task data in the redis cache task queue. If so, if there is a packaging server whose number of tasks in the packaging is less than the preset packaging number threshold, the operation of taking out the target task data in the waiting state in the redis cache task queue based on the preset rules is performed until all the task data in the redis cache task queue are processed.

[0107] In one embodiment, when the redis cache task queue takes out a task data in a waiting state according to a preset rule, the task data in the redis cache task can be directly deleted. After the packaging subprocess completes the packaging operation of a task data, the packaging server queries whether there is unexecuted task data in the redis cache task queue. If there is, the task data continues to be taken out; if not, the packaging ends.

[0108] In one embodiment, when the redis cache task queue takes out a task data in a waiting state according to a preset rule, the task state of the redis cache task can also be modified. After the packaging subprocess completes the packaging operation of a task data, the packaging server queries the task state of each task data in the redis cache task queue. If there is a task data in a waiting state, the task data continues to be taken out; if all task data are being packaged or completed, the packaging ends.

[0109] In order to deepen the understanding of the implementation process of the present disclosure, the following Figure 6 Provide detailed explanation.

[0110] like Figure 6 As shown, the distributed multi-task packaging method of this example includes the following processes:

[0111] S601: Prepare task data.

[0112] The user can click the packaging button on the display page of the terminal device to request the packaging interface and send a packaging request, which is distributed to the corresponding packaging server through the load balancer. The packaging server prepares the packaging task data according to the packaging request. It should be noted that this process includes uploading the floor source code file to be packaged to the cloud server.

[0113] S602: Add the task data required for the packaging request to the global redis cache task queue allWaitList.

[0114] Specifically, a distributed lock (or the first distributed lock) is added, the task data is added to the end of the redis cache task queue allWaitList, and the distributed lock (or the first distributed lock) is released.

[0115] S603. Determine whether the number of tasks in the current packaging server (curPackingList.size()) is greater than or equal to the preset packaging number threshold. If so, wait. If not, it means that there is still task data in the waiting state in the redis cache task queue allWaitList, and execute S604.

[0116] It should be noted that when the number of tasks in the packaging of the current packaging server is greater than or equal to the preset packaging number threshold, the packaging operation of the current packaging server is waited until the task data in the current packaging server is packaged, so that the number of tasks in the packaging of the current packaging server is less than the preset packaging number threshold.

[0117] S604: Pack the task data in the waiting state, generate a packing result, upload the packing result to the cloud server, and the packing task is completed.

[0118] Specifically, the above S604 packages the target task data, generates a packaging result, uploads the packaging result to the cloud server, and the packaging task is completed, which specifically includes:

[0119] S6041. Pack the next task data and add a distributed lock (also the second distributed lock mentioned above).

[0120] S6042. Take out a task data in the waiting state from the head of the allWaitList queue.

[0121] S6043: Add the task data in the waiting state to the current packaging server packaging list cutPackingList.

[0122] S6044. Create a packaging subprocess and release the distributed lock (here is the second distributed lock).

[0123] S6045. Execute packaging.

[0124] S6046, packaging is completed, check whether there is unexecuted task data in the redis cache task queue, if so, execute S603 until all task data in the redis cache task queue are processed.

[0125] It should be noted that the process of the packing sub-thread packing the target task data has been disclosed in the aforementioned embodiment and will not be repeated here.

[0126] Based on the same inventive concept, the embodiments of the present disclosure also provide a distributed multi-task packaging system, such as the following embodiments. Since the principle of solving the problem in the system embodiment is similar to that in the above method embodiment, the implementation of the system embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0127] Figure 7 FIG. 2 is a schematic diagram of a distributed multi-task packaging system provided by an embodiment of the present disclosure. Figure 7 As shown, the distributed multi-task packaging system of this embodiment includes a load balancer 101 and multiple packaging servers 102. Figure 7 The packaging servers in the package are 1 to n, where n is a positive integer.

[0128] The load balancer 101 is used to receive the packaging request and distribute the packaging request to multiple packaging servers 102;

[0129] Each packaging server 102 is used to save the task data required for the packaging request to the distributed database redis cache task queue;

[0130] Among them, if there is a packaging server 102 whose number of tasks in packaging is less than the preset packaging number threshold, then the above-mentioned packaging server 102 takes out the target task data in the waiting state in the redis cache task queue based on the preset rules, creates a packaging sub-process, passes the target task data to the packaging sub-process, and the packaging sub-process executes packaging on the target task data to obtain the packaging result, and uploads the packaging result to the cloud server.

[0131] It should be noted that redis cache data is operated in a distributed lock manner.

[0132] In one embodiment, each packaging server 102 is used to add a first distributed lock; add the task data to the end of the redis cache task queue; and release the first distributed lock.

[0133] In one embodiment, the above-mentioned packaging server 102 is also used to add a second distributed lock before the above-mentioned packaging server 102 takes out the target task data in a waiting state in the redis cache task queue based on preset rules; after the above-mentioned packaging server 102 takes out the target task data in a waiting state in the redis cache task queue based on preset rules, the target task data is added to the task list of a packaging server 102, and the task list of a packaging server 102 is maintained in the redis cache; after creating the packaging subprocess, the second distributed lock is released.

[0134] In one embodiment, each packaging server 102 is used to upload the floor source code file of the packaging request to the cloud server, so that the packaging sub-process performs performance safety detection on the floor source code file.

[0135] In one embodiment, the packaging server 102 is used to terminate the packaging subprocess if a packaging exception, verification failure, packaging timeout or normal packaging completion occurs during the packaging of task data, wherein an exception log is generated for the packaging exception.

[0136] In one embodiment, the above-mentioned packaging server 102 is used to check whether there is any unexecuted task data in the redis cache task queue after the above-mentioned packaging server 102 ends the packaging sub-process. If so, it executes the operation of taking out the target task data in the waiting state in the redis cache task queue based on the preset rules if the number of packaged tasks of a packaging server is less than the preset packaging number threshold until all task data in the redis cache task queue are processed.

[0137] In the disclosed embodiment, the load balancer receives a packaging request and distributes the packaging request to multiple packaging servers. Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue; if there is a packaging server whose number of packaged tasks is less than a preset packaging number threshold, the above-mentioned packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rule, creates a packaging sub-process, and passes the target task data to the packaging sub-process; the packaging sub-process performs packaging on the target task data, obtains the packaging result, and uploads the packaging result to the cloud server, so that the packaging task data is persistently stored. When the packaging server crashes or restarts, the task queue data will not be lost. All task data are managed using a global redis cache task queue, which can be evenly distributed to all packaging server resources, thereby improving packaging efficiency.

[0138] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0139] Refer to the following Figure 8 An electronic device 800 according to this embodiment of the present invention will be described. Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0141] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 The method shown in is applied to a distributed multi-task packaging system, which includes a load balancer and multiple packaging servers, including: the load balancer receives a packaging request and distributes the packaging request to multiple packaging servers; each packaging server saves the task data required for the packaging request to a distributed database redis cache task queue; if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, then the above-mentioned packaging server takes out the target task data in the waiting state in the redis cache task queue based on a preset rule, creates a packaging sub-process, and passes the target task data to the packaging sub-process; the packaging sub-process executes packaging on the target task data, obtains a packaging result, and uploads the packaging result to a cloud server.

[0142] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0143] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0144] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the system, and / or may communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the system may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 860. Figure 8 As shown, the network adapter 860 communicates with other modules of the electronic device 800 via the bus 830. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0147] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. In an exemplary embodiment of the present disclosure, a computer program product is also provided, which includes a computer program or a computer instruction, and the computer program or the computer instruction is loaded and executed by a processor to enable the computer to implement the steps of the method disclosed in the above embodiment.

[0148] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0151] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0152] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0153] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A distributed multi-task packaging method, characterized in that: Applied to a distributed multi-task packaging system, the distributed multi-task packaging system includes a load balancer and multiple packaging servers, and the method includes: The load balancer receives the packaging request and distributes the packaging request to the multiple packaging servers; Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue; If there is a packaging server whose number of tasks in packaging is less than the preset packaging number threshold, the packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rules, creates a packaging sub-process, passes the target task data to the packaging sub-process, and the packaging sub-process executes packaging on the target task data to obtain a packaging result, and uploads the packaging result to the cloud server.

2. The distributed multi-task packaging method according to claim 1, characterized in that: The redis cache data is operated in a distributed lock manner.

3. The distributed multi-task packaging method according to claim 2, characterized in that: Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue, including: Add the first distributed lock; Add the task data to the end of the redis cache task queue; Release the first distributed lock.

4. The distributed multi-task packaging method according to claim 2, characterized in that: Before the packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rule, the method further includes: Add a second distributed lock; After the packaging server retrieves the target task data in the waiting state in the redis cache task queue based on a preset rule, the method further includes: Adding the target task data to a task list of the packaging server, wherein the task list of the packaging server is maintained in the redis cache; After creating the packaging subprocess, the method further includes: The second distributed lock is released.

5. The distributed multi-task packaging method according to claim 1, characterized in that: Each packaging server saves the task data required for the packaging request to the distributed database redis cache task queue, including: Each packaging server uploads the floor source code file of the packaging request to the cloud server, so that the packaging sub-process performs performance safety detection on the floor source code file.

6. The distributed multi-task packaging method according to claim 1, characterized in that: The method further comprises: If during the process of packaging the target task data, a packaging exception occurs, verification fails, packaging times out, or packaging is completed normally, then the one packaging server ends the packaging subprocess, wherein an exception log is generated for the packaging exception.

7. The distributed multi-task packaging method according to claim 6, characterized in that: After the one packaging server ends the packaging sub-process, the method further includes: The one packaging server checks whether there is any unexecuted task data in the redis cache task queue. If so, the one packaging server performs the operation of taking out the target task data in the waiting state in the redis cache task queue based on the preset rules until all the task data in the redis cache task queue are processed.

8. A distributed multi-task packaging system, characterized in that: The distributed multi-task packaging system includes a load balancer and multiple packaging servers, wherein: The load balancer is used to receive the packaging request and distribute the packaging request to the multiple packaging servers; Each packaging server is used to save the task data required for the packaging request to the distributed database redis cache task queue; Among them, if there is a packaging server whose number of tasks in packaging is less than a preset packaging number threshold, the packaging server takes out the target task data in the waiting state in the redis cache task queue based on the preset rules, creates a packaging sub-process, and passes the target task data to the packaging sub-process. The packaging sub-process executes packaging on the target task data, obtains the packaging result, and uploads the packaging result to the cloud server.

9. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the distributed multi-task packaging method as described in any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed multi-task packaging method according to any one of claims 1 to 7 is implemented.

Images