Data transfer method and apparatus in distributed cluster

Abstract

The application discloses a data transmission method and device in a distributed cluster. The method comprises the following steps: when a processing request of a target task in the distributed cluster is received, structural information of the target task is acquired; when the target task is processed according to the structural information, to-be-transmitted data generated in a processing process of a current processed job in the target task is acquired; a target transmission object of the to-be-transmitted data is determined according to a dependency relationship between the current processed job and other unprocessed jobs in the target task; and the to-be-transmitted data is transmitted to the target transmission object. The application solves the technical problem that data generated in a job processing process in a task under a distributed architecture cannot be automatically transmitted according to a dependency relationship of jobs in the task in the prior art.

Description

Technical Field

[0001] This invention relates to the field of task processing technology under distributed architecture, and more specifically, to a data transmission method and apparatus in a distributed cluster. Background Technology

[0002] Driven by various complex business scenarios, a method has emerged that can transform complex business processes into models within code. This method orchestrates various stages of the business into a single task, where each task represents a specific business scenario. Within a task, a job is a specific execution logic. A centralized scheduling service schedules tasks, and jobs within a task are sent to execution services for execution. However, if the execution results or data generated during the execution of a specific job are to be retrieved or processed in a subsequent specific job, it involves different cluster services within a distributed architecture. A method is needed to transmit the identifiers of the generated data so that subsequent services can identify and retrieve the corresponding data for processing. This multi-level parameter passing design scheme within a task under a distributed architecture is specifically designed to address this scenario.

[0003] However, this approach lacks flexibility in handling parameter transfer between multi-level jobs. Parameters generated by one job need to be passed to the next job, requiring the ability to locate a specific job within the complex structure of the task. Furthermore, during the cross-level parameter transfer process, if the job that generated the parameters has already completed execution, parameter loss may occur if a service outage occurs during the cross-level process.

[0004] There is currently no effective solution to the problem that the aforementioned technologies cannot automatically transfer data generated during job processing in a distributed architecture based on the job dependencies within the task. Summary of the Invention

[0005] This invention provides a data transfer method and apparatus in a distributed cluster, which at least solves the technical problem in related technologies that data generated during job processing in a distributed architecture cannot be automatically transferred according to the job dependencies in the task.

[0006] According to one aspect of the present invention, a data transfer method in a distributed cluster is provided, comprising: upon receiving a processing request for a target task in the distributed cluster, obtaining structural information of the target task, wherein the target task is a pending task generated by multiple processes and jobs under each of the multiple processes; while processing the target task according to the structural information, obtaining pending data generated during the processing of currently processed jobs in the target task; determining a target transfer object for the pending data based on the dependency relationship between the currently processed jobs and other unprocessed jobs in the target task; and transferring the pending data to the target transfer object.

[0007] Optionally, obtaining the structural information of the target task includes: determining a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among the multiple processes, and the first execution order represents the execution order of the multiple processes; determining a second dependency relationship and a second execution order among the jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among the jobs under each process, and the second execution order represents the execution order of the jobs under each process; and determining the structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

[0008] Optionally, obtaining the data to be transmitted generated during the processing of the currently processed job in the target task includes: receiving the job processing information returned by the currently processed job; parsing the job processing information to obtain the data to be transmitted.

[0009] Optionally, after obtaining the data to be transmitted generated during the processing of the currently processed job in the target task, the data transmission method in the distributed cluster further includes: storing the data to be transmitted to a local storage medium and a scheduling service memory respectively, wherein the local storage medium is used to transmit parameters when the target transmission object and the currently processed job are not on the same scheduling server under abnormal circumstances, and the scheduling service memory is used to transmit parameters under normal circumstances.

[0010] Optionally, storing the data to be transmitted in both local storage and scheduling service memory includes: obtaining the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; obtaining the ownership information of the target transmission object; determining the storage path information of the data to be transmitted; adding the task identifier, the ownership information, and the storage path information to the record information corresponding to the data to be transmitted to obtain a data record of the data to be transmitted, wherein the data record includes a fixed prefix, the fixed prefix being used to identify the data of the currently processed job; and storing the data record in both local storage and scheduling service memory.

[0011] Optionally, after transmitting the data to be transmitted to the target transmission object, the data transmission method in the distributed cluster further includes: determining whether the target job corresponding to the target transmission object has successfully received the data record, and obtaining a determination result; when the determination result indicates that the target job has successfully received the data record, loading the data to be transmitted from the scheduling service memory; and when the determination result indicates that the target job has not successfully received the data record, loading the data to be transmitted from the local storage medium.

[0012] Optionally, the data transmission method in the distributed cluster further includes: detecting the data record to obtain a detection result; when the detection result indicates that the data record includes the fixed prefix, if the data in the data record is consistent with the current job to be executed, it is determined that the target job has successfully received the data record.

[0013] Optionally, loading the data to be transmitted from the local storage medium includes: loading the data to be transmitted from the local storage medium when it is determined that the scheduling service executing the target task has encountered an anomaly and the target task has been restarted.

[0014] According to another aspect of the present invention, a data transmission apparatus in a distributed cluster is also provided, comprising: a first acquisition unit, configured to acquire structural information of the target task when receiving a processing request for a target task in the distributed cluster, wherein the target task is a pending task generated by multiple processes and jobs under each of the multiple processes; a second acquisition unit, configured to acquire pending data generated during the processing of currently processed jobs in the target task when processing the target task according to the structural information; a first determination unit, configured to determine a target transmission object of the pending data based on the dependency relationship between the currently processed jobs and other unprocessed jobs in the target task; and a transmission unit, configured to transmit the pending data to the target transmission object.

[0015] Optionally, the first acquisition unit includes: a first determining module, configured to determine a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among the multiple processes, and the first execution order represents the execution order of the multiple processes; a second determining module, configured to determine a second dependency relationship and a second execution order among the jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among the jobs under each process, and the second execution order represents the execution order of the jobs under each process; and a third determining module, configured to determine the structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

[0016] Optionally, the second acquisition unit includes: a receiving module, configured to receive job processing information returned by the currently processed job; and a parsing module, configured to parse the job processing information to obtain the data to be transmitted.

[0017] Optionally, the data transmission device in the distributed cluster further includes: a storage unit, used to store the data to be transmitted to a local storage medium and a scheduling service memory respectively after obtaining the data to be transmitted generated during the processing of the currently processed job in the target task, wherein the local storage medium is used to transmit parameters when the target transmission object and the currently processed job are not on the same scheduling server under abnormal circumstances, and the scheduling service memory is used to transmit parameters under normal circumstances.

[0018] Optionally, the storage unit includes: a first acquisition module, configured to acquire the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; a second acquisition module, configured to acquire the ownership information of the target transmission object; a fourth determination module, configured to determine the storage path information of the data to be transmitted; a third acquisition module, configured to add the task identifier, the ownership information, and the storage path information to the record information corresponding to the data to be transmitted, to obtain the data record of the data to be transmitted, wherein the data record includes a fixed prefix, the fixed prefix being used to identify the data of the currently processed job; and a storage module, configured to store the data record in the local storage medium and the scheduling service memory respectively.

[0019] Optionally, the data transmission device in the distributed cluster further includes: a judgment unit, configured to, after transmitting the data to be transmitted to the target transmission object, determine whether the target job corresponding to the target transmission object has successfully received the data record, and obtain a judgment result; a first loading unit, configured to, when the judgment result indicates that the target job has successfully received the data record, load the data to be transmitted from the scheduling service memory; and a second loading unit, configured to, when the judgment result indicates that the target job has not successfully received the data record, load the data to be transmitted from the local storage medium.

[0020] Optionally, the data transmission device in the distributed cluster further includes: a detection unit, used to detect the data record and obtain a detection result; and a second determination unit, used to determine that the target job has successfully received the data record when the detection result indicates that the data record includes the fixed prefix and the data in the data record is consistent with the current job to be executed.

[0021] Optionally, the second loading unit includes a loading module, configured to load the data to be transmitted from the local storage medium when it is determined that the scheduling service executing the target task has encountered an anomaly and the target task has been restarted.

[0022] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the data transfer method in a distributed cluster as described in any of the preceding embodiments.

[0023] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a program, wherein the program, when running, executes the data transfer method in a distributed cluster as described in any of the preceding embodiments.

[0024] According to another aspect of the present invention, a data transfer system in a distributed cluster is also provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing the data transfer method in a distributed cluster as described in any one of the preceding embodiments.

[0025] In this embodiment of the invention, when a processing request for a target task is received in a distributed cluster, the structural information of the target task can be obtained; when processing the target task according to the structural information, the data to be transmitted generated during the processing of the currently processed job in the target task can be obtained; the target transmission object of the data to be transmitted can be determined according to the dependency relationship between the currently processed job and other unprocessed jobs in the target task; and the data to be transmitted can be transmitted to the target transmission object. Through this data transmission method in the distributed cluster, it is possible to determine the transmission object of the data generated during the execution of the currently processed job based on the relationship between the currently processed job and other unprocessed jobs in the target task, thereby automatically transmitting the data to be transmitted to the determined transmission object, thereby improving the efficiency of task processing and solving the technical problem in related technologies that it is impossible to automatically transmit the data generated during the processing of jobs in a distributed architecture according to the dependency relationship of jobs in the task. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 This is a flowchart of a data transfer method in a distributed cluster according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the internal structure and execution order of a task according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart of an optional data transfer method in a distributed cluster according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of a data transmission device in a distributed cluster according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] For ease of description, the following explains some of the nouns or terms used in the embodiments of the present invention:

[0034] Execution order: The order is composed of certain parallel and serial relationships.

[0035] Dependency: Among some nodes executing in parallel, there exists a relationship where the completion of one node is a prerequisite for the start of another node. This is called the execution of the other node depending on the first node.

[0036] Task: A task is a logical collection of multiple processes and jobs arranged in a certain execution order and with dependencies. A process is a logical concept, consisting of multiple processes and jobs; a job is a specific execution concept, representing a particular execution logic.

[0037] Cross-level transfer: Cross-level transfer refers to the process where, after a job in a certain process is completed, the parameters generated need to be transferred to a job in another process that is parallel to this process at the same level, or to a job in a process that is sequentially connected to the next higher-level process.

[0038] As described in the background section, related technologies cannot automatically transfer data generated during job processing in a distributed architecture based on the job dependencies within the task. To address this issue, embodiments of the present invention provide a data transfer method in a distributed cluster. The following description, in conjunction with specific embodiments, illustrates this data transfer method in a distributed cluster.

[0039] According to an embodiment of the present invention, a method embodiment for data transfer in a distributed cluster is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 1 This is a flowchart of a data transfer method in a distributed cluster according to an embodiment of the present invention, such as... Figure 1 As shown, the data transfer method in this distributed cluster includes the following steps:

[0041] Step S102: When a processing request for a target task is received in the distributed cluster, the structure information of the target task is obtained. The target task consists of multiple processes and jobs generated by each process.

[0042] Optionally, in various complex business scenarios, methods have emerged to transform complex business processes into models within code. Specifically, various stages in the business process can be orchestrated into a single task, where each task represents a specific business scenario. Within each task, there is a job, i.e., a specific execution logic. A centralized scheduling service schedules the tasks, and the jobs within each task are sent by the scheduling service to the execution service for execution.

[0043] Therefore, the target task in this embodiment of the invention is also a task arranged from various links in the business, which represents a specific business scenario.

[0044] Optionally, the above structural information may refer to the dependencies and execution order between processes in a task, as well as the dependencies and execution order of jobs under a process.

[0045] Step S104: When processing the target task according to the structure information, obtain the data to be transmitted generated during the processing of the currently processed job in the target task.

[0046] Optionally, the jobs in the aforementioned target tasks generate data during execution, and this data needs to be passed to the next job to be executed. Here, the currently processed job generates data during execution and passes this data to the scheduling service.

[0047] Step S106: Determine the target object for the data to be transferred based on the dependency relationship between the currently processed jobs and other unprocessed jobs in the target task.

[0048] Optionally, after obtaining the data to be transferred, it is necessary to determine which job the data should be transferred to. This can be done by considering the dependencies between currently processed jobs and other unprocessed jobs in the target task, thus determining which job should be executed next and using that job as the target transfer object.

[0049] Step S108: Transfer the data to be transferred to the target object.

[0050] As can be seen from the above, in this embodiment of the invention, when a processing request for a target task in a distributed cluster is received, the structural information of the target task can be obtained; when processing the target task according to the structural information, the data to be transmitted generated during the processing of the currently processed job in the target task can be obtained; the target transmission object of the data to be transmitted can be determined according to the dependency relationship between the currently processed job and other unprocessed jobs in the target task; and the data to be transmitted can be transmitted to the target transmission object. This realizes that the transmission object of the data generated during the execution of the currently processed job can be determined according to the relationship between the currently processed job and other unprocessed jobs in the target task, thereby automatically transmitting the data to be transmitted to the determined transmission object, thus improving the efficiency of task processing.

[0051] Therefore, the technical solution provided by the embodiments of the present invention solves the technical problem in the related art that the data generated during the job processing in a distributed architecture cannot be automatically transferred according to the job dependency relationship in the task.

[0052] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the data transmission method in the distributed cluster of the present invention will be described in detail below with reference to specific embodiments.

[0053] According to the above embodiments of the present invention, obtaining the structural information of a target task may include: determining a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among multiple processes, and the first execution order represents the execution order of multiple processes; determining a second dependency relationship and a second execution order among the jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among the jobs under each process, and the second execution order represents the execution order of the jobs under each process; and determining structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

[0054] Figure 2 This is a schematic diagram illustrating the internal structure and execution order of a task according to an embodiment of the present invention, as shown below. Figure 2 As shown, under the task, processes 1, 2, and 3 are executed sequentially. Process 1 contains only job 1. Process 2 contains processes 2.1 and 2.2, with job 2.1 executed in parallel. Process 2.1 contains only job 2.1.1, process 2.2 contains only job 2.2.1, and process 3 contains only job 3.1. It should be noted that... Figure 2 This merely illustrates one task structure; tasks can be arranged in other forms depending on the business scenario, meaning tasks can exist in other forms. The task structure is not specifically limited in this embodiment of the invention and can be determined based on the specific business scenario.

[0055] According to the above embodiments of the present invention, obtaining the data to be transmitted generated during the processing of the currently processed job in the target task includes: receiving the job processing information returned by the currently processed job; parsing the job processing information to obtain the data to be transmitted.

[0056] In this embodiment, after a currently processed job is completed, the data generated during its execution is processed into job processing information and sent back to the scheduling service. Upon receiving the job processing information, the scheduling service parses it to obtain the data to be transmitted. The job processing information can be presented in the form of a compressed package, which may include information about which process the data to be transmitted was generated during its execution.

[0057] According to the above embodiments of the present invention, after obtaining the data to be transmitted generated during the processing of the currently processed job in the target task, the data transmission method in the distributed cluster may further include: storing the data to be transmitted to a local storage medium and a scheduling service memory respectively, wherein the local storage medium is used to transmit parameters when the target transmission object and the currently processed job are not on the same scheduling server under abnormal circumstances, and the scheduling service memory is used to transmit parameters under normal circumstances.

[0058] In this embodiment, in order to ensure data security, the data to be transmitted is simultaneously stored in local storage media, such as a local database and the scheduling service memory.

[0059] In normal circumstances, data is transmitted through the scheduling service memory, which increases the transmission speed and reduces the pressure on the database. Database storage serves as a backup solution to ensure data transmission in abnormal situations.

[0060] According to the above embodiments of the present invention, storing the data to be transmitted in a local storage medium and a scheduling service memory respectively includes: obtaining the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; obtaining the ownership information of the target transmission object; determining the storage path information of the data to be transmitted; adding the task identifier, ownership information and storage path information to the record information corresponding to the data to be transmitted to obtain the data record of the data to be transmitted, wherein the data record includes a fixed prefix, the fixed prefix is ​​used to identify the data of the currently processed job; and storing the data record in a local storage medium and a scheduling service memory respectively.

[0061] Figure 3 This is a flowchart of an optional data transfer method in a distributed cluster according to an embodiment of the present invention, such as... Figure 3As shown, after the currently processed job 1.1 has finished executing, the scheduling service receives the data test / 000 / file1 returned by job 1.1, along with the information that it needs to be passed to job 3.1 under process 3. The scheduler then stores the data in memory in key:value format, where the key is: fixed prefix - unique task identifier - process 2 - job 2.1, and the value is test / 000 / file1. Before executing each job, it checks if a key containing the fixed prefix exists in memory. If it does, it checks if the entire content of the key matches the currently executing job. If a match is found, the value is sent to the execution service along with the job's schedule.

[0062] The above processing methods represent data transmission methods under normal scenarios. For processing methods in abnormal scenarios, such as... Figure 3 As shown, after job 1.1 is completed and the scheduling service saves the key:value pair, when scheduling to process 2.2, the scheduling service executing the task crashes, and the data in memory is lost. When the task is failovered, a special identifier will be added when the task is restarted. This identifier will cause all jobs under process 2.2 to use the information of the current job to access the database to try to obtain parameters. Before job 2.1 is executed, the database will find the data belonging to the current job, and the parameter transfer from job 1.1 to job 2.1 will be successful.

[0063] The values ​​of the jobs and parameters to be received are stored in memory using a fixed prefix-task unique identifier-process name-job name. By using a fixed prefix, the CPU pressure is reduced when no parameters are passed, and by using a task unique identifier, the confusion of parameter passing between multiple parallel tasks is avoided.

[0064] According to the above embodiments of the present invention, after the data to be transmitted is transmitted to the target transmission object, the data transmission method in the distributed cluster may further include: determining whether the target job corresponding to the target transmission object has successfully received the data record, and obtaining a determination result; when the determination result indicates that the target job has successfully received the data record, loading the data to be transmitted from the scheduling service memory; when the determination result indicates that the target job has not successfully received the data record, loading the data to be transmitted from the local storage medium.

[0065] It should be noted that in this embodiment of the invention, data transmission is divided into two scenarios: a normal scenario, in which the scheduling server is uninterrupted and does not crash during the entire task execution process, and the task ends normally; and an abnormal scenario, in which the scheduling service node executing the task crashes during the task execution process.

[0066] In normal processing scenarios, when the target job corresponding to the target object successfully receives the data record, it will prioritize loading the data to be transmitted from the scheduling service memory, reducing access to the local database and thus lowering the pressure on the local database. Conversely, it will load the data to be transmitted from the local database only if the target job fails to receive the data record.

[0067] According to the above embodiments of the present invention, the data transmission method in the distributed cluster may further include: detecting the data record and obtaining the detection result; when the detection result indicates that the data record includes a fixed prefix, if the data in the data record is consistent with the current job to be executed, determining that the target job has successfully received the data record.

[0068] That is, in this embodiment of the invention, before executing each job, it is determined whether there is a key in memory containing a fixed prefix. If there is a fixed prefix, it is then determined whether the entire content of the key matches the currently executed job. If they match, the value is sent to the execution service along with the job scheduling.

[0069] According to the above embodiments of the present invention, loading data to be transmitted from a local storage medium includes: loading data to be transmitted from a local storage medium when it is determined that the scheduling service executing the target task has encountered an anomaly and the target task has been restarted.

[0070] In this embodiment, if it is determined that the scheduling service executing the target task is abnormal, but the target task is subsequently restarted, the data in the scheduling service memory is lost due to the scheduling service crashing. In this case, a failover of the task is required, and the data to be transferred will be loaded from the local storage medium.

[0071] As can be seen from the above, in the technical solution provided by the embodiments of the present invention, the values ​​of the jobs and parameters to be received are stored in memory using a fixed prefix-task unique identifier-process name-job name. Using the fixed prefix reduces CPU pressure when no parameters are passed, and using the task unique identifier avoids confusion in parameter passing between multiple parallel tasks. Parameter passing is normally done through memory, reducing database pressure. Database storage serves as a fallback solution to ensure parameter passing in abnormal situations. This effectively overcomes the following drawbacks of the prior art: 1) It cannot flexibly judge the passing between multi-level jobs; parameters generated by one job need to be passed to the next job, requiring the ability to locate a specific job within the complex structure of the task; 2) During cross-level transfer of job parameters, if the job that generated the parameters has already completed execution, parameter loss may occur if a service crash occurs during the cross-level process; 3) Normal and abnormal scenarios are not separated, increasing database pressure in normal scenarios. This solves the problem of how to ensure that parameters generated in any job in a complex task with execution order and dependencies can be received by a specified job or, if not specified, by the next sequentially executed job.

[0072] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0074] According to embodiments of the present invention, a data transmission apparatus for implementing the above-described data transmission method in a distributed cluster is also provided. Figure 4 This is a schematic diagram of a data transmission device in a distributed cluster according to an embodiment of the present invention, such as... Figure 4As shown, the device includes: a first acquisition unit 41, a second acquisition unit 43, a first determination unit 45, and a transmission unit 47. The data transmission device in this distributed cluster will be described below.

[0075] The first acquisition unit 41 is used to acquire the structural information of the target task when it receives a processing request for the target task in the distributed cluster. The target task is a task to be processed generated by multiple processes and jobs under each process.

[0076] The second acquisition unit 43 is used to acquire the data to be transmitted generated during the processing of the currently processed job in the target task when processing the target task according to the structure information.

[0077] The first determining unit 45 is used to determine the target object of the data to be transmitted based on the dependency relationship between the currently processed job and other unprocessed jobs in the target task.

[0078] The transmission unit 47 is used to transmit the data to be transmitted to the target transmission object.

[0079] It should be noted here that the first acquisition unit 41, the second acquisition unit 43, the first determination unit 45 and the transmission unit 47 mentioned above correspond to steps S102 to S108 in the method embodiment. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above method embodiment.

[0080] As can be seen from the above, in the solution described in the above method embodiment of the present invention, the first acquisition unit can acquire the structural information of the target task when receiving the processing request of the target task in the distributed cluster. The target task is a task to be processed generated by multiple processes and jobs under each process. The second acquisition unit can acquire the data to be transmitted generated by the currently processed job in the target task during the processing of the target task when processing the target task according to the structural information. The first determination unit can determine the target transmission object of the data to be transmitted based on the dependency relationship between the currently processed job and other unprocessed jobs in the target task. The transmission unit can then transmit the data to be transmitted to the target transmission object. This achieves the goal of determining the transmission object of the data generated when the currently processed job is executed based on the relationship between the currently processed job and other unprocessed jobs in the target task, thereby automatically transmitting the data to be transmitted to the determined transmission object and improving the efficiency of task processing.

[0081] Therefore, the technical solution provided by the embodiments of the present invention solves the technical problem in the related art that the data generated during the job processing in a distributed architecture cannot be automatically transferred according to the job dependency relationship in the task.

[0082] Optionally, the first acquisition unit includes: a first determining module, configured to determine a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among multiple processes, and the first execution order represents the execution order of multiple processes; a second determining module, configured to determine a second dependency relationship and a second execution order among the jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among the jobs under each process, and the second execution order represents the execution order of the jobs under each process; and a third determining module, configured to determine structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

[0083] Optionally, the second acquisition unit includes: a receiving module for receiving job processing information returned by the currently processed job; and a parsing module for parsing the job processing information to obtain the data to be transmitted.

[0084] Optionally, the data transmission device in the distributed cluster further includes: a storage unit, used to store the data to be transmitted to a local storage medium and a scheduling service memory respectively after obtaining the data to be transmitted generated during the processing of the currently processed job in the target task. The local storage medium is used to transmit parameters when the target transmission object and the currently processed job are not on the same scheduling server in abnormal circumstances, and the scheduling service memory is used to transmit parameters under normal circumstances.

[0085] Optionally, the storage unit includes: a first acquisition module for acquiring the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; a second acquisition module for acquiring the ownership information of the target transfer object; a fourth determination module for determining the storage path information of the data to be transferred; a third acquisition module for adding the task identifier, ownership information, and storage path information to the record information corresponding to the data to be transferred to obtain the data record of the data to be transferred, wherein the data record includes a fixed prefix, the fixed prefix being used to identify the data of the currently processed job; and a storage module for storing the data record to the local storage medium and the scheduling service memory respectively.

[0086] Optionally, the data transmission device in the distributed cluster further includes: a judgment unit, used to determine whether the target job corresponding to the target transmission object has successfully received the data record after the data to be transmitted is transmitted to the target transmission object, and to obtain a judgment result; a first loading unit, used to load the data to be transmitted from the scheduling service memory when the judgment result indicates that the target job has successfully received the data record; and a second loading unit, used to load the data to be transmitted from the local storage medium when the judgment result indicates that the target job has not successfully received the data record.

[0087] Optionally, the data transmission device in the distributed cluster further includes: a detection unit for detecting data records and obtaining detection results; and a second determination unit for determining that the target job has successfully received the data record when the detection result indicates that the data record includes a fixed prefix and the data in the data record is consistent with the current job to be executed.

[0088] Optionally, the second loading unit includes a loading module for loading data to be transferred from local storage medium when it is determined that the scheduling service for executing the target task has encountered an error and the target task has been restarted.

[0089] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the data transfer method in a distributed cluster of any of the above.

[0090] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any communication device in a group of communication devices.

[0091] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: upon receiving a processing request for a target task in a distributed cluster, obtaining the structure information of the target task, wherein the target task is a pending task generated by multiple processes and jobs under each of the multiple processes; when processing the target task according to the structure information, obtaining the data to be transferred generated by the currently processed jobs in the target task during the processing; determining the target transfer object of the data to be transferred based on the dependency relationship between the currently processed jobs and other unprocessed jobs in the target task; and transferring the data to be transferred to the target transfer object.

[0092] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining a first dependency relationship and a first execution order among multiple processes in a target task, wherein the first dependency relationship represents the association relationship among multiple processes, and the first execution order represents the execution order of multiple processes; determining a second dependency relationship and a second execution order among jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among jobs under each process, and the second execution order represents the execution order of jobs under each process; and determining structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

[0093] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: receiving job processing information returned by the currently processed job; parsing the job processing information to obtain data to be transmitted.

[0094] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: storing the data to be transmitted to a local storage medium and a scheduling service memory respectively, wherein the local storage medium is used to transmit parameters when the target object to be transmitted and the currently processed job are not on the same scheduling server under abnormal circumstances, and the scheduling service memory is used to transmit parameters under normal circumstances.

[0095] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; obtaining the ownership information of the target delivery object; determining the storage path information of the data to be delivered; adding the task identifier, ownership information, and storage path information to the record information corresponding to the data to be delivered to obtain the data record of the data to be delivered, wherein the data record includes a fixed prefix, the fixed prefix is ​​used to identify the data of the currently processed job; and storing the data record to the local storage medium and the scheduling service memory respectively.

[0096] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining whether the target job corresponding to the target delivery object has successfully received the data record, and obtaining a determination result; when the determination result indicates that the target job has successfully received the data record, loading the data to be delivered from the scheduling service memory; when the determination result indicates that the target job has not successfully received the data record, loading the data to be delivered from the local storage medium.

[0097] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: detecting a data record and obtaining a detection result; when the detection result indicates that the data record includes a fixed prefix, if the data in the data record is consistent with the current job to be executed, determining that the target job has successfully received the data record.

[0098] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when it is determined that the scheduling service for executing the target task has encountered an anomaly and the target task has been restarted, loading the data to be transferred from the local storage medium.

[0099] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes the data transfer method in a distributed cluster described above during runtime.

[0100] According to another aspect of the present invention, a data transfer system in a distributed cluster is also provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a data transfer method in a distributed cluster for performing any of the above.

[0101] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0102] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0107] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0108] 1) The values ​​of the jobs and parameters to be received are stored in memory using a fixed prefix-task unique identifier-process name-job name. By using a fixed prefix, the CPU pressure is reduced when no parameters are passed. By using a task unique identifier, the confusion of parameter passing between multiple parallel tasks is avoided.

[0109] 2) Parameters are normally passed through memory to reduce the pressure on the database. Database storage serves as a backup solution to ensure parameter passing in abnormal situations.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data transfer method in a distributed cluster, characterized in that, include: When a processing request for a target task is received in a distributed cluster, the structural information of the target task is obtained, wherein the target task is a task to be processed generated by multiple processes and jobs under each of the multiple processes. When processing the target task according to the structure information, the data to be transmitted generated during the processing of the currently processed job in the target task is obtained; The target object for transmitting the data to be transmitted is determined based on the dependency relationship between the currently processed jobs and other unprocessed jobs in the target task. The data to be transmitted is transmitted to the target transmission object. Obtaining the structural information of the target task includes: Determine a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among the multiple processes, and the first execution order represents the execution order of the multiple processes; Determine a second dependency relationship and a second execution order among the jobs in each of the plurality of processes, wherein the second dependency relationship represents the association relationship between the jobs in each process, and the second execution order represents the execution order of the jobs in each process; The structural information is determined based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

2. The data transmission method in a distributed cluster according to claim 1, characterized in that, Obtain the data to be transmitted generated during the processing of the currently processed job in the target task, including: Receive the job processing information returned by the currently processed job; The job processing information is parsed to obtain the data to be transmitted.

3. The data transmission method in a distributed cluster according to any one of claims 1 to 2, characterized in that, After obtaining the data to be transmitted generated during the processing of the currently processed job in the target task, the method further includes: The data to be transmitted is stored in a local storage medium and a scheduling service memory, respectively. The local storage medium is used to transmit parameters when the target object to be transmitted is not on the same scheduling server as the currently processed job in an abnormal situation. The scheduling service memory is used to transmit parameters under normal circumstances.

4. The data transmission method in a distributed cluster according to claim 3, characterized in that, The data to be transmitted is stored in both local storage medium and scheduling service memory, including: Obtain the task identifier of the target task, wherein the task identifier is used to uniquely identify the target task; Obtain the ownership information of the target object; Determine the storage path information of the data to be transmitted; The task identifier, the attribution information, and the storage path information are added to the record information corresponding to the data to be transmitted to obtain the data record of the data to be transmitted. The data record includes a fixed prefix, which is used to identify the data of the currently processed job. The data records are stored in the local storage medium and the scheduling service memory, respectively.

5. The data transmission method in a distributed cluster according to claim 4, characterized in that, After transmitting the data to be transmitted to the target object, the process further includes: Determine whether the target job corresponding to the target transmission object has successfully received the data record, and obtain the determination result; When the determination result indicates that the target job has successfully received the data record, the data to be transmitted is loaded from the scheduling service memory. When the determination result indicates that the target job has not successfully received the data record, the data to be transmitted is loaded from the local storage medium.

6. The data transmission method in a distributed cluster according to claim 5, characterized in that, Also includes: The data records are inspected to obtain inspection results; When the detection result indicates that the data record includes the fixed prefix, if the data in the data record is consistent with the current job to be executed, it is determined that the target job has successfully received the data record.

7. The data transmission method in a distributed cluster according to claim 5 or 6, characterized in that, Loading the data to be transmitted from the local storage medium includes: When it is determined that the scheduling service executing the target task has encountered an error and the target task has been restarted, the data to be transmitted is loaded from the local storage medium.

8. A data transmission device in a distributed cluster, characterized in that, include: The first acquisition unit is used to acquire the structure information of the target task when it receives a processing request for the target task in the distributed cluster, wherein the target task is a task to be processed generated by multiple processes and jobs under each of the multiple processes. The second acquisition unit is used to acquire, when processing the target task according to the structure information, the data to be transmitted generated during the processing of the currently processed job in the target task; The first determining unit is used to determine the target object for transmitting the data to be transmitted based on the dependency relationship between the currently processed job and other unprocessed jobs in the target task. The transmission unit is used to transmit the data to be transmitted to the target transmission object. The first acquisition unit includes: a first determining module, configured to determine a first dependency relationship and a first execution order among multiple processes in the target task, wherein the first dependency relationship represents the association relationship among the multiple processes, and the first execution order represents the execution order of the multiple processes; a second determining module, configured to determine a second dependency relationship and a second execution order among the jobs under each of the multiple processes, wherein the second dependency relationship represents the association relationship among the jobs under each process, and the second execution order represents the execution order of the jobs under each process; and a third determining module, configured to determine the structural information based on the first dependency relationship, the first execution order, the second dependency relationship, and the second execution order.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the data transfer method in a distributed cluster as described in any one of claims 1 to 7.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data transfer method in a distributed cluster according to any one of claims 1 to 7.

11. A data transmission system in a distributed cluster, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing data transfer in a distributed cluster as described in any one of claims 1 to 7.

Images