Message deduplication method, device, computer equipment, storage medium and program product

Abstract

The present application relates to a message deduplication method, device, computer equipment, storage medium and program product. The above method includes: in the process of multiple computing nodes executing a target job, for each computing node, if an abnormal job step occurs in the target job, then a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step; determine the communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among multiple computing nodes, and there is a corresponding relationship between one communication node and multiple computing nodes; deduplicate the job step operation message through the communication node to generate an intermediate job step operation message. The present method can be used to deduplicate messages for a job scheduling system.

Description

Technical Field

[0001] The present application relates to the technical field of message processing, and in particular to a message deduplication method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] The job scheduling system is a system for monitoring and managing resources and jobs in a cluster. This job scheduling system includes multiple computing nodes and management nodes. When computing nodes are executing jobs, job steps in the job often have execution exceptions. Therefore, for these abnormal job steps (abnormal job steps for short), the computing nodes need to send a message to cancel the job steps to the management node, and then the management node executes the operation of canceling the abnormal job steps. For example, the job scheduling system includes the SLURM (Simple Linux Utility for Resource Management) job scheduling system.

[0003] However, the number of computing nodes in the job scheduling system is relatively large. If a large number of computing nodes send messages to the management node to cancel the job step, the management node will receive a large number of duplicate messages. Therefore, it is necessary to deduplicate the large number of duplicate messages received by the management node in the job scheduling system.

[0004] However, there is no message deduplication method suitable for job scheduling systems in traditional technologies. Summary of the invention

[0005] Based on this, it is necessary to provide a message deduplication method, device, computer equipment, storage medium and program product that can be applicable to a job scheduling system in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a message deduplication method. Applied to multiple computing nodes in a preset job scheduling system, the multiple computing nodes corresponding to target jobs; the method comprises:

[0007] During the process of the plurality of computing nodes executing the target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step;

[0008] Determine a communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes;

[0009] The job step operation message is deduplicated through the communication node to generate an intermediate job step operation message.

[0010] In the above-mentioned message deduplication method, first, when an abnormal job step occurs in the target job, the job step operation message of the computing node can be generated, so that the job step operation message including the duplicate message in the job scheduling system can be obtained. Secondly, at least one communication node is determined from multiple computing nodes, so that the communication node corresponding to each computing node can be determined. After that, for each computing node, the job step operation message of the computing node is sent to the communication node corresponding to the computing node, so that at least one common message list that can store the job step operation message can be established through the communication node corresponding to each computing node, and the job step operation message can be deduplicated based on at least one message list, so that the deduplicated intermediate job step operation message can be generated. Therefore, the message deduplication method of the present application can dedupe the job step operation message in the job scheduling system through the communication node corresponding to the computing node in the job scheduling system, and the message deduplication method of the present application is a message deduplication method that can be applied to the job scheduling system.

[0011] In one embodiment, determining the communication node corresponding to the computing node includes:

[0012] Determining a communication area width of the communication node according to the number of the plurality of computing nodes;

[0013] Dividing the plurality of computing nodes according to the width of the communication area to obtain a corresponding relationship between the communication nodes and the computing nodes corresponding to each of the communication nodes;

[0014] Based on the correspondence between each of the communication nodes and the computing nodes corresponding to each of the communication nodes, the communication node corresponding to the computing node is determined from the plurality of computing nodes.

[0015] In this embodiment, the communication area width of the communication node is determined according to the number of multiple computing nodes, so that the number of computing nodes included in a communication area can be determined more accurately; the multiple computing nodes are divided according to the communication area width, so that the communication area position of each computing node can be determined more accurately, so as to more accurately obtain the corresponding relationship between the communication node and the computing node corresponding to each communication node; based on the more accurate corresponding relationship between each communication node and the computing node corresponding to each communication node, the communication node corresponding to the computing node can be more accurately determined from the multiple computing nodes. This embodiment adopts the idea of ​​regional communication, and can select a computing node in the communication area as the communication node in each communication area based on the more accurate corresponding relationship between each communication node and the computing node corresponding to each communication node, so as to more accurately determine the communication node corresponding to each computing node.

[0016] In one embodiment, the step of removing duplicates from the job step operation message by the communication node to generate an intermediate job step operation message includes:

[0017] If the communication node receives the job step operation message, creating a first message list according to the identifier of the job step operation message through the communication node;

[0018] If the communication node receives the job step operation message again, the communication node deduplicates the job step operation message received again based on the job step operation message received again and the first message list to generate the intermediate job step operation message.

[0019] In this embodiment, if the communication node receives a job step operation message, the communication node creates a first message list according to the identifier of the job step operation message, and the first message list in the job scheduling system can be used to store the identifier of the job step operation message received for the first time. If the communication node receives the job step operation message again, the job step operation message received again is deduplicated based on the job step operation message received again and the first message list in the job scheduling system, so that the deduplicated intermediate job step operation message can be generated. Therefore, the process of deduplicating messages by the communication node can be applied to the job scheduling system.

[0020] In one embodiment, the deduplication of the re-received job step operation message based on the re-received job step operation message and the first message list to generate the intermediate job step operation message includes:

[0021] Compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result;

[0022] Deduplication is performed on the job step operation message received again according to the first comparison result to generate a first deduplication result;

[0023] The intermediate job step operation message is generated based on the first deduplication result and the job step operation messages in the first message list.

[0024] In this embodiment, the identifier of the re-received work step operation message is compared with the identifier of the work step operation message in the first message list to generate a first comparison result, so that it can be more accurately determined whether the re-received work step operation message and the work step operation message in the first message list are duplicate messages. The re-received work step operation message is deduplicated according to the first comparison result to generate a first deduplication result, so that it is possible to more accurately deduplication the duplicated work step operation message and more accurately store the non-duplicate work step operation message. Therefore, based on the first deduplication result and the work step operation message in the first message list, it is possible to more accurately generate the deduplicated intermediate work step operation message.

[0025] In one of the embodiments, the job step operation message includes at least one of an operation message for canceling a job step, an operation message for pausing a job step, an operation message for restarting a job step, and an operation message for deleting a job step.

[0026] In one embodiment, the preset job scheduling system includes a job scheduling system.

[0027] In a second aspect, the present application also provides a message deduplication method. Applied to a management node in a preset job scheduling system, where the management node corresponds to a plurality of computing nodes in the preset job scheduling system; the method comprises:

[0028] receiving an intermediate job step operation message; the intermediate job step operation message is obtained by removing duplicates from the job step operation messages of each of the computing nodes based on the communication nodes in the multiple computing nodes;

[0029] Creating a second message list based on the identifier of the intermediate job step operation message;

[0030] The intermediate job step operation messages are deduplicated according to the second message list to generate a target job step operation message.

[0031] In this embodiment, an intermediate job step operation message is received; the intermediate job step operation message is obtained by deduplicating the job step operation messages of each computing node based on the communication nodes in multiple computing nodes; the intermediate job step operation message that needs to be deduplicated again can be obtained. Afterwards, a second message list in the job scheduling system is created based on the intermediate job step operation message, and the intermediate job step operation message is deduplicated according to the second message list in the job scheduling system, so that the target job step operation message after deduplication again can be generated. Therefore, the process of deduplicating messages through the management node can be applied to the job scheduling system. In addition, the message deduplication method of the embodiment of the present application can not only no longer receive the same message before the same message processing is completed, so as to realize the one-to-one correspondence between message sending and message processing, and realize the idempotence of the message; and the message deduplication method of the embodiment of the present application can reduce the thread resource overhead caused by processing duplicate messages, so as to reduce the pressure of the slurmctld service in the management node, and improve the response speed of the slurmctld service, and then improve the operation stability of the job scheduling system.

[0032] In one embodiment, the step of removing duplicates from the intermediate job step operation messages according to the second message list to generate a target job step operation message includes:

[0033] Compare the identifier of the intermediate job step operation message received again with the identifiers of the intermediate job step operation messages in the second message list to generate a second comparison result;

[0034] Deduplication is performed on the intermediate job step operation message received again according to the second comparison result to generate a second deduplication result;

[0035] The target job step operation message is generated based on the second deduplication result and the intermediate job step operation messages in the second message list.

[0036] In this embodiment, the identifier of the intermediate work step operation message received again is compared with the identifier of the intermediate work step operation message in the second message list to generate a second comparison result, so that it can be more accurately determined whether the intermediate work step operation message received again and the intermediate work step operation message in the second message list are duplicate messages. According to the second comparison result, the intermediate work step operation message received again is deduplicated to generate a second deduplication result, so that it is more accurate to deduplication the duplicate intermediate work step operation message and more accurate to store the non-duplicate intermediate work step operation message. Therefore, based on the second deduplication result and the intermediate work step operation message in the second message list, the target work step operation message can be more accurately generated.

[0037] In a third aspect, the present application further provides a preset job scheduling system, the preset job scheduling system comprising a management node and a plurality of computing nodes corresponding to the management node;

[0038] The multiple computing nodes are used to implement the message deduplication method in any one of the embodiments of the first aspect above;

[0039] The management node is used to implement the message deduplication method in any one of the embodiments of the second aspect above.

[0040] In a fourth aspect, the present application also provides a message deduplication device. Applicable to multiple computing nodes in a preset job scheduling system, the multiple computing nodes corresponding to the target job; the device comprises:

[0041] A generating module, configured to generate a job step operation message of each computing node during the process of the plurality of computing nodes executing the target job, if an abnormal job step occurs in the target job; the job step operation message is used to operate the abnormal job step;

[0042] a sending module, configured to determine a communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the plurality of computing nodes, and there is a corresponding relationship between one communication node and the plurality of computing nodes;

[0043] The deduplication module is used to deduplicate the job step operation message through the communication node to generate an intermediate job step operation message.

[0044] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any one of the embodiments of the first aspect are implemented.

[0045] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments of the first aspect are implemented.

[0046] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in any one of the embodiments of the first aspect are implemented.

[0047] The above-mentioned message deduplication method, device, computer equipment, storage medium and program product, in the process of multiple computing nodes executing the target job, for each computing node, if an abnormal job step appears in the target job, then a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step; determine the communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes; deduplicate the job step operation message through the communication node to generate an intermediate job step operation message. The present application can firstly generate the job step operation message of the computing node when an abnormal job step appears in the target job, so as to obtain the job step operation message including duplicate messages in the job scheduling system. Secondly, at least one communication node is determined from the multiple computing nodes, so as to determine the communication node corresponding to each computing node. Afterwards, for each computing node, the job step operation message of the computing node is sent to the communication node corresponding to the computing node, and at least one common message list that can store the job step operation message can be established through the communication node corresponding to each computing node, and the job step operation message is deduplicated based on at least one message list, so that the deduplicated intermediate job step operation message can be generated. Therefore, the message deduplication method of the present application can deduplicate the job step operation message in the job scheduling system through the communication node corresponding to the computing node in the job scheduling system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 An application environment diagram of a message deduplication method in an embodiment;

[0049] Figure 2 A schematic diagram of a flow chart of a message deduplication method in an embodiment;

[0050] Figure 3 A schematic diagram of a flow chart of a communication node determination step in one embodiment;

[0051] Figure 4 A schematic diagram of the relationship between a computing node and a communication node in one embodiment;

[0052] Figure 5 A schematic diagram of a flow chart of the steps of sending a message from a computing node to a communication node in one embodiment;

[0053] Figure 6 A schematic diagram of a flow chart of a step of generating an intermediate job step operation message in one embodiment;

[0054] Figure 7 It is a flowchart of the first message deduplication step in one embodiment;

[0055] Figure 8 A schematic diagram of a flow chart of a message deduplication step performed by a communication node in one embodiment;

[0056] Fig. 9 A schematic diagram of a flow chart of a message deduplication method in another embodiment;

[0057] Fig.10 A schematic flow chart of a target job step operation message generation step in another embodiment;

[0058] Fig.11 It is a flowchart of a step of performing message deduplication by a management node in another embodiment;

[0059] Fig.12 A schematic diagram of a flow chart of a message deduplication method in an optional embodiment;

[0060] Fig.13 A schematic diagram of the structure of a preset job scheduling system in one embodiment;

[0061] Fig.14 is a structural block diagram of a message deduplication device in an embodiment;

[0062] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The job scheduling system is a system for monitoring and managing resources and jobs in a cluster. This job scheduling system includes multiple computing nodes and management nodes. For example, the job scheduling system includes the SLURM (Simple Linux Utility for Resource Management) job scheduling system. Among them, the slurmd (job scheduling service, Slurm local daemon) daemon process runs on each computing node, and the slurmctld (control management service, Slurm Central Daemon) daemon process runs on the management node. The slurmd daemon process and the slurmctld daemon process can communicate based on remote procedure call (RPC) messages. When the computing node is executing a job, the job steps in the job often have execution exceptions. Therefore, for these abnormal job steps (abnormal job steps for short), the computing node needs to send a message to cancel the job step to the management node, that is, the slurmd daemon sends a message to cancel the job step to the slurmctld daemon in the form of an RPC message, and then the slurmctld daemon in the management node receives and processes the message to cancel the job step, and performs the operation of canceling the job step on the abnormal job step according to the processed message to cancel the job step.

[0065] However, since the same job step can be executed on multiple computing nodes, and the number of computing nodes in such a job scheduling system is relatively large, if a large number of computing nodes send messages to the management node to cancel the job step, the management node will receive a large number of duplicate messages. Due to the limited processing capacity of the management node, the management node cannot process these large numbers of duplicate messages in time, resulting in excessive load on some management nodes, and even paralysis of the entire job scheduling system service. Therefore, it is necessary to deduplicate the large number of duplicate messages received by the management node in the job scheduling system.

[0066] Traditionally, message middleware is used to deduplicate a large number of duplicate messages. However, there is no message middleware in the job scheduling system. Therefore, there is no message deduplication method suitable for the job scheduling system in traditional technology.

[0067] The message deduplication method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the preset job scheduling system includes the above-mentioned job scheduling system. For example, the job scheduling system can be a SLURM job scheduling system. The job scheduling system includes a management node 102 and multiple computing nodes 104, and the management node 102 communicates with the computing node 104 through a network. The data storage system can store data that the management node 102 needs to process and data that the computing node 104 needs to process. The data storage system can be integrated on the management node 102, or on the computing node 104, or on a cloud or other network server. In the process of multiple computing nodes executing the target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step; the computing node 104 determines the communication node corresponding to the computing node, and sends the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes; the computing node 104 removes the duplication of the job step operation message through the communication node to generate an intermediate job step operation message. The management node 102 may be implemented by an independent server, and the computing node 104 may be implemented by a server cluster composed of multiple servers.

[0068] In one embodiment, Figure 2 As shown, a message deduplication method is provided, which is applied to Figure 1 Taking the computing node 104 in FIG. 1 as an example, the following steps are included:

[0069] S220, when multiple computing nodes are executing a target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step.

[0070] The target job is a job that needs to be executed by the job scheduling system. The job scheduling system includes multiple computing nodes, and the target job can be executed by multiple computing nodes, that is, multiple computing nodes correspond to the target job. The target job includes multiple job steps, wherein a job is composed of different job steps connected in sequence, and a job step refers to a relatively independent work done by a computer during the processing of a job. An abnormal job step refers to an abnormal job step in the process of executing a job. The job step operation message is used to operate the abnormal job step.

[0071] Optionally, in the process of executing the target job on multiple computing nodes, especially when starting or exiting a job step on a computing node, the job step often has execution exceptions due to the need to consider factors such as environment variables, hardware resources, job script syntax, and compilation of executable programs. Therefore, for each computing node, if an abnormal job step occurs in the target job, the computing node 104 can generate a job step operation message of the computing node, so that the computing node can deduplicate the job step operation message and send the deduplicated job step operation message to the management node. In an optional embodiment, the job step operation message can include at least one of an operation message to cancel the job step (i.e., stop executing the job step), an operation message to suspend the job step, an operation message to restart the job step, and an operation message to delete the job step.

[0072] S240, determine the communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes.

[0073] Optionally, since the process of each computing node sending a job step operation message to the management node is independent, it is impossible to obtain at least one common message list that can store job step operation messages between the computing nodes. Based on this, for each computing node, the computing node 104 can determine the communication node corresponding to the computing node from the multiple computing nodes that execute the target job, and send the job step operation message of the computing node to the communication node corresponding to the computing node. Among them, the communication node includes at least one computing node among the multiple computing nodes, and the number of communication nodes is much smaller than the number of computing nodes that execute the target job. Of course, the embodiment of the present application does not limit the number of communication nodes. And there is a corresponding relationship between a communication node and multiple computing nodes, that is, it can be understood that one computing node corresponds to one communication node, and one communication node corresponds to multiple computing nodes.

[0074] Optionally, the communication node corresponding to each computing node can be directly determined from the multiple computing nodes; or, multiple candidate communication nodes can be first determined from the multiple computing nodes, and then the communication node corresponding to each computing node can be determined from the multiple candidate communication nodes, thereby further reducing the number of communication nodes. For example, assuming that there are 10,000 computing nodes executing the target job, 100 computing nodes can be first selected as candidate communication nodes, and then 10 candidate communication nodes can be selected from the 100 candidate communication nodes as communication nodes.

[0075] S260, deduplicate the job step operation message through the communication node to generate an intermediate job step operation message.

[0076] Optionally, after at least one communication node receives the job step operation message sent by each computing node, the computing node 104 may establish at least one common message list that can store the job step operation message through at least one communication node, wherein the communication node corresponds to the message list one by one. Afterwards, at least one communication node may detect duplicate messages in the job step operation message based on at least one message list, and remove the duplicate messages in the job step operation message to generate a deduplicated intermediate job step operation message. The intermediate job step operation message refers to the job step operation message obtained after the job step operation message is deduplicated through the communication node.

[0077] In the above-mentioned message deduplication method, in the process of multiple computing nodes executing the target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step; the communication node corresponding to the computing node is determined, and the job step operation message of each computing node is sent to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes; the job step operation message is deduplicated through the communication node to generate an intermediate job step operation message. The present application can firstly generate the job step operation message of the computing node when an abnormal job step occurs in the target job, so as to obtain the job step operation message including duplicate messages in the job scheduling system. Secondly, at least one communication node is determined from the multiple computing nodes, so as to determine the communication node corresponding to each computing node. Afterwards, for each computing node, the job step operation message of the computing node is sent to the communication node corresponding to the computing node, so that at least one common message list that can store the job step operation message can be established through the communication node corresponding to each computing node, and the job step operation message is deduplicated based on at least one message list, so that the deduplicated intermediate job step operation message can be generated. Therefore, the message deduplication method of the present application can dedupe the job step operation message in the job scheduling system through the communication node corresponding to the computing node in the job scheduling system, and the message deduplication method of the present application is a message deduplication method that can be applied to the job scheduling system.

[0078] In the above embodiment, it is involved to determine the communication node corresponding to the computing node, and the specific method thereof is introduced below. Figure 3 As shown, S240 includes:

[0079] S320, determining a communication area width of a communication node according to the number of the plurality of computing nodes.

[0080] Optionally, the computing node 104 may obtain the number of multiple computing nodes executing the target job, and determine the communication area width of the communication node according to the number of multiple computing nodes. Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram of the relationship between a computing node and a communication node in an embodiment. Figure 4 As shown, the number of multiple computing nodes executing the target job is 9. After that, the square root of the number of multiple computing nodes is taken, and the square root is determined as the communication area width of the communication node, and the communication area width of the communication node can be obtained as 3. Since the communication node is limited to 255 concurrent threads when receiving messages, the square root of the number of multiple computing nodes is selected as the communication area width, which can not only deduplicate the received messages through the communication node, but also avoid excessive pressure on the communication node when receiving messages.

[0081] S340, dividing the plurality of computing nodes according to the width of the communication area, and obtaining a corresponding relationship between the communication nodes and the computing nodes corresponding to each communication node.

[0082] Optionally, the computing node 104 may obtain the identifier of each computing node, so that according to the communication area width of the communication node and the identifier of each computing node, the computing node 104 may divide the multiple computing nodes, that is, determine the communication area position of each computing node. Then, according to the communication area position and communication area width of each computing node, the corresponding relationship between the communication node and the computing node corresponding to each communication node is obtained.

[0083] Exemplary, combined Figure 4As shown, assuming that the identifiers of each computing node are 0-8, the identifier of each computing node is divided by the communication area width, and the value obtained by rounding down is the communication area position of each computing node. Then, the communication area position of each computing node is multiplied by the communication area width, and the value obtained is the identifier of the communication node. Taking computing node 4 as an example, the communication area position of computing node 4 is equal to the identifier of the computing node 4 divided by the communication area width 3 and rounded down, which is 1. The identifier of the communication node corresponding to computing node 4 is equal to the communication area position 1 of computing node 4 multiplied by the communication area width 3, which is 3. Thus, the correspondence between the communication nodes and the computing nodes corresponding to each communication node can be obtained, that is, the communication node corresponding to computing node 0 is computing node 0, the communication node corresponding to computing node 1 is computing node 0, and the communication node corresponding to computing node 2 is computing node 0; the communication node corresponding to computing node 3 is computing node 3, the communication node corresponding to computing node 4 is computing node 3, and the communication node corresponding to computing node 5 is computing node 3; the communication node corresponding to computing node 6 is computing node 6, the communication node corresponding to computing node 7 is computing node 6, and the communication node corresponding to computing node 8 is computing node 6.

[0084] S360, based on the correspondence between each communication node and the computing node corresponding to each communication node, determine the communication node corresponding to the computing node from the plurality of computing nodes.

[0085] Optionally, the computing node 104 may determine the communication node corresponding to the computing node from the plurality of computing nodes based on the correspondence between each communication node and the computing node corresponding to each communication node. Figure 4 As shown, according to the corresponding relationship between each communication node and the computing node corresponding to each communication node, it can be determined that the communication node corresponding to computing node 0, computing node 1, and computing node 2 is computing node 0, the communication node corresponding to computing node 3, computing node 4, and computing node 5 is computing node 3, and the communication node corresponding to computing node 6, computing node 7, and computing node 8 is computing node 6.

[0086] In addition, the address of the communication node can also be determined according to the identifier of the communication node, so that the job step operation message can be sent to the address of the communication node. Figure 5 As shown, Figure 5The flowchart of the steps of sending a message from a computing node to a communication node in one embodiment is shown in FIG. First, the computing node 104 can determine whether the type of message to be sent is a job step operation message. If the type of message to be sent is not a job step operation message, the computing node 104 can directly send the message to the management node. If the type of message to be sent is a job step operation message, the computing node 104 can obtain the list of computing nodes used in the process of executing the target job from the job structure. Secondly, based on the computing node list, the number of multiple computing nodes is obtained, and the square root of the number of multiple computing nodes is taken to obtain the communication area width of the communication node. Afterwards, the communication area position of each computing node is determined according to the communication area width and the identification of each computing node in the computing node list. Then, according to the communication area position of each computing node and the communication area width, the identification and address of the communication node are determined. Finally, the computing node 104 can send the job step operation message to the address of the communication node.

[0087] In this embodiment, the communication area width of the communication node is determined according to the number of multiple computing nodes, so that the number of computing nodes included in a communication area can be determined more accurately; the multiple computing nodes are divided according to the communication area width, so that the communication area position of each computing node can be determined more accurately, so as to more accurately obtain the corresponding relationship between the communication node and the computing node corresponding to each communication node; based on the more accurate corresponding relationship between each communication node and the computing node corresponding to each communication node, the communication node corresponding to the computing node can be more accurately determined from the multiple computing nodes. This embodiment adopts the idea of ​​regional communication, and can select a computing node in the communication area as the communication node in each communication area based on the more accurate corresponding relationship between each communication node and the computing node corresponding to each communication node, so as to more accurately determine the communication node corresponding to each computing node.

[0088] In the above embodiment, it involves deduplication of job step operation messages through communication nodes to generate intermediate job step operation messages. The specific method is introduced below. Figure 6 As shown, S260 includes:

[0089] S620: If the communication node receives the job step operation message, a first message list is created according to the identifier of the job step operation message through the communication node.

[0090] Optionally, the computing node 104 can detect whether the job step operation message is received through the communication node. It should be noted that the job step operation message received by the communication node at this time is the job step operation message received by the communication node for the first time. If it is detected that the communication node receives the job step operation message, the computing node 104 can obtain the identifier of the job step operation message through the communication node. Afterwards, the computing node 104 can create a first message list according to the identifier of the job step operation message through the communication node to store the identifier of the job step operation message in the first message list.

[0091] The first message list is used to record or store the identifier of the job step operation message. The identifier of the job step operation message includes the job identifier (JobId) and the job step identifier (StepId). Exemplarily, assuming that the job identifier (JobId) and the job step identifier (StepId) can include but are not limited to 0, 1, 2, 3, etc., then the identifier of the job step operation message can include but are not limited to 1.0, 1.1, 1.2, 1.3, 2.0, 2.1, etc. Therefore, the identifier of the job step operation message corresponding to any job step in any job is unique.

[0092] S640, if the communication node receives the job step operation message again, based on the job step operation message received again and the first message list, deduplicate the job step operation message received again, and generate an intermediate job step operation message.

[0093] Optionally, the computing node 104 may continue to detect whether the job step operation message is received again through the communication node. If it is detected that the communication node receives the job step operation message again, the computing node 104 may detect whether the job step operation message received again includes a repeated job step operation message based on the job step operation message received again and the first message list through the communication node, and remove the repeated messages in the job step operation message received again to generate a deduplicated intermediate job step operation message. The repeated job step operation message may be a message that is repeated with the job step operation message received by the communication node for the first time, or the repeated job step operation message may be a message that is repeated with the job step operation message received again by the communication node.

[0094] In this embodiment, if the communication node receives a job step operation message, the communication node creates a first message list according to the identifier of the job step operation message, and the first message list in the job scheduling system can be used to store the identifier of the job step operation message received for the first time. If the communication node receives the job step operation message again, the job step operation message received again is deduplicated based on the job step operation message received again and the first message list, and an intermediate job step operation message is generated. The job step operation message received again can be deduplicated according to the first message list in the job scheduling system, so that the deduplicated intermediate job step operation message can be generated. Therefore, the process of deduplicating messages through the communication node can be applied to the job scheduling system.

[0095] In the above embodiment, based on the re-received job step operation message and the first message list, the re-received job step operation message is deduplicated to generate an intermediate job step operation message. The specific method is described below. Figure 7 As shown, S640 includes:

[0096] S720: Compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result.

[0097] Alternatively, if Figure 8 As shown, Figure 8 The flowchart of the message deduplication step through the communication node in one embodiment is shown. After creating the first message list according to the identifier of the job step operation message through the communication node, the computing node 104 can continue to detect whether the job step operation message is received again through the communication node. If it is detected that the communication node has not received the job step operation message again, it can wait for a preset time period and continue to detect whether the job step operation message is received again. If it is detected that the communication node receives the job step operation message again, the computing node 104 can determine whether the type of the message received again is a job step operation message through the communication node. If the type of the message received again is not a job step operation message, the computing node 104 can directly send the message to the management node through the communication node.

[0098] If the type of the message received again is a job step operation message, the computing node 104 can obtain the identifier of the job step operation message received again through the communication node. And the computing node 104 can compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list through the communication node to generate a first comparison result, that is, it can be understood that the computing node 104 can determine whether the identifier of the job step operation message received again has been stored in the first message list through the communication node.

[0099] S740, deduplicate the job step operation message received again according to the first comparison result to generate a first deduplication result.

[0100] Optionally, based on the first comparison result, the computing node 104 may deduplicate the job step operation message received again through the communication node to generate a first deduplication result. Figure 8 As shown, if the first comparison result is that the identifier of the re-received job step operation message is the same as the identifier of the job step operation message in the first message list, it means that the identifier of the re-received job step operation message has been stored in the first message list. Then, the computing node 104 can abandon processing the re-received job step operation message through the communication node, that is, it does not need to send the re-received job step operation message to the management node. At this time, the first deduplication result does not include the re-received job step operation message.

[0101] If the first comparison result is that the identifier of the re-received job step operation message is different from the identifier of the job step operation message in the first message list, it means that the identifier of the re-received job step operation message is not stored in the first message list. Then, the computing node 104 can obtain the identifier of the re-received job step operation message through the communication node, and store the identifier of the re-received job step operation message in the first message list. At this time, the first deduplication result includes the re-received job step operation message and the identifier of the re-received job step operation message.

[0102] In addition, the computing node 104 can also determine the job step operation message corresponding to each identifier according to the identifier of the job step operation message stored in the first message list through the communication node. And send the job step operation message corresponding to each identifier in the first message list to the management node through the communication node. Optionally, after each deduplication, the non-duplicate job step operation message can be sent to the management node through the communication node; or, after the deduplication of multiple computing nodes executing the target job is completed, the deduplicated intermediate job step operation message can be uniformly sent to the management node. Of course, the embodiment of the present application does not limit the time for sending the deduplicated job step operation message to the management node through the communication node.

[0103] S760: Generate an intermediate operation step operation message based on the first deduplication result and the operation step operation message in the first message list.

[0104] Optionally, the computing node 104 may generate an intermediate job step operation message based on the first deduplication result and the job step operation message in the first message list. Optionally, if the first deduplication result does not include the job step operation message received again, then the intermediate job step operation message includes the job step operation message corresponding to each identifier in the first message list. If the first deduplication result includes the job step operation message received again and the identifier of the job step operation message received again, then the intermediate job step operation message includes the job step operation message received again and the job step operation message corresponding to each identifier in the first message list.

[0105] In addition, taking the SLURM job scheduling system as an example, combined with Figure 8 As shown, computing node 104 can also judge whether the target job is executed through the communication node. If the target job is not executed, computing node 104 can continue to detect whether the job step operation message is received again through the communication node. If the target job is executed, computing node 104 can also remove the job step operation message related to the target job from the first message list through the communication node. Afterwards, computing node 104 can also judge whether the slurmd service corresponding to the slurmd daemon process in the computing node stops running through the communication node. If the slurmd service corresponding to the slurmd daemon process does not stop running, computing node 104 can continue to detect whether the job step operation message is received again through the communication node. If the slurmd service corresponding to the slurmd daemon process stops running, then end the step of deduplication through the communication node.

[0106] In this embodiment, the identifier of the re-received work step operation message is compared with the identifier of the work step operation message in the first message list to generate a first comparison result, so that it can be more accurately determined whether the re-received work step operation message and the work step operation message in the first message list are duplicate messages. The re-received work step operation message is deduplicated according to the first comparison result to generate a first deduplication result, so that it is possible to more accurately deduplication the duplicated work step operation message and more accurately store the non-duplicate work step operation message. Therefore, based on the first deduplication result and the work step operation message in the first message list, it is possible to more accurately generate the deduplicated intermediate work step operation message.

[0107] In one embodiment, Fig. 9 As shown, a message deduplication method is provided, which is applied to Figure 1 Taking the management node 102 in FIG. 1 as an example, the following steps are included:

[0108] S920, receiving an intermediate job step operation message; the intermediate job step operation message is obtained by removing duplicate job step operation messages of each computing node based on communication nodes in multiple computing nodes.

[0109] Wherein, the job scheduling system includes a management node, and the management node corresponds to a plurality of computing nodes in the job scheduling system. The management node is responsible for monitoring the status of each computing node in the job scheduling system, and for periodically checking the information in the slurmd daemon process. The intermediate job step operation message is obtained by deduplicating the job step operation message of each computing node based on the communication node in the plurality of computing nodes. Optionally, after the deduplicated intermediate job step operation message is generated by the communication node in the computing node, the intermediate job step operation message can also be sent to the management node through the communication node. Thus, the management node 102 can receive the deduplicated intermediate job step operation message.

[0110] S940: Create a second message list based on the intermediate job step operation message.

[0111] Optionally, when the management node 102 receives the intermediate operation step operation message sent by the communication node for the first time, it can create a second message list based on the intermediate operation step operation message received for the first time, wherein the second message list is used to record or store the identifier of the intermediate operation step operation message.

[0112] In one embodiment, S940 includes:

[0113] S942, obtaining the identifier of the intermediate job step operation message.

[0114] Optionally, the management node 102 may obtain the identifier of the intermediate job step operation message received for the first time. The identifier of the intermediate job step operation message includes a job identifier (JobId) and a job step identifier (StepId). Exemplarily, assuming that both the job identifier (JobId) and the job step identifier (StepId) may include but are not limited to 0, 1, 2, 3, etc., then the identifier of the intermediate job step operation message may include but are not limited to 1.0, 1.1, 1.2, 1.3, 2.0, 2.1, etc. Therefore, the identifier of the intermediate job step operation message corresponding to any job step in any job is unique.

[0115] S944: Create a second message list according to the identifier of the intermediate job step operation message.

[0116] Optionally, the management node 102 may create a second message list based on the identifier of the intermediate job step operation message, that is, it can be understood that the management node 102 may store the identifier of the intermediate job step operation message in the second message list, so that the intermediate job step operation message can be easily deduplicated based on the identifier of the intermediate job step operation message in the second message list.

[0117] S960, deduplicate the intermediate job step operation messages according to the second message list to generate a target job step operation message.

[0118] Optionally, the management node 102 may detect whether the received intermediate job step operation messages include duplicate intermediate job step operation messages according to the second message list, and remove duplicate messages from the received intermediate job step operation messages to generate a target job step operation message after deduplication. The target job step operation message refers to a job step operation message obtained after the management node removes duplicate intermediate job step operation messages.

[0119] In this embodiment, an intermediate job step operation message is received; the intermediate job step operation message is obtained by deduplicating the job step operation messages of each computing node based on the communication nodes in multiple computing nodes; the intermediate job step operation message that needs to be deduplicated again can be obtained. Afterwards, a second message list in the job scheduling system is created based on the intermediate job step operation message, and the intermediate job step operation message is deduplicated according to the second message list in the job scheduling system, so that the target job step operation message after deduplication again can be generated. Therefore, the process of deduplicating messages through the management node can be applied to the job scheduling system. In addition, the message deduplication method of the embodiment of the present application can not only no longer receive the same message before the same message processing is completed, so as to realize the one-to-one correspondence between message sending and message processing, and realize the idempotence of the message; and the message deduplication method of the embodiment of the present application can reduce the thread resource overhead caused by processing duplicate messages, so as to reduce the pressure of the slurmctld service in the management node, and improve the response speed of the slurmctld service, and then improve the operation stability of the job scheduling system.

[0120] In the above embodiment, it involves removing duplicates from the intermediate job step operation messages according to the second message list to generate the target job step operation message. The specific method is introduced below. Fig.10 As shown, S960 includes:

[0121] S1020: Compare the identifier of the intermediate work step operation message received again with the identifier of the intermediate work step operation message in the second message list to generate a second comparison result.

[0122] Alternatively, if Fig.11 As shown, Fig.11 The flowchart of the message deduplication step performed by the management node in one embodiment is shown. After the management node creates the second message list according to the identifier of the intermediate operation step operation message received for the first time, the management node 102 can continue to detect whether the intermediate operation step operation message is received again. If it is detected that the management node has not received the intermediate operation step operation message again, it can wait for a preset time period and continue to detect whether the intermediate operation step operation message is received again. If it is detected that the management node receives the intermediate operation step operation message again, the management node 102 can obtain the identifier of the intermediate operation step operation message received again. And the management node 102 can compare the identifier of the intermediate operation step operation message received again with the identifier of the intermediate operation step operation message in the second message list to generate a second comparison result, that is, it can be understood that the management node 102 can determine whether the identifier of the intermediate operation step operation message received again has been stored in the second message list.

[0123] S1040: De-duplicate the intermediate job step operation message received again according to the second comparison result to generate a second de-duplication result.

[0124] Optionally, according to the second comparison result, the management node 102 may deduplicate the intermediate job step operation message received again to generate a second deduplication result. Fig.11 As shown in FIG. 1 , if the second comparison result is that the identifier of the intermediate job step operation message received again is the same as the identifier of the intermediate job step operation message in the second message list, it means that the identifier of the intermediate job step operation message received again has been stored in the second message list. Then, the management node 102 may abandon processing the intermediate job step operation message received again, that is, it does not need to process the intermediate job step operation message received again. At this time, the second deduplication result does not include the intermediate job step operation message received again.

[0125] If the second comparison result is that the identifier of the intermediate job step operation message received again is different from the identifier of the intermediate job step operation message in the second message list, it means that the identifier of the intermediate job step operation message received again is not stored in the second message list. Then, the management node 102 can obtain the identifier of the intermediate job step operation message received again, and store the identifier of the intermediate job step operation message received again in the second message list. At this time, the second deduplication result includes the intermediate job step operation message received again and the identifier of the intermediate job step operation message received again.

[0126] In addition, the management node 102 can determine the intermediate job step operation messages corresponding to each identifier based on the identifier of the intermediate job step operation message stored in the second message list. The management node 102 can also process the intermediate job step operation messages corresponding to each identifier in the second message list, so as to perform the operation of canceling the job step on the abnormal job step according to the processed message of canceling the job step. Optionally, after each deduplication, the management node processes the intermediate job step operation messages that are not repeated; or, after all the intermediate job step operation messages are deduplicated, the target job step operation messages after deduplication are uniformly processed. Of course, the embodiment of the present application does not limit the time for processing the intermediate job step operation messages after deduplication.

[0127] S1060: Generate a target operation step operation message based on the second deduplication result and the intermediate operation step operation message in the second message list.

[0128] Optionally, the management node 102 may generate a target operation step operation message based on the second deduplication result and the intermediate operation step operation message in the second message list. Optionally, if the second deduplication result does not include the intermediate operation step operation message received again, then the target operation step operation message includes the intermediate operation step operation message corresponding to each identifier in the second message list. If the second deduplication result includes the intermediate operation step operation message received again and the identifier of the intermediate operation step operation message received again, then the target operation step operation message includes the intermediate operation step operation message received again and the intermediate operation step operation message corresponding to each identifier in the second message list.

[0129] In addition, taking the SLURM job scheduling system as an example, combined with Fig.11 As shown, the management node 102 can also determine whether the target job is executed. If the target job is not executed, the management node 102 can continue to detect whether the intermediate job step operation message is received again. If the target job is executed, the management node 102 can remove the intermediate job step operation message related to the target job from the second message list. Afterwards, the management node 102 can also determine whether the slurmctld service corresponding to the slurmctld daemon process in the management node has stopped running. If the slurmctld service corresponding to the slurmctld daemon process has not stopped running, the management node 102 can continue to detect whether the intermediate job step operation message is received again. If the slurmctld service corresponding to the slurmctld daemon process has stopped running, the step of deduplication through the management node is terminated.

[0130] In this embodiment, the identifier of the intermediate work step operation message received again is compared with the identifier of the intermediate work step operation message in the second message list to generate a second comparison result, so that it can be more accurately determined whether the intermediate work step operation message received again and the intermediate work step operation message in the second message list are duplicate messages. According to the second comparison result, the intermediate work step operation message received again is deduplicated to generate a second deduplication result, so that it is more accurate to deduplication the duplicate intermediate work step operation message and more accurate to store the non-duplicate intermediate work step operation message. Therefore, based on the second deduplication result and the intermediate work step operation message in the second message list, the target work step operation message can be more accurately generated.

[0131] In an optional embodiment, if Fig.12 As shown, a message deduplication method is provided, which is applied to the computing node 104, including:

[0132] S1202, when multiple computing nodes are executing a target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step;

[0133] S1204, determining a communication area width of a communication node according to the number of the plurality of computing nodes;

[0134] S1206, dividing the plurality of computing nodes according to the width of the communication area, and obtaining a correspondence between the communication nodes and the computing nodes corresponding to the communication nodes;

[0135] S1208, determining a communication node corresponding to a computing node from a plurality of computing nodes based on a correspondence relationship between each communication node and a computing node corresponding to each communication node;

[0136] S1210, sending the job step operation message of each computing node to a communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes;

[0137] S1212, if the communication node receives the job step operation message, create a first message list according to the identifier of the job step operation message through the communication node;

[0138] S1214, if the communication node receives the job step operation message again, compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result;

[0139] S1216, deduplicating the job step operation message received again according to the first comparison result to generate a first deduplication result;

[0140] S1218: Generate an intermediate job step operation message based on the first deduplication result and the job step operation message in the first message list.

[0141] In the above-mentioned message deduplication method, first, when an abnormal job step occurs in the target job, the job step operation message of the computing node is generated, so that the job step operation message including the duplicate message in the job scheduling system can be obtained. Secondly, at least one communication node is determined from multiple computing nodes, so that the communication node corresponding to each computing node can be determined. After that, for each computing node, the job step operation message of the computing node is sent to the communication node corresponding to the computing node, so that at least one common message list that can store the job step operation message can be established through the communication node corresponding to each computing node, and the job step operation message is deduplicated based on at least one message list, so that the deduplicated intermediate job step operation message can be generated. Therefore, the message deduplication method of the present application can dedupe the job step operation message in the job scheduling system through the communication node corresponding to the computing node in the job scheduling system, and the message deduplication method of the present application is a message deduplication method that can be applied to the job scheduling system.

[0142] In summary, the present application also provides a preset job scheduling system, such as Fig.13 As shown, the preset job scheduling system includes a job scheduling system. The preset job scheduling system includes a management node and a plurality of computing nodes corresponding to the management node, wherein the management node is connected to the plurality of computing nodes.

[0143] The above multiple computing nodes are used to execute Figure 2-Figure 8 For details of the message deduplication method in any embodiment, please refer to the aforementioned Figure 2-Figure 8 The description content will not be repeated here.

[0144] The above management nodes are used to execute Figure 9-11 For details of the message deduplication method in any embodiment, please refer to the aforementioned Figure 9-11 The description content will not be repeated here.

[0145] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] Based on the same inventive concept, the embodiment of the present application also provides a message deduplication device for implementing the message deduplication method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more message deduplication device embodiments provided below can refer to the limitations on the message deduplication method above, and will not be repeated here.

[0147] In one embodiment, Fig.14 As shown, a message deduplication device 1400 is provided, which is applied to multiple computing nodes in a preset job scheduling system, and the multiple computing nodes correspond to target jobs; the message deduplication device 1400 includes: a generation module 1420, a sending module 1440 and a deduplication module 1460, wherein:

[0148] The generation module 1420 is used to generate a job step operation message of the computing node for each computing node during the process of executing the target job by multiple computing nodes, if an abnormal job step occurs in the target job; the job step operation message is used to operate the abnormal job step.

[0149] The sending module 1440 is used to determine the communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among multiple computing nodes, and there is a corresponding relationship between one communication node and multiple computing nodes.

[0150] The deduplication module 1460 is used to deduplicate the job step operation message through the communication node to generate an intermediate job step operation message.

[0151] In one embodiment, the sending module 1440 includes:

[0152] A communication area width determination unit, used to determine the communication area width of the communication node according to the number of the plurality of computing nodes;

[0153] A corresponding relationship acquisition unit, used for dividing the plurality of computing nodes according to the width of the communication area, and obtaining a corresponding relationship between the communication nodes and the computing nodes corresponding to each communication node;

[0154] The communication node determination unit is used to determine the communication node corresponding to the computing node from the plurality of computing nodes based on the corresponding relationship between each communication node and the computing node corresponding to each communication node.

[0155] In one embodiment, the deduplication module 1460 includes:

[0156] A first message list creating unit, configured to create a first message list according to an identifier of the job step operation message through the communication node if the communication node receives the job step operation message;

[0157] The intermediate operation step operation message generating unit is used for, if the communication node receives the operation step operation message again, deduplicating the operation step operation message received again based on the operation step operation message received again and the first message list, and generating the intermediate operation step operation message.

[0158] In one embodiment, the intermediate job step operation message generating unit includes:

[0159] A first comparison result generating subunit, used for comparing the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result;

[0160] A first deduplication result generating subunit, configured to perform deduplication on the job step operation message received again according to the first comparison result, and generate a first deduplication result;

[0161] The intermediate job step operation message generation subunit is used to generate an intermediate job step operation message based on the first deduplication result and the job step operation messages in the first message list.

[0162] In one embodiment, the job step operation message includes at least one of an operation message for canceling a job step, an operation message for pausing a job step, an operation message for restarting a job step, and an operation message for deleting a job step.

[0163] In one embodiment, a message deduplication device is also provided, which is applied to a management node in a preset job scheduling system, in which the management node corresponds to a plurality of computing nodes; the message deduplication device comprises: an intermediate job step operation message receiving module, a second message list creating module and an intermediate job step operation message deduplication module, wherein:

[0164] The intermediate job step operation message receiving module is used to receive the intermediate job step operation message; the intermediate job step operation message is obtained by removing duplicate job step operation messages of each computing node based on the communication nodes in the multiple computing nodes;

[0165] A second message list creation module, used to create a second message list based on the intermediate job step operation message;

[0166] The intermediate job step operation message deduplication module is used to deduplicate the intermediate job step operation message according to the second message list to generate a target job step operation message.

[0167] In one embodiment, the second message list creation module includes:

[0168] An identification acquisition unit for an intermediate operation step operation message, used to acquire an identification of the intermediate operation step operation message;

[0169] The second message list creating unit is used to create the second message list according to the identifier of the intermediate job step operation message.

[0170] In one embodiment, the intermediate job step operation message deduplication module includes:

[0171] A second comparison result generating unit, configured to compare the identifier of the intermediate operation step operation message received again with the identifier of the intermediate operation step operation message in the second message list to generate a second comparison result;

[0172] A second deduplication result generating unit, configured to perform deduplication on the intermediate job step operation message received again according to the second comparison result, and generate a second deduplication result;

[0173] The target operation step operation message generating unit is used to generate a target operation step operation message based on the second deduplication result and the intermediate operation step operation messages in the second message list.

[0174] Each module in the above-mentioned message deduplication device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0175] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.15As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store message deduplication data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a message deduplication method is implemented.

[0176] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0178] In the process of executing the target job on multiple computing nodes, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step;

[0179] Determine a communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the plurality of computing nodes, and there is a corresponding relationship between one communication node and the plurality of computing nodes;

[0180] The job step operation messages are deduplicated through the communication nodes to generate intermediate job step operation messages.

[0181] In one embodiment, when determining a communication node corresponding to a computing node, the processor further implements the following steps when executing the computer program:

[0182] Determine the communication area width of the communication node according to the number of the plurality of computing nodes;

[0183] Dividing multiple computing nodes according to the width of the communication area, obtaining a corresponding relationship between the communication nodes and the computing nodes corresponding to each communication node;

[0184] Based on the correspondence between each communication node and the computing node corresponding to each communication node, a communication node corresponding to the computing node is determined from the plurality of computing nodes.

[0185] In one embodiment, the communication node is used to deduplicate the job step operation message to generate an intermediate job step operation message, and the processor further implements the following steps when executing the computer program:

[0186] If the communication node receives the job step operation message, a first message list is created according to the identifier of the job step operation message through the communication node;

[0187] If the communication node receives the work step operation message again, based on the work step operation message received again and the first message list, the work step operation message received again is deduplicated to generate an intermediate work step operation message.

[0188] In one embodiment, based on the re-received job step operation message and the first message list, the re-received job step operation message is deduplicated to generate an intermediate job step operation message, and the processor further implements the following steps when executing the computer program:

[0189] Compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result;

[0190] Deduplication is performed on the job step operation message received again according to the first comparison result to generate a first deduplication result;

[0191] Based on the first deduplication result and the job step operation messages in the first message list, an intermediate job step operation message is generated.

[0192] In one embodiment, the job step operation message includes at least one of an operation message for canceling a job step, an operation message for pausing a job step, an operation message for restarting a job step, and an operation message for deleting a job step.

[0193] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0194] receiving an intermediate job step operation message; the intermediate job step operation message is obtained by removing duplicate job step operation messages of each computing node based on communication nodes in multiple computing nodes;

[0195] Creating a second message list based on the identifier of the intermediate job step operation message;

[0196] The intermediate job step operation messages are deduplicated according to the second message list to generate a target job step operation message.

[0197] In one embodiment, the intermediate job step operation messages are deduplicated according to the second message list to generate the target job step operation message, and the processor further implements the following steps when executing the computer program:

[0198] Compare the identifier of the intermediate job step operation message received again with the identifiers of the intermediate job step operation messages in the second message list to generate a second comparison result;

[0199] Deduplication is performed on the intermediate job step operation message received again according to the second comparison result to generate a second deduplication result;

[0200] A target job step operation message is generated based on the second deduplication result and the intermediate job step operation messages in the second message list.

[0201] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0202] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0203] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0204] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A message deduplication method, characterized in that: Applied to a plurality of computing nodes in a preset job scheduling system, the plurality of computing nodes corresponding to target jobs; the method comprises: During the process of the plurality of computing nodes executing the target job, for each computing node, if an abnormal job step occurs in the target job, a job step operation message of the computing node is generated; the job step operation message is used to operate the abnormal job step; Determine a communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and the multiple computing nodes; The job step operation message is deduplicated through the communication node to generate an intermediate job step operation message.

2. The method according to claim 1, characterized in that The determining of the communication node corresponding to the computing node includes: Determining a communication area width of the communication node according to the number of the plurality of computing nodes; Dividing the plurality of computing nodes according to the width of the communication area to obtain a corresponding relationship between the communication nodes and the computing nodes corresponding to each of the communication nodes; Based on the correspondence between each of the communication nodes and the computing nodes corresponding to each of the communication nodes, the communication node corresponding to the computing node is determined from the plurality of computing nodes.

3. The method according to claim 1 or 2, characterized in that: The step of removing duplicates from the job step operation message by the communication node to generate an intermediate job step operation message includes: If the communication node receives the job step operation message, creating a first message list according to the identifier of the job step operation message through the communication node; If the communication node receives the job step operation message again, the communication node deduplicates the job step operation message received again based on the job step operation message received again and the first message list to generate the intermediate job step operation message.

4. The method according to claim 3, characterized in that The deduplication of the re-received job step operation message based on the re-received job step operation message and the first message list to generate the intermediate job step operation message includes: Compare the identifier of the job step operation message received again with the identifier of the job step operation message in the first message list to generate a first comparison result; Deduplication is performed on the job step operation message received again according to the first comparison result to generate a first deduplication result; The intermediate job step operation message is generated based on the first deduplication result and the job step operation messages in the first message list.

5. The method according to claim 1 or 2, characterized in that: The job step operation message includes at least one of an operation message for canceling a job step, an operation message for pausing a job step, an operation message for restarting a job step, and an operation message for deleting a job step.

6. A message deduplication method, characterized in that: A management node applied to a preset job scheduling system, wherein the management node corresponds to a plurality of computing nodes in the preset job scheduling system; the method comprises: Receive an intermediate job step operation message; the intermediate job step operation message is generated for each computing node when an abnormal job step occurs in the target job during the execution of the target job by the multiple computing nodes, and the communication node corresponding to each computing node is determined, and the job step operation message of each computing node is sent to the communication node corresponding to each computing node, and the job step operation message is obtained after deduplication of the job step operation message through each communication node; the job step operation message is used to operate the abnormal job step; the communication node includes at least one computing node among the multiple computing nodes, and there is a corresponding relationship between one communication node and multiple computing nodes; Creating a second message list based on the identifier of the intermediate job step operation message; The intermediate job step operation messages are deduplicated according to the second message list to generate a target job step operation message.

7. The method according to claim 6, characterized in that The step of removing duplicates from the intermediate job step operation messages according to the second message list to generate a target job step operation message includes: Compare the identifier of the intermediate job step operation message received again with the identifiers of the intermediate job step operation messages in the second message list to generate a second comparison result; Deduplication is performed on the intermediate job step operation message received again according to the second comparison result to generate a second deduplication result; The target job step operation message is generated based on the second deduplication result and the intermediate job step operation messages in the second message list.

8. A preset job scheduling system, characterized in that: The preset job scheduling system includes a management node and a plurality of computing nodes corresponding to the management node; The multiple computing nodes are used to execute the message deduplication method described in any one of claims 1 to 5; The management node is used to execute the message deduplication method described in any one of claims 6-7.

9. A message deduplication device, characterized in that: Applied to a plurality of computing nodes in a preset job scheduling system, the plurality of computing nodes corresponding to target jobs; the device comprises: A generating module, configured to generate a job step operation message of each computing node during the process of the plurality of computing nodes executing the target job, if an abnormal job step occurs in the target job; the job step operation message is used to operate the abnormal job step; a sending module, configured to determine a communication node corresponding to the computing node, and send the job step operation message of each computing node to the communication node corresponding to the computing node; the communication node includes at least one computing node among the plurality of computing nodes, and there is a corresponding relationship between one communication node and the plurality of computing nodes; The deduplication module is used to deduplicate the job step operation message through the communication node to generate an intermediate job step operation message.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images