A distributed task collaboration method based on big data

Abstract

The present invention relates to the field of data processing technology, and in particular to a distributed task collaboration method based on big data, comprising: step S1, splitting a big data task submitted by any user into a plurality of subtask data; step S2, merging the calculation results of the plurality of nodes, verifying the information of the user and issuing a permission access failure notification when the user does not have permission; step S3, obtaining the number of successful runs of the big data task within a plurality of collaboration cycles; step S4, judging the collaborative stability of the distributed task based on the variance of the proportion of the successful runs of the big data task; step S5, if it is determined that the stability does not meet the requirements, adjusting the storage capacity of the subtask data block; step S6, if it is determined that the effectiveness does not meet the requirements, adjusting the interval duration of adjacent concurrent processing batches. The present invention achieves an improvement in the collaborative stability of distributed tasks.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed task collaboration method based on big data. Background Art

[0002] In the existing technology, large models are now one of the more popular auxiliary development solutions, usually used for data cleaning and creation. At present, the community uses the interfaces of TensorFlow, PyTorch and MXNet for this content, and can communicate and coordinate across nodes through technologies such as Kubernetes, MPI, NCCL, etc. The distributed training method can distribute the computing load to multiple computers or servers, thereby accelerating the training speed and improving the computing efficiency, but it can only support the same model at the same time.

[0003] Chinese Patent Publication No.: CN108170861A discloses a distributed database system collaborative optimization method based on dynamic programming, including the following steps: A. Using a data flow-based block method, the distributed data set is divided into a series of indivisible data primitives according to the planned task sequence and the indistinguishable relationship of the data unit operation; B. Optimization of concurrent processing tasks; C. Load balancing optimization of distributed resources, and calculation of the planned task cost; D. Dynamic scheduling method of planned tasks. It can be seen that the distributed database system collaborative optimization method based on dynamic programming has the problem that due to the relationship between some data in the task data to be split, the data with the relationship is split during the splitting, resulting in the inability of the task to run correctly, thereby causing the collaborative stability of the distributed task to decrease. Summary of the invention

[0004] To this end, the present invention provides a distributed task collaboration method based on big data, which is used to overcome the problem in the prior art that due to the existence of relationships between some data in the task data to be split, the data with relationships are split apart during splitting, resulting in the task being unable to run correctly and thus causing the collaborative stability of the distributed task to decrease.

[0005] To achieve the above-mentioned purpose, the present invention provides a distributed task collaboration method based on big data, comprising: step S1, splitting a big data task submitted by any user into a number of subtask data, and distributing the several subtask data to a number of nodes for collaborative calculation; step S2, merging the calculation results of the several nodes, verifying the information of the user and issuing a permission access failure notification when the user does not have permission; step S3, obtaining the number of successful runs of the big data task within a number of collaborative cycles; step S4, judging the collaborative stability of the distributed tasks based on the variance of the proportion of the successful runs of the big data task; step S5, if it is determined that the stability does not meet the requirements, adjusting the storage capacity of the subtask data block or executing a process for determining the validity of the verification of the user information; step S6, if it is determined that the validity does not meet the requirements, adjusting the interval duration of adjacent concurrent processing batches or adjusting the number of shards of the subtask data based on the average packet loss rate of the subtask data.

[0006] Further, adjusting the storage capacity of the subtask data block includes:

[0007] Get the number of successful executions of big data tasks within several collaborative cycles;

[0008] The variance of the proportion of the successful number of big data task operations is calculated according to the successful number of big data task operations in the several collaborative cycles and the total number of big data task operations in the several collaborative cycles.

[0009] Comparing the variance of the percentage of successful times of the big data task execution with a preset first variance and a preset second variance respectively;

[0010] If the variance of the percentage of successful times of the big data task execution is greater than the preset first variance, it is determined that the collaborative stability of the distributed task does not meet the requirements.

[0011] Further, determining the verification validity of the user information includes:

[0012] Comparing the variance of the percentage of successful times of the big data task execution with the preset first variance and the preset second variance respectively;

[0013] If the variance of the percentage of successful times of the big data task running is greater than the preset first variance and less than or equal to the preset second variance, it is preliminarily determined that the verification validity of the user information does not meet the requirements, and a secondary determination is made on the verification validity of the user information based on the percentage of failed times of user verification.

[0014] Furthermore, if the variance of the percentage of successful times of the big data task execution is greater than the preset second variance, the storage capacity of the subtask database is increased.

[0015] Furthermore, the increase in the storage capacity of the subtask data block is determined by the difference between the variance of the percentage of successful times of the big data task running and a preset second variance.

[0016] Further, adjusting the interval between adjacent concurrent processing batches according to the proportion of the number of user authentication failures includes:

[0017] Obtain the number of user authentication failures and the total number of user authentications in several authentication cycles respectively, and calculate the proportion of the number of user authentication failures;

[0018] Compare the proportion of the number of failed user authentications with the preset first proportion and the preset second proportion respectively;

[0019] If the proportion of the number of times the user verification fails is greater than the preset first proportion, it is determined that the verification validity of the user information does not meet the requirements;

[0020] If the proportion of the number of failed user authentications is greater than the preset first proportion and less than or equal to the preset second proportion, the interval between adjacent concurrent processing batches is increased;

[0021] If the proportion of the number of user verification failures is greater than the preset second proportion, it is preliminarily determined that the distribution validity of the subtask data does not meet the requirements, and a secondary determination is made on the distribution validity of the subtask data based on the average packet loss rate of the subtask data.

[0022] Furthermore, the proportion of the number of user authentication failures is the ratio of the number of user authentication failures to the total number of user authentications in a number of authentication cycles.

[0023] Furthermore, the increase range of the interval duration between adjacent concurrently processed batches is determined by the difference between the proportion of the number of user authentication failures and a preset first proportion.

[0024] Further, adjusting the number of shards of the subtask data according to the average packet loss rate of the subtask data includes:

[0025] Obtain the packet loss rate of subtask data in several distribution cycles, and calculate the average packet loss rate of subtask data;

[0026] Comparing the average packet loss rate of the subtask data with a preset packet loss rate;

[0027] If the average packet loss rate of the subtask data is greater than the preset packet loss rate, it is determined that the distribution validity of the subtask data does not meet the requirements, and the number of fragments of the subtask data is increased.

[0028] Furthermore, the increase range of the number of fragments of the subtask data is determined by the difference between the average packet loss rate of the subtask data and a preset packet loss rate.

[0029] Compared with the prior art, the beneficial effect of the present invention lies in that the method of the present invention adjusts the storage capacity of the subtask data block according to the variance of the proportion of successful times of the big data task running by setting steps S1-S6. Since there is a relationship between some data in the task data to be split, the data with the relationship is split during the splitting, resulting in the task being unable to run correctly, thereby causing the task collaboration failure. By increasing the storage capacity of the subtask data block, the amount of data contained in each data block is increased, and the risk of incorrect splitting of the data with the relationship is reduced. By adjusting the interval time of adjacent concurrent processing batches according to the proportion of the number of user verification failures, since two concurrent processing operations may occur during the concurrent processing of data, Each node processes the same data, but the final processing results are different, resulting in data conflicts, which in turn causes the failure of user permission verification. By reducing the interval between adjacent concurrent processing batches, the probability of data conflicts is reduced, thereby improving the effectiveness of user information verification. By adjusting the number of shards of subtask data according to the average packet loss rate of subtask data, the network is congested due to the large amount of task data processed at the same time, resulting in the inability to establish effective contact with the node, which in turn causes data distribution interruption, resulting in an increase in the data packet loss rate. By increasing the number of shards of subtask data, the size of a single data packet is reduced, the probability of data packet loss is reduced, and the collaborative stability of distributed tasks is improved.

[0030] Furthermore, the method of the present invention adjusts the storage capacity of the subtask data block by setting a preset first variance and a preset second variance. Since there is a relationship between some data in the task data that needs to be split, the data with the relationship is split apart during the splitting, resulting in the task being unable to run correctly, thereby causing the task collaboration to fail. By increasing the storage capacity of the subtask data block, the amount of data contained in each data block is increased, reducing the risk of incorrect splitting of data with relationships, thereby further achieving improved collaborative stability for distributed tasks.

[0031] Furthermore, the method of the present invention adjusts the interval duration of adjacent concurrent processing batches by setting a preset first ratio and a preset second ratio. Since in the process of concurrent processing of data, two nodes may process the same data, and the final processing results are different, resulting in data conflicts, thereby causing the user's authority verification to fail. By reducing the interval duration of adjacent concurrent processing batches, the probability of data conflicts is reduced, thereby improving the effectiveness of user information verification, and further achieving an improvement in the collaborative stability of distributed tasks.

[0032] Furthermore, the method of the present invention adjusts the number of shards of subtask data by setting a preset packet loss rate. Since the amount of task data processed simultaneously is too large, the network is blocked, and it is impossible to establish an effective connection with the node, which in turn leads to data distribution interruption, thereby increasing the data packet loss rate. By increasing the number of shards of subtask data, the size of a single data packet is reduced, the probability of data packet loss is reduced, and the collaborative stability of distributed tasks is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an overall flow chart of a distributed task collaboration method based on big data according to an embodiment of the present invention;

[0034] Figure 2 It is a logic flow chart of a distributed task collaboration method based on big data according to an embodiment of the present invention;

[0035] Figure 3 A specific flow chart of the process of adjusting the interval length of adjacent concurrent processing batches in the distributed task collaboration method based on big data according to an embodiment of the present invention;

[0036] Figure 4 The present invention is a specific flow chart of the process of adjusting the number of shards of subtask data in the distributed task collaboration method based on big data in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0039] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0040] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0041] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively an overall flow chart of a distributed task collaboration method based on big data, a logical flow chart, a specific flow chart of a process of adjusting the interval length of adjacent concurrent processing batches, and a specific flow chart of a process of adjusting the number of shards of subtask data. A distributed task collaboration method based on big data of the present invention comprises:

[0042] Step S1, splitting a big data task submitted by any user into a number of subtask data, and distributing the subtask data to a number of nodes for collaborative computing;

[0043] Step S2, merging the calculation results of the plurality of nodes, verifying the user's information and issuing a permission access failure notification if the user does not have permission;

[0044] Step S3, obtaining the number of successful executions of the big data tasks within a number of collaborative cycles;

[0045] Step S4, judging the collaborative stability of distributed tasks based on the variance of the percentage of successful times of running big data tasks;

[0046] Step S5, if it is determined that the stability does not meet the requirements, the storage capacity of the subtask data block is adjusted or a verification validity determination process of the user information is performed;

[0047] Step S6: If it is determined that the effectiveness does not meet the requirement, the interval between adjacent concurrent processing batches is adjusted or the number of shards of the subtask data is adjusted based on the average packet loss rate of the subtask data.

[0048] Specifically, big data tasks include the storage space for task data, the type of task data, and the storage location of task data.

[0049] Specifically, the types of task data include user transaction records, log files of task data, and monitoring data of task data.

[0050] Specifically, the subtask data includes the storage location of the subtask data, the partition identifier of the subtask data, and the type of the subtask data.

[0051] Specifically, the type of subtask data includes the format of the subtask data, the creation time of the subtask data, and the online time of the user.

[0052] Specifically, the successful operation of a big data task means that there is no data loss or data conflict in the process of splitting, distributing, calculating, and merging the calculation results of the big data task submitted by the user.

[0053] Specifically, the process of verifying user information is that after the user enters information, the role access control mechanism is triggered to confirm the information. If the information matches the corresponding information in the mechanism, the user is deemed to have authority, that is, the verification is successful; if the information does not match the corresponding information in the mechanism or there is no corresponding information, the user is deemed to have no authority, that is, the verification fails.

[0054] Specifically, user information includes user name, user password, and user contact information.

[0055] Specifically, the storage capacity of a subtask data block means the storage space size of the subtask data blocks where the data of a large data task is split into several subtasks.

[0056] Specifically, several subtask data are stored in several subtask data blocks respectively.

[0057] Specifically, the interval between adjacent concurrently processed batches means the interval between two consecutive batches of concurrent processing operations on subtask data.

[0058] In implementation, the method of the present invention adjusts the storage capacity of the subtask data block according to the variance of the percentage of successful times of the big data task running by setting steps S1-S6. Since there is a relationship between some data in the task data to be split, the data with the relationship is split during the splitting, resulting in the task being unable to run correctly, thereby causing the task collaboration failure. By increasing the storage capacity of the subtask data block, the amount of data contained in each data block is increased, and the risk of incorrect splitting of the data with the relationship is reduced. By adjusting the interval length of adjacent concurrent processing batches according to the percentage of the number of user verification failures, since in the process of concurrent processing of data, two nodes may have the same The data is processed, and the final processing results are different, resulting in data conflicts, which in turn leads to the failure of user permission verification. By reducing the interval between adjacent concurrent processing batches, the probability of data conflicts is reduced, thereby improving the effectiveness of user information verification. By adjusting the number of shards of subtask data according to the average packet loss rate of subtask data, the network is congested due to the large amount of task data processed at the same time, resulting in the inability to establish effective contact with the node, which in turn leads to data distribution interruption, resulting in an increase in the data packet loss rate. By increasing the number of shards of subtask data, the size of a single data packet is reduced, the probability of data packet loss is reduced, and the collaborative stability of distributed tasks is improved.

[0059] Specifically, adjusting the storage capacity of the subtask data block includes:

[0060] Get the number of successful executions of big data tasks within several collaborative cycles;

[0061] The variance of the proportion of the successful number of big data task operations is calculated based on the number of successful number of big data task operations in the several collaborative cycles and the total number of big data task operations in the several collaborative cycles.

[0062] Compare the variance of the percentage of successful times of the big data task execution with the preset first variance and the preset second variance respectively;

[0063] If the variance of the percentage of successful times of the big data task execution is greater than the preset first variance, it is determined that the collaborative stability of the distributed task does not meet the requirements.

[0064] It can be understood that the three intervals corresponding to the preset first variance Q1 and the preset second variance Q2 correspond to three situations respectively. The first situation is Q≤Q1, and the collaborative stability of the distributed task is judged to meet the requirements; the second situation is Q1<Q≤Q2. In the process of concurrent data processing, two nodes may process the same data, and the final processing results are different, resulting in data conflicts, which leads to the failure of user permission verification; the third situation is Q>Q2. There is a relationship between some data in the task data that needs to be split. When splitting, the data with a relationship is split, resulting in the inability to run the task correctly, resulting in the failure of task collaboration. In practice, the range of Q1 is generally [0.65, 0.75], and the range of Q2 is generally [0.76, 0.86].

[0065] Preferably, the first variance Q1 is preset to be 0.7, and the second variance Q2 is preset to be 0.8.

[0066] Specifically, the variance of the percentage of successful runs of big data tasks is recorded as Q.

[0067] Specifically, the variance of the percentage of successful times of big data task running is the variance of the percentage of successful times of big data task running within several collaborative cycles. The calculation method of the variance of the percentage of successful times of big data task running is a conventional technical means well known to technical personnel in this field. Therefore, the calculation process of the variance of the percentage of successful times of big data task running will not be repeated here.

[0068] Specifically, the calculation formula for the percentage of successful runs of big data tasks is:

[0069]

[0070] Among them, G is the percentage of successful runs of big data tasks, K1 is the number of successful runs of big data tasks, and K2 is the total number of runs of big data tasks.

[0071] In implementation, the method of the present invention determines the collaborative stability of distributed tasks by setting a preset first variance and a preset second variance, thereby reducing the impact of the decreased collaborative effectiveness of distributed tasks due to inaccurate judgment of the collaborative stability of distributed tasks, and further achieving an improvement in the collaborative stability of distributed tasks.

[0072] Specifically, determining the verification validity of the user information includes:

[0073] Comparing the variance of the percentage of successful times of the big data task execution with the preset first variance and the preset second variance respectively;

[0074] If the variance of the percentage of successful times of the big data task running is greater than the preset first variance and less than or equal to the preset second variance, it is preliminarily determined that the verification validity of the user information does not meet the requirements, and a secondary determination is made on the verification validity of the user information based on the percentage of failed times of user verification.

[0075] Specifically, if the variance of the percentage of successful times of the big data task execution is greater than the preset second variance, the storage capacity of the subtask database is increased.

[0076] Specifically, the increase in the storage capacity of the subtask data block is determined by the difference between the variance of the percentage of successful times of the big data task running and a preset second variance.

[0077] Specifically, the calculation formula for the storage capacity of the enlarged subtask data block is:

[0078]

[0079] Among them, V' is the storage capacity of the increased subtask data block, V is the storage capacity of the current subtask data block, A is the minimum unit variance for adjusting the storage capacity of the subtask data block, and v is the storage capacity of the subtask data block when the variance of the percentage of successful times of the big data task running is the minimum unit variance.

[0080] In practice, the value of A is generally 0.7 to 0.8, and preferably, the preferred embodiment of A is 0.75.

[0081] In implementation, the optional value range of v is [1500Byte, 2500Byte]. Preferably, when the storage capacity of the subtask data block is less than 2000Byte, the task may not be able to run normally. Therefore, the preferred embodiment of v is 2000Byte.

[0082] Specifically, when the value of V' is a decimal, the value of V' is automatically rounded up to an integer.

[0083] For example, a distributed task collaboration method based on big data is used to process large-scale data, where V = 1800 Byte, Q = 0.9, and the calculation is

[0084] In implementation, the method of the present invention adjusts the storage capacity of the subtask data block by setting a preset first variance and a preset second variance. Since there is a relationship between some data in the task data that needs to be split, the data with the relationship is split apart during the splitting, resulting in the task being unable to run correctly, thereby causing the task collaboration failure. By increasing the storage capacity of the subtask data block, the amount of data contained in each data block is increased, reducing the risk of incorrect splitting of data with relationships, and further achieving improved collaborative stability for distributed tasks.

[0085] Specifically, adjusting the interval between adjacent concurrent processing batches according to the proportion of the number of user authentication failures includes:

[0086] Obtain the number of user authentication failures and the total number of user authentications in several authentication cycles respectively, and calculate the proportion of the number of user authentication failures;

[0087] Compare the proportion of the number of failed user authentications with the preset first proportion and the preset second proportion respectively;

[0088] If the proportion of the number of times the user verification fails is greater than the preset first proportion, it is determined that the verification validity of the user information does not meet the requirements;

[0089] If the proportion of the number of failed user authentications is greater than the preset first proportion and less than or equal to the preset second proportion, the interval between adjacent concurrent processing batches is increased;

[0090] If the proportion of the number of user verification failures is greater than the preset second proportion, it is preliminarily determined that the distribution validity of the subtask data does not meet the requirements, and a secondary determination is made on the distribution validity of the subtask data based on the average packet loss rate of the subtask data.

[0091] It can be understood that the three intervals corresponding to the preset first proportion P1 and the preset second proportion P2 correspond to three situations respectively. The first situation is P≤P1, and the secondary determination of the verification validity of the user information meets the requirements; the second situation is P1<P≤P2. In the process of concurrent data processing, two nodes may process the same data, and the final processing results are different, resulting in data conflicts, which leads to the failure of user authority verification; the third situation is P>P2. The amount of task data processed at the same time is too large, resulting in network congestion, making it impossible to establish an effective connection with the node, and then resulting in data distribution interruption, which leads to an increase in the data packet loss rate. In practice, the range of P1 is generally [0.1, 0.3], and the range of P2 is generally [0.35, 0.45].

[0092] Preferably, the first proportion P1 is preset to be 0.2, and the second proportion P2 is preset to be 0.4.

[0093] Specifically, the percentage of user verification failures is denoted as P.

[0094] In implementation, the method of the present invention performs a secondary judgment on the verification validity of user information by setting a preset first ratio and a preset second ratio, thereby reducing the impact of the decreased collaborative stability of distributed tasks due to inaccurate secondary judgment on the verification validity of user information, and further improving the collaborative stability of distributed tasks.

[0095] Specifically, the proportion of the number of user verification failures is the ratio of the number of user verification failures in a number of verification cycles to the total number of user verifications.

[0096] Specifically, the increase range of the interval duration between adjacent concurrently processed batches is determined by the difference between the proportion of the number of user authentication failures and a preset first proportion.

[0097] Specifically, the calculation formula for the increased interval between adjacent concurrent processing batches is:

[0098]

[0099] Among them, H' is the increased interval length between adjacent concurrent processing batches, H is the current interval length between adjacent concurrent processing batches, B is the minimum unit ratio for adjusting the interval length between adjacent concurrent processing batches, and h is the interval length between adjacent concurrent processing batches when the number of user verification failures accounts for the minimum unit ratio.

[0100] In practice, the value of B is generally 0.4 to 0.6, preferably, the preferred embodiment of B is 0.5.

[0101] In implementation, the optional value range of h is [0.04s, 0.06s]. Preferably, when the interval between adjacent concurrently processed batches is less than 0.05s, data conflicts may occur. Therefore, the preferred embodiment of h is 0.05s.

[0102] Specifically, when the value of H' is a decimal, the value of H' is automatically rounded up to an integer.

[0103] For example, a distributed task collaboration method based on big data is used to process large-scale data, where H = 0.03s, P = 0.5, and the calculation is

[0104] In implementation, the method of the present invention adjusts the interval duration of adjacent concurrent processing batches by setting a preset first ratio and a preset second ratio. Since two nodes may process the same data during concurrent data processing operations, the final processing results may be different, resulting in data conflicts, which in turn cause the user's authority verification to fail. By reducing the interval duration of adjacent concurrent processing batches, the probability of data conflicts is reduced, thereby improving the effectiveness of user information verification and further improving the collaborative stability of distributed tasks.

[0105] Specifically, adjusting the number of shards of the subtask data according to the average packet loss rate of the subtask data includes:

[0106] Obtain the packet loss rate of subtask data in several distribution cycles, and calculate the average packet loss rate of subtask data;

[0107] Comparing the average packet loss rate of the subtask data with a preset packet loss rate;

[0108] If the average packet loss rate of the subtask data is greater than the preset packet loss rate, it is determined that the distribution validity of the subtask data does not meet the requirements, and the number of fragments of the subtask data is increased.

[0109] It can be understood that the two intervals corresponding to the preset packet loss rate Y0 correspond to two situations. The first situation is Y≤Y0, and the secondary judgment of the distribution validity of the subtask data meets the requirements; the second situation is Y>Y0, and the amount of task data processed at the same time is too large, resulting in network congestion, making it impossible to establish an effective connection with the node, and then resulting in data distribution interruption, which leads to an increase in the data packet loss rate. In practice, Y0 is generally selected in the range of [3%, 7%].

[0110] Preferably, the preset packet loss rate Y0=5%.

[0111] Specifically, the average packet loss rate of subtask data is recorded as Y.

[0112] Specifically, the calculation formula for the average packet loss rate of subtask data is:

[0113]

[0114] Among them, Y is the average packet loss rate of subtask data, X a is the packet loss rate of the subtask data in the ath distribution cycle, n is the number of distribution cycles, and n is a natural number greater than or equal to 1.

[0115] In implementation, the method of the present invention performs a secondary judgment on the distribution validity of subtask data by setting a preset packet loss rate, thereby reducing the impact of the decreased collaborative stability of distributed tasks due to inaccurate secondary judgment on the distribution validity of subtask data, and further improving the collaborative stability of distributed tasks.

[0116] Specifically, the increase range of the number of fragments of the subtask data is determined by the difference between the average packet loss rate of the subtask data and the preset packet loss rate.

[0117] Specifically, the calculation formula for the number of shards of the increased subtask data is:

[0118]

[0119] Among them, L' is the number of shards of the increased subtask data, L is the current number of shards of the subtask data, C is the minimum unit packet loss rate for adjusting the number of shards of the subtask data, and l is the number of shards of the subtask data when the average packet loss rate of the subtask data is the minimum unit packet loss rate.

[0120] In practice, the value of C is generally 3% to 5%, and preferably, the preferred embodiment of C is 4%.

[0121] In implementation, the optional range of the value of l is [70, 90]. Preferably, when the number of shards of the subtask data is less than 80, the packet loss rate may increase. Therefore, the preferred embodiment of l is 80.

[0122] Specifically, when the value of L' is a decimal, the value of L' is automatically rounded up to an integer.

[0123] For example, a distributed task collaboration method based on big data is used to process large-scale data, where L = 60, Y = 7%, and the calculation is

[0124] In implementation, the method of the present invention adjusts the number of shards of subtask data by setting a preset packet loss rate. Since the amount of task data processed simultaneously is too large, the network is blocked, and it is impossible to establish an effective connection with the node, which leads to interruption of data distribution, thereby increasing the packet loss rate of the data. By increasing the number of shards of subtask data, the size of a single data packet is reduced, the probability of data packet loss is reduced, and the collaborative stability of distributed tasks is further improved.

[0125] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A distributed task collaboration method based on big data, characterized in that: include: Step S1, splitting a big data task submitted by any user into a number of subtask data, and distributing the subtask data to a number of nodes for collaborative computing; Step S2, merging the calculation results of the plurality of nodes, verifying the user's information and issuing a permission access failure notification when the user does not have permission; Step S3, obtaining the number of successful executions of the big data tasks within a number of collaborative cycles; Step S4, judging the collaborative stability of distributed tasks based on the variance of the percentage of successful times of running big data tasks; Step S5, if it is determined that the stability does not meet the requirements, the storage capacity of the subtask data block is adjusted or a verification validity determination process of the user information is performed; Step S6, if it is determined that the validity does not meet the requirement, the interval between adjacent concurrent processing batches is adjusted or the number of shards of the subtask data is adjusted based on the average packet loss rate of the subtask data; Adjusting the storage capacity of the subtask data block includes: Get the number of successful executions of big data tasks within several collaborative cycles; The variance of the proportion of the successful number of big data task operations is calculated based on the number of successful number of big data task operations in the several collaborative cycles and the total number of big data task operations in the several collaborative cycles. Compare the variance of the percentage of successful times of the big data task execution with the preset first variance and the preset second variance respectively; If the variance of the percentage of successful times of the big data task running is greater than the preset first variance, it is determined that the collaborative stability of the distributed task does not meet the requirements; Determining the verification validity of the user information includes: Comparing the variance of the percentage of successful times of the big data task execution with the preset first variance and the preset second variance respectively; If the variance of the percentage of successful times of the big data task running is greater than the preset first variance and less than or equal to the preset second variance, it is preliminarily determined that the verification validity of the user information does not meet the requirements, and a secondary determination is made on the verification validity of the user information based on the percentage of failed times of user verification; If the variance of the percentage of successful times of the big data task execution is greater than the preset second variance, the storage capacity of the subtask database is increased.

2. The distributed task collaboration method based on big data according to claim 1 is characterized in that: The increase in the storage capacity of the subtask data block is determined by the difference between the variance of the percentage of the number of successful executions of the big data task and a preset second variance.

3. The distributed task collaboration method based on big data according to claim 2 is characterized in that: Adjusting the interval between adjacent concurrent processing batches according to the proportion of user authentication failures includes: Obtain the number of user authentication failures and the total number of user authentications in several authentication cycles respectively, and calculate the proportion of the number of user authentication failures; Compare the proportion of the number of failed user authentications with the preset first proportion and the preset second proportion respectively; If the proportion of the number of times the user verification fails is greater than the preset first proportion, it is determined that the verification validity of the user information does not meet the requirements; If the proportion of the number of failed user authentications is greater than the preset first proportion and less than or equal to the preset second proportion, the interval between adjacent concurrent processing batches is increased; If the proportion of the number of user verification failures is greater than the preset second proportion, it is preliminarily determined that the distribution validity of the subtask data does not meet the requirements, and a secondary determination is made on the distribution validity of the subtask data based on the average packet loss rate of the subtask data.

4. The distributed task collaboration method based on big data according to claim 3 is characterized in that: The proportion of the number of user authentication failures is the ratio of the number of user authentication failures in a number of authentication cycles to the total number of user authentications.

5. The distributed task collaboration method based on big data according to claim 4 is characterized in that: The increase range of the interval time between adjacent concurrently processed batches is determined by the difference between the proportion of the number of user authentication failures and a preset first proportion.

6. The distributed task collaboration method based on big data according to claim 5 is characterized in that: Adjusting the number of shards of the subtask data according to the average packet loss rate of the subtask data includes: Obtain the packet loss rate of subtask data in several distribution cycles, and calculate the average packet loss rate of subtask data; Comparing the average packet loss rate of the subtask data with a preset packet loss rate; If the average packet loss rate of the subtask data is greater than the preset packet loss rate, it is determined that the distribution validity of the subtask data does not meet the requirements, and the number of fragments of the subtask data is increased.

7. The distributed task collaboration method based on big data according to claim 6 is characterized in that: The increase range of the number of fragments of the subtask data is determined by the difference between the average packet loss rate of the subtask data and the preset packet loss rate.

Images