Data segmentation method, device, equipment and computer readable storage medium

Abstract

The embodiment of the present invention relates to the field of big data technology, and discloses a data segmentation method, including: obtaining the current segment length of the data to be processed; determining the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula characterizes that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; based on the current segment length and the current overlapping data length, generating segmented data corresponding to the data to be processed. The embodiment of the present invention achieves the effect of avoiding the decrease of the boundary precision of data analysis and improving the precision of the result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and specifically to a data segmentation method, a big data task processing method, an apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] At present, most of the Internet data is strongly related to time, and usually has a certain regularity with time. This type of data is referred to as time data. In the prior art, two methods are usually used to process this data: (1) The time data is not segmented, but directly analyzed as a whole through a data mining algorithm, and then used for prediction or decision-making. The advantage of this method is that the analysis results are accurate, but the disadvantage is that the calculation time is long for huge amounts of time data, and even the result may crash and fail to be calculated. (2) The time data is segmented, then partially analyzed through a data mining algorithm, and then the results are merged for prediction or decision-making. The advantage of this method is that it uses parallel computing and distributed storage to improve the computing speed, and it will not crash even if the amount of data is large. However, the inventors found in the process of implementing the present invention that this method has the following disadvantages: Since the time is segmented, each data segment has a time start point and a time end point. When the data mining algorithm is calculated, the accuracy of the result calculation of the data near the boundary is reduced, thereby affecting the accuracy of the result. Summary of the invention

[0003] In view of the above problems, an embodiment of the present invention provides a data segmentation method, a big data task processing method, an apparatus, a computing device and a computer-readable storage medium, which are used to solve the technical problem of low accuracy of data calculation results near the boundaries of the existing time data segmentation method in the prior art.

[0004] According to one aspect of an embodiment of the present invention, a data segmentation method is provided, the method comprising:

[0005] Get the current segment length of the data to be processed;

[0006] Determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length;

[0007] Based on the current segment length and the current overlapping data length, segment data corresponding to the data to be processed is generated.

[0008] In an optional manner, the method further includes: performing calculation based on the segmented data corresponding to the data to be processed to obtain segmented result data corresponding to the segmented data corresponding to the data to be processed;

[0009] Determine overlapping data in the segmented data corresponding to the data to be processed; the length of the overlapping data is the current overlapping data length;

[0010] Determine the overlapping data result corresponding to the overlapping data in the segmentation result data;

[0011] Removing the overlapping data results to obtain result data;

[0012] Verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length, and re-execute the step of determining the current overlapping data length based on the adjusted current segment length to generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, and obtain the target segment length and target overlapping data length of the data to be processed.

[0013] In an optional manner, obtaining the current segment length of the data to be processed includes: determining the current segment length according to disk performance parameters, memory information and algorithm complexity.

[0014] In an optional manner, the segmented data corresponding to the data to be processed is generated based on the current segmentation length and the current overlapping data length, including: dividing the data to be processed based on the current segmentation length to obtain a plurality of segmented data; splicing each segmented data with the overlapping data of the current overlapping data length to obtain the segmented data corresponding to the data to be processed; the overlapping data is the data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

[0015] In an optional manner, the preset relationship formula is:

[0016]

[0017] Among them, P represents the current segment length, K represents the current overlapping data length; and m represents the length of the data to be processed.

[0018] In an optional manner, before obtaining the current segment length of the data to be processed, the method further includes:

[0019] Determine a time cost function according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed;

[0020] Determine a space cost function according to the current segment length and the current overlapping data length; the space cost function represents the space cost of storing the data to be processed;

[0021] Determining a time-space efficiency function according to the time cost function and the space cost function;

[0022] Finding the optimal solution of the time-space efficiency function, thereby determining the relationship between the current segment length and the current overlapping data length and the length of the data to be processed;

[0023] The preset relationship formula is obtained according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed.

[0024] In an optional manner, verifying the accuracy of the result data includes:

[0025] Sampling a plurality of single-point data from the data to be processed;

[0026] Performing parallel calculations on the multiple single-point data to obtain calculation results of the multiple single-point data;

[0027] The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data.

[0028] According to another aspect of an embodiment of the present invention, a method for processing a big data task is provided, including:

[0029] Determine the target segment length and target overlapping data length corresponding to the data to be processed by the data segmentation method;

[0030] Dividing the data to be processed according to the target segment length and the target overlapping data length to generate a plurality of target segment data;

[0031] Calculate the multiple target segmented data to obtain a target result.

[0032] According to another aspect of an embodiment of the present invention, there is provided a data segmentation device, comprising:

[0033] An acquisition module is used to obtain the current segment length of the data to be processed;

[0034] A determination module, used to determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length;

[0035] A generating module is used to generate segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length.

[0036] According to another aspect of an embodiment of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations such as the data segmentation method or the big data task processing method.

[0038] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, and when the executable instruction is executed on a computing device, the computing device executes the operation of the data segmentation method or the operation of the big data task processing method.

[0039] The embodiment of the present invention adds duplicate data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; based on the current segment length and the current overlapping data length, generates segmented data corresponding to the data to be processed. This makes the segmentation of the data to be processed more reasonable and avoids the problem of decreased accuracy of data analysis boundaries.

[0040] Furthermore, calculations are performed based on the segmented data corresponding to the data to be processed, and overlapping data results are removed to obtain result data; the accuracy of the result data is verified, and when the accuracy is less than a preset threshold, the current segment length is adjusted, and based on the adjusted current segment length, the step of determining the current overlapping data length is re-executed to generate segmented data corresponding to the data to be processed; calculations are performed based on the segmented data corresponding to the data to be processed, and overlapping data results are removed to obtain result data and verify the accuracy of the result data; when the accuracy is less than a preset threshold, the step of adjusting the current segment length is performed until the accuracy is greater than or equal to the preset threshold, and the target segment length and target overlapping data length of the data to be processed are obtained, thereby improving the accuracy of the result.

[0041] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of a flow chart of a data segmentation method provided by an embodiment of the present invention is shown;

[0044] Figure 2 A schematic diagram of overlapping division of time data provided by an embodiment of the present invention is shown;

[0045] Figure 3 A schematic diagram of a process flow of a method for processing a big data task provided by an embodiment of the present invention is shown;

[0046] Figure 4 A schematic diagram showing the structure of a data segmentation device provided by an embodiment of the present invention is shown;

[0047] Figure 5 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] First, the technical terms appearing in the embodiments of the present invention are explained:

[0050] Time series data: data collected at different times, used to describe the changes of phenomena over time. This type of data reflects the state or degree of change of a thing or phenomenon over time.

[0051] Figure 1 The flowchart of the data segmentation method provided by the embodiment of the present invention is shown, and the method is executed by a computing device. The computing device can be a computer device, such as a personal computer, a tablet computer, a desktop computer, a cluster device, a distributed system, etc., or a terminal device. Figure 1 As shown, the method comprises the following steps:

[0052] Step 110: Obtain the current segment length of the data to be processed.

[0053] In the embodiment of the present invention, the data to be processed is time series data, and the time series data includes a timestamp. Before the data to be processed is analyzed and calculated by the data mining algorithm, the data to be processed is segmented to obtain a plurality of segmented data, and then the data mining algorithm is analyzed and calculated. Since the length of the data segment affects the calculation time and storage space, it is necessary to determine the optimal segment length of the data to be processed.

[0054] Among them, Figure 2 As shown, in the segmented data after segmentation in the prior art, the data calculation of the boundary points has a certain correlation with the data before and after, that is, the data in front may affect the calculation result of the data behind. Therefore, in order to improve the accuracy of the calculation of the data at the boundary points, the embodiment of the present invention performs segmentation processing on the data to be processed, and the obtained segmented data includes the segmented data and overlapping data after the data to be processed is directly segmented. Taking time data as an example, the data to be processed is the original time data. After segmentation, 3 segments of segmented time data (segmented time data 1, segmented time data 2 and segmented time data 3) are obtained, and the original time data = segmented time data 1 + segmented time data 2 + segmented time data 3. After determining the length of overlapping data, the overlapping data is added before and after each segmented time data. For segmented time data 1, overlapping data is added on its right side, and the overlapping data is data selected from segmented time data 2 and located after segmented time data 1 with a preset overlapping data length; for segmented time data 2, overlapping data is added on its left and right sides respectively, and the overlapping data is data selected from segmented time data 1 and located before segmented time data 2 with a preset overlapping data length, and data selected from segmented time data 3 and located after segmented time data 2 with a preset overlapping data length; for segmented time data 3, overlapping data is added on its left side, and the overlapping data is data selected from segmented time data 2 and located before segmented time data 3 with a preset overlapping data length. In this way, data exists before and after the boundary data of each segmented data, so that the calculation result of the boundary data becomes accurate. The segment boundary accuracy is improved by overlapping data between segmented data. At the same time, each segment has a certain continuity, and the connection between each segment is retained.

[0055] Therefore, the embodiment of the present invention needs to determine the current segment length and the overlapping data length. First, it is obvious that when the data segment length P is fixed, the larger the overlapping data length K is, the higher the accuracy is, but the more computing time is spent, and the greater the storage space ratio is wasted; conversely, the smaller the overlapping data length K is, the lower the accuracy is, but the higher the time and space efficiency is. If the overlapping data length K is fixed, the larger the data segment length P is, the higher the accuracy is, the more computing time is spent, and the smaller the storage space ratio is wasted, and the smaller the data segment length P is, the lower the accuracy is, the smaller the computing time is spent, and the greater the storage space ratio is wasted. Therefore, it is necessary to find the optimal data segment length P and the optimal overlapping data length K, so as to achieve reasonable segmentation of the data to be processed.

[0056] Wherein, when initially determining the current segment length and the current overlapping data length, the current segment length may be determined based on the disk performance parameters, memory information, and algorithm complexity of the computing device. Specifically, the disk performance parameters of the computing device include disk IO performance parameters, CPU, memory, and the complexity of the data mining algorithm to determine an initial segment length as the current segment length, for example, the initial segment length may be 128MB (MByte).

[0057] Step 120: Determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length.

[0058] After the current segment length is determined, the current overlapping data length may be determined according to a preset relationship formula. In one implementation, the preset relationship formula is:

[0059]

[0060] Wherein, P represents the current segment length, K represents the current overlapping data length, and m represents the length of the data to be processed. According to the preset relationship formula, after obtaining data m of a certain length, the user can determine the segment length P to be segmented according to the specific needs of the corresponding scene, thereby determining the overlapping data length K.

[0061] The preset relationship formula is derived in the following way:

[0062] Assuming that the current segment length is P, the overlapping data length is K, the length of the data to be processed is m, and the length of each segment data corresponding to the data to be processed after the data to be processed is split is n, then we can get: n=P+2K, there are a total of m / P segments of segment data corresponding to the data to be processed.

[0063] The time cost function is determined according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed. The time cost function is shown in formula (1):

[0064]

[0065] Wherein, T(n) is the processing time of a single segmented data corresponding to the data to be processed, T(m) is the processing time of the original data to be processed, and C is the consumption of non-task processing time (such as startup time, etc.). Usually C is less than T(n) and much less than T(m). Cost T (P) is the time efficiency, because the complexity of the time data mining algorithm is generally very high, much greater than O(n) (there exists n0, when n>n0, the time complexity function f(n)≤an, a is a constant, and f(n)=O(n)). Assume that the size of m is 4, p=1, k=0.1, C=0.1, and the complexity of the time data mining algorithm T(x) is O(n 2 ), then the denominator of formula (1) is T(m)+C=4*4+0.1=16.1, and the numerator is m / p(T(n)+C)=4*(1.3*1.3+0.1)=7.16. Because the square of the sum is greater than the sum of the squares, Cost T (P) has a value range of (0, 1), Cost T The larger the (P) value, the longer the time. Generally, the larger the P, the higher the time cost.

[0066] The spatial cost function is determined according to the current segment length and the current overlapping data length; the spatial cost function represents the spatial cost of storing the data to be processed. The spatial cost function is shown in formula (2):

[0067]

[0068] in, It is the space ratio increased by the overlapping storage of the segmented standby. Because the effective storage space length P should be greater than the overlapping storage data length K, it can be deduced that Because Cost in formula (1) T (P) has been normalized, so the spatial cost function must also be normalized, and finally we get equation (2). The final spatial cost function Cost S The value range of (P) is also (0, 1).

[0069] Therefore, taking into account the time and space efficiency, the time and space efficiency function is determined according to the time cost function and the space cost function. The final time and space efficiency function is shown in formula (3):

[0070] Cost(P)=εCost T (P)+(1-ε)Cost S (P) (3)

[0071] Among them, ε is the weight of time and space, and the value range is 0<ε<1. Therefore, the problem of finding the optimal data segment length P and the optimal overlapping data length K is transformed into: when m>>n≥P≥2K, solve the optimal solution of Cost(P), that is, when P and K take what values, Cost(P) takes the minimum value. That is, find the optimal solution of the time-space efficiency function, so as to determine the relationship between the current segment length and the current overlapping data length and the length of the data to be processed.

[0072] The algorithm complexity of the embodiment of the present invention is O(n 2 ) as an example to deduce the solution.

[0073] Take the partial derivative of the above formula (3), when When , we can get formula (4):

[0074]

[0075] when When , we can get formula (5):

[0076]

[0077] According to formula (3), (4) and formula (5), (P best , ε best ) is the stationary point, which can make Cost(P) take the minimum value. In reality, the size of the temporal big data m is much larger than the current segment length P, and the time for data analysis and calculation is much longer than the time C of framework startup, so formula (4) can be approximately solved as formula (6), and formula (5) can be approximately solved as formula (7).

[0078]

[0079]

[0080] It can be seen from formula (7) that the larger m is, the larger ε is. In other words, the larger the amount of data is, the greater the weight on time efficiency is, and the higher the requirement for time efficiency is.

[0081] Because m is much larger than K, according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed, formula (6) can be further solved to obtain the preset relationship formula, such as formula (8):

[0082]

[0083] The embodiment of the present invention obtains the relationship among the current segment length P, the overlapping data length K, and the length m of the data to be processed through the above derivation, so that the setting of the current segment length P and the overlapping data length K is more reasonable.

[0084] Step 130: Generate segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length.

[0085] After determining the current segment length and the current overlapping data length, the following is used: Figure 2 In the manner shown, the data to be processed is segmented based on the current segment length to obtain a plurality of segmented data; each segmented data is spliced ​​with the overlapping data of the current overlapping data length, thereby obtaining the segmented data corresponding to the data to be processed. In an embodiment of the present invention, the overlapping data is data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

[0086] In an embodiment of the present invention, after obtaining the segmented data corresponding to the data to be processed, each segmented data corresponding to the data to be processed can be input into a data mining algorithm through a distributed system for parallel calculation to obtain the calculation results of each segmented data corresponding to the data to be processed. However, since the current segmentation length and overlapping data length may be set according to user needs during the first segmentation, although the accuracy of big data calculation is improved to a certain extent by setting overlapping data, due to the correlation between time-dependent data, the current setting of overlapping data or the segmentation of the current segmentation length may not be reasonable enough to meet the requirements for calculation accuracy. Therefore, it is necessary to verify the result data to determine the calculation accuracy of the segmented data after the data to be processed is segmented with the current segmentation length and the current overlapping data length. Specifically, the data segmentation method of the present invention also includes steps 140 and 150.

[0087] Step 140: performing calculations based on the segmented data corresponding to the data to be processed, removing overlapping data results, and obtaining result data.

[0088] After obtaining a plurality of segmented data corresponding to the data to be processed, each segmented data corresponding to the data to be processed can be input into a data mining algorithm through a distributed system for parallel calculation to obtain calculation results of each segmented data corresponding to the data to be processed. The data mining algorithm refers to the algorithm used when data mining is required based on the data to be processed.

[0089] There are overlapping data results corresponding to overlapping data in the calculation results of each segmented data corresponding to the data to be processed. The overlapping data itself is redundant data, and the corresponding calculation results can be deleted. Therefore, the overlapping data results in the calculation results of each segmented data corresponding to the data to be processed can be deleted, and the calculation results after deleting the overlapping data results are merged to obtain the result data of the data to be processed. Wherein, based on the segmented data corresponding to the data to be processed, the calculation is performed to obtain the segmented result data corresponding to the segmented data corresponding to the data to be processed; the overlapping data in the segmented data corresponding to the data to be processed is determined; the length of the overlapping data is the current overlapping data length; and the overlapping data results corresponding to the overlapping data in the segmented result data are determined. Specifically, for time series data, each time data has a timestamp or a timestamp, so the timestamp or the timestamp can be used to determine whether the data is the added overlapping data or the segmented data itself after the data to be processed is segmented.

[0090] Step 150: verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length, and re-execute the step of determining the current overlapping data length based on the adjusted current segment length to generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, and obtain the target segment length and target overlapping data length of the data to be processed.

[0091] In the embodiment of the present invention, after the result data of the data to be processed is obtained, the result data needs to be verified to determine the calculation accuracy of the segmented data after the data to be processed is segmented with the current segmentation length and the current overlapping data length.

[0092] The accuracy of the result data can be verified by sampling verification, specifically:

[0093] Sampling a plurality of single-point data from the data to be processed;

[0094] Inputting the plurality of single point data into a data mining algorithm for parallel calculation to obtain calculation results of the plurality of single point data;

[0095] The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data. That is, the calculation result of the single point data is compared with the result data of the corresponding single point data in the result data for similarity, so as to determine the accuracy of the result data.

[0096] Among them, a plurality of single-point data are sampled from the data to be processed. The plurality of single-point data may be obtained by uniform sampling, and the plurality of single-point data near the boundary point may be uniformly collected.

[0097] In the embodiment of the present invention, when the accuracy is less than the preset threshold, the current segment length is adjusted, and the current segment length can be increased or decreased, and the current overlapping data length is re-determined according to the adjusted current segment length, and the accuracy of the result data is re-verified. When the accuracy is greater than or equal to the preset threshold, the adjustment is completed, and the current segment length after the last adjustment is used as the target segment length, and the current overlapping data length corresponding to the current segment length after the last adjustment is determined as the target overlapping data length. Among them, the embodiment of the present invention does not specifically limit the specific value of the preset threshold, and technicians in this field can make corresponding settings according to specific scenarios, such as a value such as 99% similarity.

[0098] After the target segment length and the target overlapping data length of the data to be processed are obtained, step 130 can be used to complete the segmentation process of the data to be processed according to the target segment length and the target overlapping data length.

[0099] The embodiment of the present invention adds repeated data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; generates segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; performs calculation based on the segmented data corresponding to the data to be processed, removes the overlapping data result, and obtains the result data; verifies the accuracy of the result data, and when the accuracy is less than a preset threshold, adjusts the current segment length. Segment length, and according to the adjusted current segment length, re-execute the step of determining the current overlapping data length, generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data, when the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, obtain the target segment length and target overlapping data length of the data to be processed, thereby making the segmentation of the data to be processed more reasonable, avoiding the problem of decreased accuracy of data analysis boundaries, and improving the accuracy of the results.

[0100] Figure 3 The big data task processing method provided by the embodiment of the present invention is shown. Figure 3 As shown, the method comprises the following steps:

[0101] Step 210: Determine the target segment length and target overlapping data length corresponding to the data to be processed by the data segmentation method.

[0102] The specific method steps for determining the target segment length and the target overlapping data length corresponding to the to-be-processed data by the data segmentation method in the embodiment of the present invention are basically the same as those in the above embodiment and will not be repeated here.

[0103] Step 220: Segment the data to be processed according to the target segment length and the target overlapping data length to generate a plurality of target segment data.

[0104] The specific implementation method of step 220 in the embodiment of the present invention is consistent with that of step 130 in the above method embodiment, and will not be repeated here.

[0105] Step 230: Calculate the multiple target segmented data to obtain a target result.

[0106] Among them, multiple target segmented data are input into the data mining algorithm for parallel calculation to obtain the target result.

[0107] The embodiment of the present invention adds repeated data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; generates segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; performs calculation based on the segmented data corresponding to the data to be processed, removes the overlapping data result, and obtains the result data; verifies the accuracy of the result data, and when the accuracy is less than a preset threshold, adjusts the current segment length. Segment length, and according to the adjusted current segment length, re-execute the step of determining the current overlapping data length, generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data, when the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, obtain the target segment length and target overlapping data length of the data to be processed, thereby making the segmentation of the data to be processed more reasonable, avoiding the problem of decreased accuracy of data analysis boundaries, and improving the accuracy of the results.

[0108] Figure 4 FIG. 2 shows a schematic diagram of the structure of a data segmentation device provided by an embodiment of the present invention. Figure 4 As shown, the device 300 includes:

[0109] An acquisition module 310 is used to acquire the current segment length of the data to be processed;

[0110] A determination module 320, for determining the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length;

[0111] The generating module 330 is used to generate segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length.

[0112] In the embodiment of the present invention, the data segmentation device further includes a calculation module and a verification and adjustment module:

[0113] A calculation module, configured to perform calculation based on the segmented data corresponding to the data to be processed to obtain segmented result data corresponding to the segmented data corresponding to the data to be processed; determine overlapping data in the segmented data corresponding to the data to be processed; the length of the overlapping data is the current overlapping data length; determine overlapping data results corresponding to the overlapping data in the segmented result data; remove the overlapping data results to obtain result data;

[0114] A verification and adjustment module is used to verify the accuracy of the result data. When the accuracy is less than a preset threshold, the current segment length is adjusted, and based on the adjusted current segment length, the current overlapping data length is determined again to generate segment data corresponding to the data to be processed, and calculations are performed based on the segment data corresponding to the data to be processed to remove overlapping data results to obtain result data and verify the accuracy of the result data. When the accuracy is less than a preset threshold, the current segment length is adjusted until the accuracy is greater than or equal to the preset threshold, thereby obtaining a target segment length and a target overlapping data length of the data to be processed.

[0115] In an optional manner, obtaining the current segment length of the data to be processed includes: determining the current segment length according to disk performance parameters, memory information and algorithm complexity.

[0116] In an optional manner, the segmented data corresponding to the data to be processed is generated based on the current segmentation length and the current overlapping data length, including: dividing the data to be processed based on the current segmentation length to obtain a plurality of segmented data; splicing each segmented data with the overlapping data of the current overlapping data length to obtain the segmented data corresponding to the data to be processed; the overlapping data is the data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

[0117] In an optional manner, the preset relationship formula is:

[0118]

[0119] Among them, P represents the current segment length, K represents the current overlapping data length; and m represents the length of the data to be processed.

[0120] In an optional manner, before obtaining the current segment length of the data to be processed, the method further includes:

[0121] Determine a time cost function according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed;

[0122] Determine a space cost function according to the current segment length and the current overlapping data length; the space cost function represents the space cost of storing the data to be processed;

[0123] Determining a time-space efficiency function according to the time cost function and the space cost function;

[0124] Finding the optimal solution of the time-space efficiency function, thereby determining the relationship between the current segment length and the current overlapping data length and the length of the data to be processed;

[0125] The preset relationship formula is obtained according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed.

[0126] In an optional manner, verifying the accuracy of the result data includes:

[0127] Sampling a plurality of single-point data from the data to be processed;

[0128] Performing parallel calculations on the multiple single-point data to obtain calculation results of the multiple single-point data;

[0129] The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data.

[0130] The specific working process of the data segmentation device in the embodiment of the present invention is consistent with the specific method steps of the above method embodiment, and will not be repeated here.

[0131] The embodiment of the present invention adds repeated data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; generates segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; performs calculation based on the segmented data corresponding to the data to be processed, removes the overlapping data result, and obtains the result data; verifies the accuracy of the result data, and when the accuracy is less than a preset threshold, adjusts the current segment length. Segment length, and according to the adjusted current segment length, re-execute the step of determining the current overlapping data length, generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data, when the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, obtain the target segment length and target overlapping data length of the data to be processed, thereby making the segmentation of the data to be processed more reasonable, avoiding the problem of decreased accuracy of data analysis boundaries, and improving the accuracy of the results.

[0132] Figure 4 The schematic diagram of the structure of the computing device provided by the embodiment of the present invention is shown, and the specific embodiment of the present invention does not limit the specific implementation of the computing device. The computing device can be a computer device, a terminal device, a cluster device, etc.

[0133] like Figure 4 As shown, the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0134] The processor 402, the communication interface 404, and the memory 406 communicate with each other via a communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute a program 410, which can specifically execute the relevant steps in the above-mentioned data segmentation method or big data task processing method embodiment.

[0135] Specifically, the program 410 may include program code including computer executable instructions.

[0136] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0137] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0138] The program 410 may be specifically invoked by the processor 402 to cause the computing device to perform the following operations:

[0139] Get the current segment length of the data to be processed;

[0140] Determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length;

[0141] Based on the current segment length and the current overlapping data length, segment data corresponding to the data to be processed is generated.

[0142] In an optional manner, the method further includes:

[0143] Calculate based on the segmented data corresponding to the data to be processed, obtain the segmented result data corresponding to the segmented data corresponding to the data to be processed; determine the overlapping data in the segmented data corresponding to the data to be processed; the length of the overlapping data is the current overlapping data length; determine the overlapping data result corresponding to the overlapping data in the segmented result data; remove the overlapping data result to obtain result data;

[0144] Verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length, and re-execute the step of determining the current overlapping data length based on the adjusted current segment length to generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, and obtain the target segment length and target overlapping data length of the data to be processed.

[0145] In an optional manner, obtaining the current segment length of the data to be processed includes: determining the current segment length according to disk performance parameters, memory information and algorithm complexity.

[0146] In an optional manner, the generating segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length includes:

[0147] Segmenting the data to be processed based on the current segment length to obtain a plurality of segmented data;

[0148] Each segmented data after segmentation is concatenated with the overlapping data of the current overlapping data length to obtain the segmented data corresponding to the data to be processed; the overlapping data is the data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

[0149] In an optional manner, before obtaining the current segment length of the data to be processed, the method further includes:

[0150] Determine a time cost function according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed;

[0151] Determine a space cost function according to the current segment length and the current overlapping data length; the space cost function represents the space cost of storing the data to be processed;

[0152] Determining a time-space efficiency function according to the time cost function and the space cost function;

[0153] Finding the optimal solution of the time-space efficiency function, thereby determining the relationship between the current segment length and the current overlapping data length and the length of the data to be processed;

[0154] The preset relationship formula is obtained according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed.

[0155] In an optional manner, the preset relationship formula is:

[0156]

[0157] Among them, P represents the current segment length, K represents the current overlapping data length; and m represents the length of the data to be processed.

[0158] In an optional manner, verifying the accuracy of the result data includes:

[0159] Sampling a plurality of single-point data from the data to be processed;

[0160] Performing parallel calculations on the multiple single-point data to obtain calculation results of the multiple single-point data;

[0161] The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data.

[0162] Program 410 may also be specifically called by processor 402 to enable the computing device to perform the following operations:

[0163] Determine the target segment length and target overlapping data length corresponding to the data to be processed by the above data segmentation method;

[0164] Dividing the data to be processed according to the target segment length and the target overlapping data length to generate a plurality of target segment data;

[0165] Calculate the multiple target segmented data to obtain a target result.

[0166] The embodiment of the present invention adds repeated data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; generates segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; performs calculation based on the segmented data corresponding to the data to be processed, removes the overlapping data result, and obtains the result data; verifies the accuracy of the result data, and when the accuracy is less than a preset threshold, adjusts the current segment length. Segment length, and according to the adjusted current segment length, re-execute the step of determining the current overlapping data length, generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data, when the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, obtain the target segment length and target overlapping data length of the data to be processed, thereby making the segmentation of the data to be processed more reasonable, avoiding the problem of decreased accuracy of data analysis boundaries, and improving the accuracy of the results.

[0167] An embodiment of the present invention provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on a computing device, the computing device executes the data segmentation method or big data task processing method in any of the above method embodiments.

[0168] The executable instructions may be used to cause the computing device to perform the following operations:

[0169] Get the current segment length of the data to be processed;

[0170] Determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length;

[0171] Based on the current segment length and the current overlapping data length, segment data corresponding to the data to be processed is generated.

[0172] In an optional manner, the method further includes:

[0173] Calculate based on the segmented data corresponding to the data to be processed, obtain the segmented result data corresponding to the segmented data corresponding to the data to be processed; determine the overlapping data in the segmented data corresponding to the data to be processed; the length of the overlapping data is the current overlapping data length; determine the overlapping data result corresponding to the overlapping data in the segmented result data; remove the overlapping data result to obtain result data;

[0174] Verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length, and re-execute the step of determining the current overlapping data length based on the adjusted current segment length to generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data. When the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, and obtain the target segment length and target overlapping data length of the data to be processed.

[0175] In an optional manner, obtaining the current segment length of the data to be processed includes: determining the current segment length according to disk performance parameters, memory information and algorithm complexity.

[0176] In an optional manner, the generating segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length includes:

[0177] Segmenting the data to be processed based on the current segment length to obtain a plurality of segmented data;

[0178] Each segmented data after segmentation is concatenated with the overlapping data of the current overlapping data length to obtain the current segmented data; the overlapping data is the data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

[0179] In an optional manner, before obtaining the current segment length of the data to be processed, the method further includes:

[0180] Determine a time cost function according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed;

[0181] Determine a space cost function according to the current segment length and the current overlapping data length; the space cost function represents the space cost of storing the data to be processed;

[0182] Determining a time-space efficiency function according to the time cost function and the space cost function;

[0183] Finding the optimal solution of the time-space efficiency function, thereby determining the relationship between the current segment length and the current overlapping data length and the length of the data to be processed;

[0184] The preset relationship formula is obtained according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed.

[0185] In an optional manner, the preset relationship formula is:

[0186]

[0187] Among them, P represents the current segment length, K represents the current overlapping data length; and m represents the length of the data to be processed.

[0188] In an optional manner, verifying the accuracy of the result data includes:

[0189] Sampling a plurality of single-point data from the data to be processed;

[0190] Performing parallel calculations on the multiple single-point data to obtain calculation results of the multiple single-point data;

[0191] The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data.

[0192] The executable instructions may also be used to cause the computing device to perform the following operations:

[0193] Determine the target segment length and target overlapping data length corresponding to the data to be processed by the above data segmentation method;

[0194] Dividing the data to be processed according to the target segment length and the target overlapping data length to generate a plurality of target segment data;

[0195] Calculate the multiple target segmented data to obtain a target result.

[0196] The embodiment of the present invention adds repeated data before and after the segmented data, obtains the current segment length of the data to be processed; determines the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; generates segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; performs calculation based on the segmented data corresponding to the data to be processed, removes the overlapping data result, and obtains the result data; verifies the accuracy of the result data, and when the accuracy is less than a preset threshold, adjusts the current segment length. Segment length, and according to the adjusted current segment length, re-execute the step of determining the current overlapping data length, generate segment data corresponding to the data to be processed, perform calculations based on the segment data corresponding to the data to be processed, remove overlapping data results, obtain result data and verify the accuracy of the result data, when the accuracy is less than a preset threshold, adjust the current segment length until the accuracy is greater than or equal to the preset threshold, obtain the target segment length and target overlapping data length of the data to be processed, thereby making the segmentation of the data to be processed more reasonable, avoiding the problem of decreased accuracy of data analysis boundaries, and improving the accuracy of the results.

[0197] An embodiment of the present invention provides a data segmentation device, which is used to execute the above data segmentation method.

[0198] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computing device to execute the data segmentation method in any of the above method embodiments.

[0199] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer executes the data segmentation method in any of the above method embodiments.

[0200] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0201] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0202] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0203] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0204] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim that lists a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data segmentation method, characterized in that: The method comprises: Get the current segment length of the data to be processed; Determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; Based on the current segment length and the current overlapping data length, generating segment data corresponding to the data to be processed; Input each segmented data corresponding to the data to be processed into a data mining algorithm for parallel calculation to obtain calculation results of each segmented data corresponding to the data to be processed; among the calculation results of each segmented data corresponding to the data to be processed, there are overlapping data results corresponding to the overlapping data; delete the overlapping data results in the calculation results of each segmented data corresponding to the data to be processed, and merge the calculation results after deleting the overlapping data results, so as to obtain result data of the data to be processed; the result data of the data to be processed includes the target segment length and the target overlapping data length of the data to be processed; Verify the accuracy of the result data, including: Sampling a plurality of single-point data from the data to be processed, wherein the single-point data includes a plurality of single-point data near a segmentation boundary point; Performing parallel calculations on the multiple single-point data to obtain calculation results of the multiple single-point data; The result data is verified according to the calculation result of the single point data, so as to determine the accuracy of the result data.

2. The method according to claim 1, characterized in that The method further comprises: When the accuracy is less than a preset threshold, the current segment length is adjusted, and according to the adjusted current segment length, the step of determining the current overlapping data length is re-executed to generate segment data corresponding to the data to be processed, and the segment data corresponding to the data to be processed is calculated to obtain a calculation result. If there is an overlapping data result corresponding to the overlapping data in the calculation result, the overlapping data result is deleted to obtain the result data, and the accuracy of the result data is verified. When the accuracy is less than a preset threshold, the step of adjusting the current segment length is performed until the accuracy is greater than or equal to the preset threshold, and the target segment length and target overlapping data length of the data to be processed are obtained.

3. The method according to claim 1, characterized in that: The obtaining of the current segment length of the data to be processed includes: The current segment length is determined according to disk performance parameters, memory information and algorithm complexity.

4. The method according to claim 1, characterized in that The step of generating segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length includes: Segmenting the data to be processed based on the current segment length to obtain a plurality of segmented data; Each segmented data after segmentation is concatenated with the overlapping data of the current overlapping data length to obtain segmented data corresponding to the data to be processed; the overlapping data is the data in the data to be processed that is located within the current overlapping data length before and after the segmented data.

5. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the current segment length of the data to be processed, the method further includes: Determine a time cost function according to the processing time of the original data to be processed, the processing time of the segmented data and the consumption of non-task processing time; the time cost function represents the time cost of processing the data to be processed; Determine a space cost function according to the current segment length and the current overlapping data length; the space cost function represents the space cost of storing the data to be processed; Determining a time-space efficiency function according to the time cost function and the space cost function; Finding the optimal solution of the time-space efficiency function, thereby determining the relationship between the current segment length and the current overlapping data length and the length of the data to be processed; The preset relationship formula is obtained according to the relationship between the current segment length, the current overlapping data length and the length of the data to be processed.

6. The method according to claim 5, characterized in that The preset relationship formula is: Among them, P represents the current segment length, K represents the current overlapping data length; and m represents the length of the data to be processed.

7. A method for processing big data tasks, characterized in that: The method comprises: Determine the target segment length and target overlapping data length corresponding to the data to be processed by the data segmentation method according to any one of claims 1 to 6; Dividing the data to be processed according to the target segment length and the target overlapping data length to generate a plurality of target segment data; Calculate the multiple target segmented data to obtain a target result.

8. A data segmentation device, characterized in that: The device comprises: An acquisition module is used to obtain the current segment length of the data to be processed; A determination module, used to determine the current overlapping data length according to the current segment length and a preset relationship formula; the preset relationship formula indicates that the segment length of the data to be processed is positively correlated with the total length of the data to be processed and the overlapping data length; A generating module, configured to generate segmented data corresponding to the data to be processed based on the current segment length and the current overlapping data length; A calculation module, used for inputting each segmented data corresponding to the data to be processed into a data mining algorithm for parallel calculation, and obtaining calculation results of each segmented data corresponding to the data to be processed; wherein there are overlapping data results corresponding to overlapping data in the calculation results of each segmented data corresponding to the data to be processed; wherein the overlapping data results in the calculation results of each segmented data corresponding to the data to be processed are deleted, and the calculation results after deleting the overlapping data results are merged, so as to obtain result data of the data to be processed; the result data of the data to be processed includes a target segment length and a target overlapping data length of the data to be processed; A verification and adjustment module is used to verify the accuracy of the result data, specifically for sampling a plurality of single-point data from the data to be processed; the single-point data includes a plurality of single-point data near a segmentation boundary point; the plurality of single-point data are calculated in parallel to obtain calculation results of the plurality of single-point data; the result data is verified according to the calculation results of the single-point data, so as to determine the accuracy of the result data.

9. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the data segmentation method described in any one of claims 1-6 or the big data task processing method described in claim 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computing device, the computing device executes the operation of the data segmentation method as described in any one of claims 1 to 6 or the operation of the big data task processing method as described in claim 7.

Images