Data processing optimization method, device, electronic device and storage medium

Abstract

The present invention relates to the field of data processing technology, and in particular to a data processing optimization method, device, electronic device, and storage medium. The method comprises: based on an original data processing information set and multiple data processing methods used by the original data processing information set, obtaining the original value of a data processing method identification value used by each original data processing information, adding the original value of the data processing method identification value used by each original data processing information to the original data processing information, and treating the original data processing information with the added original value of the data processing method identification value as information to be processed; constructing a multi-objective optimization model based on all the information to be processed; and optimizing the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain the optimal value of the number of file fragments, the total number of threads, and the data processing method identification value corresponding to each information to be processed. The present invention can optimize data processing time to the greatest extent.

Description

Technical Field

[0001] The present invention relates to the field of data desensitization technology, and in particular to a data processing optimization method, device, electronic device and storage medium. Background Art

[0002] In some application scenarios, it's necessary to optimize the processing time of data processing information, such as optimizing masking time, to ensure data processing can be achieved with the minimum possible resources. In some practical applications, this optimization requires optimizing the processing time of data processing information without revealing the data processing method used for each piece of data processing information. Therefore, optimizing data processing time in an environment with limited information and unclear characteristics of the data processing information is a topic worth exploring. Summary of the Invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to a first aspect of the present invention, a data processing optimization method is provided, the method comprising the following steps:

[0005] S100, based on the original data processing information set that needs to be processed and the K data processing methods used by the original data processing information set, cluster all the original data processing information into K clusters, wherein the original value of the data processing method identification value used by each original data processing information belonging to the rth cluster is r; wherein each original data processing information includes the file ID corresponding to the information and the original value of the data processing parameters, the data processing parameters including file size, total number of file lines, number of file shards, number of file shard lines, total number of threads and data processing time; the value of r ranges from 1 to K.

[0006] S200: adding the original value of the data processing method identification value used by each original data processing information to the original data processing information, and using the original data processing information with the original value of the data processing method identification value added as information to be processed.

[0007] S300, constructing a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, and the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first objective function and the third objective function are the number of file shards, the total number of threads, and the data processing method identification value, and the independent variables of the second objective function are the number of file shards and the total number of threads.

[0008] S400 , optimizing the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain optimal values of the number of file fragments, the total number of threads, and the data processing method identification value corresponding to each information to be processed.

[0009] According to a second aspect of the present invention, a data desensitization optimization device is provided, the device comprising the following steps:

[0010] A data processing module is used to cluster all raw data processing information into K clusters based on the raw data processing information sets to be processed and the K data processing methods used by the raw data processing information sets, wherein the original value of the data processing method identification value used by each raw data processing information belonging to the rth cluster is r; wherein each raw data processing information includes the file ID corresponding to the information and the original values of data processing parameters, wherein the data processing parameters include file size, total number of file lines, number of file fragments, number of file fragment lines, total number of threads and data processing time; and the value of r ranges from 1 to K.

[0011] A multi-objective optimization model construction module is used to construct a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, and the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first objective function and the third objective function are the number of file shards, the total number of threads, and the data processing method identification value, and the independent variables of the second objective function are the number of file shards and the total number of threads.

[0012] The optimization module is used to optimize the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain the optimal values of the number of file fragments, the total number of threads and the data processing method identification value corresponding to each information to be processed.

[0013] According to a third aspect of the present invention, an electronic device is provided, comprising a processor and a memory; the processor is configured to execute the steps of the method according to the first aspect of the present invention by calling a program or instruction stored in the memory.

[0014] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method according to the first aspect of the present invention.

[0015] The present invention has at least the following beneficial effects:

[0016] The data processing optimization method provided by the embodiment of the present invention can quickly and accurately obtain the optimal values of parameters that affect data processing time in a limited information environment where the data processing method is unclear, thereby effectively optimizing the data processing time.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a flowchart of a data processing optimization method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0023] The embodiment of the present invention provides a data processing optimization method, such as Figure 1 As shown, the method may include the following steps:

[0024] S100 , clustering all original data processing information into K clusters based on the original data processing information set and the K data processing methods used in the original data processing information set.

[0025] Among them, the original value of the data processing method identification value used by each original data processing information belonging to the rth cluster is r; wherein, each original data processing information includes the file ID corresponding to the information and the original value of the data processing parameters, and the data processing parameters include file size, total number of file lines, number of file fragments, number of file fragment lines, total number of threads and data processing time; the data processing method identification value is used to represent which data processing method belongs to among K data processing methods; the value of r is 1 to K.

[0026] In the embodiments of the present invention, a data processing method is a method for processing a file corresponding to data processing information. The data processing method identification value is used to indicate which data processing method among the K data processing methods. For example, a data processing method identification value of 1 indicates that the first data processing method is used.

[0027] In an embodiment of the present invention, both the original data processing information set and the K data processing methods used by the original data processing information set can be information provided to the user, wherein the data processing method used by each original data processing information is not specified, that is, the data processing method used by each original data processing information is unknown. It is only known that K data processing methods are used when data processing the file corresponding to the original data processing information set is performed, but the specific data processing method used by each original data processing information is unknown.

[0028] In an actual application scenario, the original data processing information set may be an information set obtained by desensitizing sensitive information of civil aviation passengers. In this scenario, the data processing method may be a desensitization method.

[0029] In the embodiments of the present invention, the file ID refers to the file name. The file ID, file size, and total number of lines in the file are inherent attributes of each file and do not change during subsequent processing. The total number of threads refers to the number of threads used to perform data processing on the file. In the original data processing information set, the total number of threads is the same for all original data processing information. The number of file shards refers to the number of sub-files obtained by splitting the file, and the number of file shard lines refers to the number of lines in each file shard. In theory, the number of file shards is equal to the total number of lines in the file divided by the number of file shard lines.

[0030] In the embodiment of the present invention, since it is impossible to know the data processing method used by each original data processing information, the data processing information can only be clustered using an unsupervised clustering method. Further, S100 may specifically include:

[0031] S110 , randomly selecting K pieces of original data processing information from the original data processing information set as K initial cluster centers.

[0032] S120, for any original data processing information i, obtain the similarity S between the original data processing information i and the jth current cluster center in the current cluster center set ij , get the similarity set S corresponding to the original data processing information i i , and add the original data processing information i to S i The data set corresponding to the current cluster center corresponding to the minimum similarity in ; the value of i ranges from 1 to n, n is the number of original data processing information, and the value of j ranges from 1 to K; the initial value of the current cluster center set is K initial cluster centers, and the initial value of the data set corresponding to the current cluster center is the original data processing information corresponding to the current cluster center.

[0033] In the embodiment of the present invention, the similarity between two pieces of original data processing information may be the Euclidean norm between the two pieces of original data processing information.

[0034] S130, obtain the intra-cluster discreteness corresponding to the data set corresponding to each current cluster center. If the intra-cluster discreteness corresponding to any current cluster center is less than the set discreteness, the data sets corresponding to the K current cluster centers are combined as the K clusters, and execute S140; otherwise, update the current cluster center based on the data set corresponding to each current cluster center, and execute S120.

[0035] Among them, the intra-cluster dispersion D corresponding to the jth current cluster center j The following conditions are met:

[0036] D j =(1 / 4)∑ 4 v=1 ((∑ z（j） u=1 (N ju v -N0 j v )) / z(j)), N ju v is the original value of the vth associated parameter in the uth data in the data set corresponding to the jth current cluster center, N0 j v is the original value of the vth associated parameter corresponding to the jth current cluster center, u ranges from 1 to z(j), z(j) is the number of data in the dataset corresponding to the jth current cluster center, v ranges from 1 to 4, and the associated parameters include file size, total number of file lines, number of file shards, and total number of threads.

[0037] In the embodiment of the present invention, the parameter value of the updated v-th associated parameter of each current cluster center corresponding to the current cluster center is the average of the original values of the v-th associated parameters of all the to-be-processed information in the data set corresponding to the current cluster center.

[0038] S140 , setting the data processing method identification value of the original data processing information belonging to the rth cluster to r, where r ranges from 1 to K.

[0039] S200: adding the original value of the data processing method identification value used by each original data processing information to the original data processing information, and using the original data processing information with the original value of the data processing method identification value added as information to be processed.

[0040] S300, constructing a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, and the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first objective function and the third objective function are the number of file shards, the total number of threads, and the data processing method identification value, and the independent variables of the second objective function are the number of file shards and the total number of threads.

[0041] In the embodiment of the present invention, by analyzing the relationship between the file size and data processing time, the total number of file lines and data processing time, and the number of file fragments and data processing time of the original data processing information set, the following conclusions are obtained:

[0042] The relationship between file size and data processing time includes the following situations:

[0043] (1) The file size is large and the content is sensitive, resulting in a long data processing time;

[0044] (2) The file size is small and the content is sensitive, resulting in longer data processing time;

[0045] (3) The file size is large and the content is not sensitive, resulting in a short data processing time;

[0046] (4) The file size is small and the content is not sensitive, resulting in shorter data processing time.

[0047] In an embodiment of the present invention, content sensitivity means that the data processing time of a file is greatly affected by the file size, the total number of lines in the file, the number of file fragments and the total number of threads, and internal insensitivity means that the data processing time of a file is less affected by the file size, the total number of lines in the file, the number of file fragments and the total number of threads.

[0048] The relationship between the number of file lines and data processing time includes the following situations:

[0049] (1) The file has many lines and the content is sensitive, which results in a long data processing time;

[0050] (2) The file has a small number of lines and the content is sensitive, resulting in a long data processing time;

[0051] (3) The file has many lines and the content is not sensitive, resulting in shorter data processing time;

[0052] (4) The file has fewer lines and the content is not sensitive, resulting in shorter data processing time.

[0053] The relationship between the number of file shards and data processing time includes: there are cases where the data processing time is still high even with a large number of shards. The possible reasons are large file size, sensitive content, inappropriate data processing methods, and a small number of allocated threads. There are also cases where the data processing time is low even with a small number of shards. The reasons are also related to factors such as file size and content.

[0054] The above data analysis shows that there is no obvious linear relationship in this problem. The various parameters influence each other, and various special circumstances need to be comprehensively considered. This paper optimizes data processing time by adjusting three decision variables: the number of file shards, the number of threads, and the data processing method. The allocation of shards and threads should be kept within a reasonable and acceptable range as much as possible, thus forming the above multi-objective optimization model. The first objective function aims to find the three decision variables that minimize data processing time while minimizing data consumption. The second objective function aims to find the most reasonable total number of threads. The third objective function aims to find the most suitable data processing method.

[0055] In the embodiment of the present invention, the data preprocessing time refers to the time consumed before data processing is performed on a file, including the time consumed for file segmentation and thread allocation.

[0056] Furthermore, the first objective function Y1 corresponding to the g-th information to be processed is g The following conditions are met:

[0057] Y1 g =min T, where T is the data processing time calculation function, T=[(1+ min (β×NF g +γ×NL g ))×α g ×(Size g / (NF g +NL g ))+min (β×NF g +γ×NL g )].

[0058] Among them, NF g is the number of file fragments of the g-th information to be processed during the optimization process, NL g is the total number of threads for the gth information to be processed during the optimization process, α g is the time-consuming impact coefficient of the g-th information to be processed, Size g is the original value of the file size of the gth information to be processed; the value of g ranges from 1 to n, and n is the number of information to be processed.

[0059] In an embodiment of the present invention, the time-consuming impact coefficient is used to characterize the degree to which the data processing time is affected by the file size, the total number of file lines, the number of file fragments, and the total number of threads. The larger the time-consuming impact coefficient, the greater the degree to which the data processing time is affected by the file size, the total number of file lines, the number of file fragments, and the total number of threads. Vice versa, that is, for files 1 and 2 with the same file size, the total number of file lines, the number of file fragments, and the total number of threads, if the data processing time of file 1 is greater than the data processing time of file 2, it means that the data processing time of file 1 is more susceptible to the influence of file size, the total number of file lines, the number of file fragments, and the total number of threads. For example, a file with a large file size, many lines, many fragments, many total threads, and a long processing time means that its content sensitivity will naturally be very high, and the time-consuming impact coefficient will be large; in contrast, a file with a large file size, many lines, many fragments, many total threads, but a short processing time means that its content sensitivity is poor, and the time-consuming impact coefficient is low.

[0060] In the embodiment of the present invention, the time-consuming impact coefficient of the information to be processed can be obtained based on the original values of the file size, the total number of lines in the file, the number of file fragments, and the total number of threads corresponding to the information. Specifically, α g The following conditions are met:

[0061] α g =T g ×(Size g -1 +N g -1 )×NL g org ×NF g org -λ.

[0062] Among them, T g is the original value of the data processing time corresponding to the g-th information to be processed, N g is the total number of lines in the file corresponding to the g-th information to be processed, NL g org is the original value of the total number of threads corresponding to the g-th information to be processed, NF g org is the original value of the number of file fragments corresponding to the g-th information to be processed, and λ is the adjustment factor.

[0063] The second objective function Y2 corresponding to the g-th information to be processed g The following conditions are met:

[0064] Y2 g =min (β×NF g +γ×NL g ); β represents the file shard weight, and γ represents the thread weight.

[0065] The third objective function Y3 corresponding to the g-th information to be processed g The following conditions are met:

[0066] Y3 g =min(∑ 4 v=1 (N gv -N0 gv ) 2 / (N0 gv ) 2 ), N gv is the parameter value of the vth associated parameter of the gth information to be processed in the optimization process, and the value of v ranges from 1 to 4. N0 gv is the parameter value of the vth data processing parameter at the center of the cluster to which the data processing method identification value corresponding to the gth information to be processed belongs during the optimization process.

[0067] The constraints corresponding to the g-th information to be processed meet the following conditions:

[0068] (1) 1≤C g ≤K; C g The data processing method identification value of the g-th information to be processed.

[0069] (2) 1<NL g <NL0; NL0 is the threshold of the total number of threads, which can be an empirical value, for example, 448;

[0070] (3) n1≤NR g / NF g ≤n2,NR g is the total number of lines in the file of the gth information to be processed, n1 is the minimum file fragment line number threshold, and n2 is the maximum file fragment line number threshold. Both n1 and n2 can be empirical values.

[0071] Furthermore, in the embodiment of the present invention, β, γ, and λ are parameters that need to be optimized to make the constructed data processing time calculation function more accurate. In an exemplary embodiment, β, γ, and λ can be obtained by the following steps:

[0072] S10, substitute the original values of the file size, total number of file lines, number of file fragments, total number of threads and data processing method identification value of each information to be processed into the current data processing time calculation function to obtain the current predicted data processing time corresponding to each information to be processed, and obtain n current predicted data processing times.

[0073] In the embodiment of the present invention, the initial value of the current data processing time calculation function is a function for initializing β, γ, and λ.

[0074] The data processing time calculation function satisfies the following conditions: T = [(1 + (β × NF + γ × NL)) × α × (Size / (NF + NL)) + (β × NF + γ × NL)], where T is the data processing time calculation function, α is the time consumption impact coefficient, Size is the file size, NF is the number of file fragments, and NL is the total number of threads.

[0075] S20, obtain the deviation between n currently predicted data processing times and the corresponding n real data processing times. If the deviation is less than or equal to the set deviation, use the current β, γ and λ as the final β, γ and λ. Otherwise, adjust β, γ and λ in the current data processing time calculation function based on the deviation and execute S10.

[0076] The actual data processing time of the information to be processed is the original value of the data processing time of the information to be processed. The deviation between the n currently predicted data processing times and the corresponding n actual data processing times can be calculated using the t-statistic, p-value, and R-squared value. The t-statistic is used to measure the degree of variation between the difference between two sample means relative to the sample data. A larger absolute value indicates a more significant difference between the two sample means. The p-value indicates the probability of observing a statistic of 0 or higher if the null hypothesis is true. A smaller value indicates stronger evidence for rejecting the null hypothesis. The R-squared value is an indicator of how well the model fits the data. It ranges from 0 to 1. The closer the value is to 1, the stronger the model's ability to explain the data.

[0077] In the embodiment of the present invention, the deviation results of the data processing time calculation function are finally obtained as shown in Table 1 below:

[0078] Table 1

[0079]

[0080] As shown in Table 1, the t-statistic is close to 0, indicating that the difference between the two sample means is very small. The p-value is much greater than 0.05, indicating that there is insufficient evidence to reject the null hypothesis that the difference between the two sample means is not significant. The R-squared value is very close to 1, indicating that the model well explains the variance of the data. The test results show that the data processing time calculation function established in this embodiment of the present invention can effectively predict the data processing time of files under different policy combinations.

[0081] S400 , optimizing the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain optimal values of the number of file fragments, the total number of threads, and the data processing method identification value corresponding to each information to be processed.

[0082] The present invention uses a non-dominated sorting genetic algorithm to optimize a multi-objective optimization model to solve the limited information problem of unclear decision variables in real business scenarios.

[0083] Furthermore, S300 may specifically include:

[0084] S301 , randomly selecting P1 pieces of information to be processed from n pieces of information to be processed as an initial population.

[0085] P1 can be set based on actual needs. In an exemplary embodiment, P1=100.

[0086] S302, perform a non-dominated sorting operation on the current population to be processed to obtain a non-dominated solution in the current population. If a non-dominated solution is obtained, use the obtained non-dominated solution as the current parent and execute S303; if no non-dominated solution is obtained, execute S301; the initial value of the current population to be processed is the initial population.

[0087] In the embodiment of the present invention, a non-dominated solution refers to information to be processed that has at least one objective function value that is not the minimum in the current population to be processed. It is known to those skilled in the art that a non-dominated solution can be obtained by using existing technologies.

[0088] S303, performing a crossover operation on the current parent generation, so that any two individuals in the current parent generation randomly exchange the number of file shards, the total number of threads, and the data processing method identification value, and obtain a corresponding crossover operation result as the current crossover operation result; wherein, one individual represents one piece of information to be processed.

[0089] In the embodiment of the present invention, a crossover operation may be performed on the parent generation with a probability of 70%.

[0090] S304, performing a mutation operation on the current crossover operation result, so that the number of file fragments, the total number of threads and the data processing method identification value of the individuals in the current crossover operation result are randomly mutated to obtain a corresponding mutation result as the current offspring.

[0091] In the embodiment of the present invention, individuals may be randomly mutated with a probability of 20%.

[0092] S305, merge the current parent generation and the current child generation to obtain the current merged population, and obtain the data processing time and data processing time optimization value of each individual in the current merged population, which are respectively used as the current data processing time and current data processing time optimization value of the individual; the current data processing time optimization value of each individual is equal to the difference between the current data processing time of the individual and the data processing time of the previous time.

[0093] In the embodiment of the present invention, the data processing time of each individual can be calculated by substituting the parameter value of the current data processing time influencing parameter of the individual into the data processing time calculation function.

[0094] S306, sort all the current data processing time optimization values corresponding to the current merged population in descending order, and use the individuals corresponding to the first P1 optimization values in the sorted data processing time optimization values as the current population to be processed, and add the current data processing time of each individual in the current merged population to the corresponding current data processing time record set; set the iteration counter C=C+1, the initial value of C is 0; the initial value of the current data processing time record set is empty.

[0095] S307, if C≤C0, execute S302, if C>C0, use the number of file fragments, total number of threads and data processing method identification value corresponding to the maximum data processing time in the current data processing time record set corresponding to each individual as the optimal value of the number of file fragments, total number of threads and data processing method identification value corresponding to the information to be processed; C0 is the preset iteration number threshold, which can be an empirical value, for example, C0=100.

[0096] In summary, the data desensitization optimization algorithm provided by the embodiment of the present invention is aimed at a limited information problem environment in which the number of threads in the original data set has a single feature and the data processing method features are unclear. A mathematical model of file data processing time based on personalized time-consuming influence coefficients and clustering is established, and the effectiveness of the mathematical model is verified through statistical tests. According to the constructed mathematical model, the research objective is established as a multi-objective optimization problem, an evaluation function is designed according to the optimization objective, and the objective function is solved using a non-dominated sorting genetic algorithm, so that the data processing time of the information to be processed can be effectively optimized while consuming minimal resources.

[0097] Another embodiment of the present invention provides a data desensitization optimization device, which includes the following steps:

[0098] A data processing module is used to cluster all raw data processing information into K clusters based on the raw data processing information sets to be processed and the K data processing methods used by the raw data processing information sets, wherein the original value of the data processing method identification value used by each raw data processing information belonging to the rth cluster is r; wherein each raw data processing information includes the file ID corresponding to the information and the original values of data processing parameters, wherein the data processing parameters include file size, total number of file lines, number of file fragments, number of file fragment lines, total number of threads and data processing time; and the value of r ranges from 1 to K.

[0099] A multi-objective optimization model construction module is used to construct a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, and the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first objective function and the third objective function are the number of file shards, the total number of threads, and the data processing method identification value, and the independent variables of the second objective function are the number of file shards and the total number of threads.

[0100] The optimization module is used to optimize the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain the optimal values of the number of file fragments, the total number of threads and the data processing method identification value corresponding to each information to be processed.

[0101] The device can be used to perform Figure 1 The method shown in the embodiment shown, therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is omitted for brevity.

[0102] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0103] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0104] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0105] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data processing optimization method, characterized in that: The method comprises the following steps: S100, based on a set of raw data processing information to be processed and K data processing methods used by the set of raw data processing information, clustering all raw data processing information into K clusters, wherein the original value of the data processing method identification value used by each piece of raw data processing information belonging to the rth cluster is r; wherein each piece of raw data processing information includes a file ID corresponding to the information and original values of data processing parameters, wherein the data processing parameters include file size, total number of file lines, number of file shards, number of file shard lines, total number of threads, and data processing time; and r ranges from 1 to K; S200, adding the original value of the data processing method identification value used by each original data processing information to the original data processing information, and using the original data processing information with the original value of the data processing method identification value added as information to be processed; S300: Construct a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, wherein the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first and third objective functions are the number of file shards, the total number of threads, and the data processing method identifier, and the independent variables of the second objective function are the number of file shards and the total number of threads; S400 , optimizing the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain optimal values of the number of file fragments, the total number of threads, and the data processing method identification value corresponding to each information to be processed.

2. The method according to claim 1, characterized in that S100 specifically includes: S110, randomly selecting K pieces of original data processing information from the original data processing information set as K initial cluster centers; S120, for any original data processing information i, obtain the similarity S between the original data processing information i and the jth current cluster center in the current cluster center set ij , get the similarity set S corresponding to the original data processing information i i , and add the original data processing information i to S i The data set corresponding to the current cluster center corresponding to the minimum similarity in ; the value of i ranges from 1 to n, n is the number of original data processing information, and the value of j ranges from 1 to K; the initial value of the current cluster center set is K initial cluster centers, and the initial value of the data set corresponding to the current cluster center is the original data processing information corresponding to the current cluster center; S130, obtaining the intra-cluster dispersion corresponding to the data set corresponding to each current cluster center. If the intra-cluster dispersion corresponding to any current cluster center is less than the set dispersion, the data sets corresponding to the K current cluster centers are combined as the K clusters, and S140 is executed; otherwise, the current cluster center is updated based on the data set corresponding to each current cluster center, and S120 is executed. S140 , setting the data processing method identification value of the original data processing information belonging to the r th cluster to r.

3. The method according to claim 2, characterized in that The intra-cluster dispersion D corresponding to the jth current cluster center j The following conditions are met: D j =(1 / 4)∑ 4 v=1 ((∑ z（j） u=1 (N ju v -N0 j v )) / z(j)), N ju v is the original value of the vth associated parameter in the uth data in the data set corresponding to the jth current cluster center, N0 j v is the original value of the vth associated parameter corresponding to the jth current cluster center, u ranges from 1 to z(j), z(j) is the number of data in the dataset corresponding to the jth current cluster center, v ranges from 1 to 4, and the associated parameters include file size, total number of file lines, number of file shards, and total number of threads.

4. The method according to claim 3, characterized in that The first objective function Y1 corresponding to the g-th information to be processed g The following conditions are met: Y1 g =min [(1+ min (β×NF g +γ×NL g ))×α g ×(Size g / (NF g +NL g ))+min(β×NF g +γ×NL g )]; NF g is the number of file fragments of the g-th information to be processed during the optimization process, NL g is the total number of threads for the gth information to be processed during the optimization process, α g is the time-consuming impact coefficient of the g-th information to be processed, Size g is the original value of the file size of the gth information to be processed; the value of g ranges from 1 to n, where n is the number of information to be processed; β represents the file shard weight, and γ represents the thread weight; The second objective function Y2 corresponding to the g-th information to be processed g The following conditions are met: Y2 g =min (β×NF g +γ×NL g ); β represents the file shard weight, γ represents the thread weight; The third objective function Y3 corresponding to the g-th information to be processed g The following conditions are met: Y3 g =min(∑ 4 v=1 (N gv -N0 gv ) 2 / (N0 gv ) 2 ), N gv is the parameter value of the vth associated parameter of the gth information to be processed in the optimization process, N0 gv is the parameter value of the vth data processing parameter at the cluster center of the data processing method identification value corresponding to the gth information to be processed during the optimization process; The constraints corresponding to the g-th information to be processed meet the following conditions: (1) 1≤C g ≤K; C g The data processing method identification value of the g-th information to be processed; (2) 1<NL g <NL0; NL0 is the total thread number threshold; (3) n1≤NR g / NF g ≤n2,NR g is the total number of lines in the file of the gth information to be processed, n1 is the minimum file fragment line number threshold, and n2 is the maximum file fragment line number threshold.

5. The method according to claim 4, characterized in that α g The following conditions are met: α g =T g ×(Size g -1 +NR g -1 )×NL g org ×NF g org -λ; Among them, T g is the original value of the data processing time corresponding to the g-th information to be processed, NL g org is the original value of the total number of threads corresponding to the g-th information to be processed, NF g org is the original value of the number of file fragments corresponding to the g-th information to be processed, and λ is the adjustment factor.

6. The method according to claim 5, characterized in that β, γ, and λ are obtained by the following steps: S10, substituting the original values of the file size, total number of file lines, number of file fragments, total number of threads, and data processing method identification value of each information to be processed into the current data processing time calculation function to obtain the current predicted data processing time corresponding to each information to be processed, and obtaining n current predicted data processing times; the initial value of the current data processing time calculation function is a function for initializing β, γ, and λ; the data processing time calculation function satisfies the following condition: T=[(1+(β×NF+γ×NL))×α×(Size / (NF+NL))+(β×NF+γ×NL)], where T is the data processing time calculation function, α is the time consumption influence coefficient, Size is the file size, NF is the number of file fragments, and NL is the total number of threads; S20, obtain the deviation between n currently predicted data processing times and the corresponding n real data processing times. If the deviation is less than or equal to the set deviation, use the current β, γ and λ as the final β, γ and λ. Otherwise, adjust β, γ and λ in the current data processing time calculation function based on the deviation and execute S10.

7. The method according to claim 1, characterized in that S300 specifically includes: S301, randomly selecting P1 pieces of information to be processed from n pieces of information to be processed as the initial population; S302: Perform a non-dominated sorting operation on the current population to be processed to obtain non-dominated solutions in the current population. If a non-dominated solution is obtained, use the obtained non-dominated solution as the current parent and execute S303; if no non-dominated solution is obtained, execute S301; the initial value of the current population to be processed is the initial population; S303: Perform a crossover operation on the current parent generation, so that any two individuals in the current parent generation randomly exchange the number of file shards, the total number of threads, and the data processing method identification value, and obtain a corresponding crossover operation result as the current crossover operation result; wherein one individual represents one piece of information to be processed; S304: performing a mutation operation on the current crossover operation result, so that the number of file shards, the total number of threads, and the data processing method identification value of the individual in the current crossover operation result are randomly mutated, and a corresponding mutation result is obtained as the current offspring; S305: Merge the current parent generation and the current child generation to obtain a current merged population, and obtain the data processing time and optimized data processing time value of each individual in the current merged population, which are used as the current data processing time and the optimized data processing time value of the individual, respectively. The optimized data processing time value of each individual is equal to the difference between the current data processing time of the individual and the previous data processing time. S306: sort all current data processing time optimization values corresponding to the current merged population in descending order, and use the individuals corresponding to the first P1 optimization values among the sorted data processing time optimization values as the current population to be processed, and add the current data processing time of each individual in the current merged population to the corresponding current data processing time record set; set the iteration count counter C=C+1, and the initial value of C is 0; the initial value of the current data processing time record set is empty; S307, if C≤C0, execute S302, if C>C0, use the number of file fragments, total number of threads and data processing method identification value corresponding to the maximum data processing time in the current data processing time record set corresponding to each individual as the optimal value of the number of file fragments, total number of threads and data processing method identification value corresponding to the information to be processed; C0 is the preset iteration number threshold.

8. A data desensitization optimization device, characterized in that: The device comprises the following steps: a data processing module, configured to cluster all raw data processing information into K clusters based on a set of raw data processing information to be processed and K data processing methods used by the set of raw data processing information, wherein the original value of the data processing method identification value used by each raw data processing information belonging to the rth cluster is r; wherein each raw data processing information includes a file ID corresponding to the information and original values of data processing parameters, wherein the data processing parameters include file size, total number of file lines, number of file fragments, number of file fragment lines, total number of threads, and data processing time; and r ranges from 1 to K; A multi-objective optimization model construction module is used to construct a multi-objective optimization model based on all the information to be processed; wherein the multi-objective optimization model includes constraints and objective functions, wherein the objective functions include a first objective function corresponding to minimizing data processing time, a second objective function corresponding to minimizing data preprocessing time, and a third objective function minimizing cluster center distance, wherein the independent variables of the first objective function and the third objective function are the number of file shards, the total number of threads, and the data processing method identification value, and the independent variables of the second objective function are the number of file shards and the total number of threads; The optimization module is used to optimize the multi-objective optimization model based on a non-dominated sorting genetic algorithm to obtain the optimal values of the number of file fragments, the total number of threads and the data processing method identification value corresponding to each information to be processed.

9. An electronic device, characterized in that: including processor and memory; The processor is configured to execute the steps of the method according to any one of claims 1 to 7 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 7.

Images