Memory data optimization method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of clustering algorithms, and discloses a memory data optimization method, device, computer equipment and storage medium, wherein the method comprises: obtaining a data set to be processed; determining whether the data processing amount of the data set is greater than a preset threshold; if the data processing amount of the data set is greater than the preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism; processing the data set according to the first preset memory optimization mechanism and / or the second preset memory optimization mechanism to obtain a calculation result. The present invention can reduce errors in large-scale data calculations, ensure the required calculation results, and reduce the calculation time.

Description

Technical Field

[0001] The present invention relates to the field of clustering algorithms, and in particular to a memory data optimization method, device, computer equipment and storage medium. Background Art

[0002] Silhouette Coefficient is a clustering result evaluation method used to evaluate the effect of unsupervised clustering algorithms and determine the number of clusters (i.e. groups) in the clustering process. Silhouette Coefficient combines the cohesion and separation of clusters to evaluate the clustering effect.

[0003] However, unknown errors often occur when calculating the silhouette coefficient on large-scale data, resulting in the inability to obtain calculation results. The existing solution is generally to reduce the number of samples in the data set and calculate the silhouette coefficient of a subset of the original data set. This calculation method cannot evaluate the overall effect of the clustering results, and the calculation results are prone to large deviations. In addition, when calculating the silhouette coefficient of a large-scale high-dimensional data set, even if the calculation results can be obtained, the calculation takes too long, which affects the application of the silhouette coefficient. Summary of the invention

[0004] Based on this, it is necessary to provide a memory data optimization method, device, computer equipment and storage medium to address the above technical problems, so as to reduce errors in large-scale data calculations, ensure that the required calculation results are obtained, and reduce the calculation time.

[0005] After many experiments, the inventor found that when processing large-scale data, by changing the memory usage of the data set, calculation errors can be effectively reduced and effective data processing results can be obtained more quickly.

[0006] A memory data optimization method, comprising:

[0007] Acquire a data set to be processed, where the data set is used for cluster analysis;

[0008] Determining whether the data processing amount of the data set is greater than a preset threshold;

[0009] If the data processing amount of the data set is greater than a preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism;

[0010] Modifying the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modifying the memory recycling mechanism and / or preset parameters of a preset data calculation method according to the second preset memory optimization mechanism, wherein the preset data calculation method includes a plurality of cyclic calculation steps;

[0011] The data set of the modified data type is processed according to the preset data calculation method to obtain a calculation result; or, the data set is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, the data set of the modified data type is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result.

[0012] A memory data optimization device, comprising:

[0013] An acquisition module, used for acquiring a data set to be processed, wherein the data set is used for cluster analysis;

[0014] A judgment module, used to judge whether the data processing amount of the data set is greater than a preset threshold;

[0015] An acquisition mechanism module, configured to acquire a first preset memory optimization mechanism and / or a second preset memory optimization mechanism if the data processing amount of the data set is greater than a preset threshold;

[0016] A computing optimization module, configured to modify the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modify the memory recycling mechanism and / or preset parameters of a preset data computing method according to the second preset memory optimization mechanism, wherein the preset data computing method includes a plurality of cyclic computing steps;

[0017] A result acquisition module is used to process the data set of the modified data type according to the preset data calculation method to obtain the calculation result; or, to process the data set according to the preset data calculation method with the modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, to process the data set of the modified data type according to the preset data calculation method with the modified memory recovery mechanism and\or preset parameters to obtain the calculation result.

[0018] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned memory data optimization method when executing the computer program.

[0019] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the memory data optimization method is implemented.

[0020] The above-mentioned memory data optimization method, device, computer equipment and storage medium obtain a data set to be processed, and the data set is used for cluster analysis to obtain data samples. Determine whether the data processing amount of the data set is greater than a preset threshold to preliminarily evaluate the calculation amount of the data set. If the data processing amount of the data set is greater than the preset threshold, obtain the first preset memory optimization mechanism and / or the second preset memory optimization mechanism to adopt an appropriate optimization mechanism to improve the processing capability of the data set. Modify the data type of all samples in the data set according to the first preset memory optimization mechanism, and / or modify the memory recovery mechanism and / or preset parameters of the preset data calculation method according to the second preset memory optimization mechanism. The preset data calculation method includes several cyclic calculation steps to execute the corresponding memory optimization mechanism. The data set of the modified data type is processed according to the preset data calculation method to obtain a calculation result; or, the data set is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, the data set of the modified data type is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result, so as to complete the processing of the data set, obtain the required calculation result, reduce errors in large-scale data calculations, ensure that the required calculation results are obtained, and reduce the calculation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a schematic diagram of an application environment of a memory data optimization method in one embodiment of the present invention;

[0023] Figure 2 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0024] Figure 3 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0025] Figure 4 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0026] Figure 5 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0027] Figure 6 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0028] Figure 7 It is a flow chart of a method for optimizing memory data in one embodiment of the present invention;

[0029] Figure 8 is a structural schematic diagram of a memory data optimization device in one embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The memory data optimization method provided in this embodiment can be applied to Figure 1 In an application environment, a client communicates with a server through a network. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0033] In one embodiment, if Figure 2 As shown, a memory data optimization method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the following steps:

[0034] S10, obtaining a data set to be processed, where the data set is used for cluster analysis;

[0035] S20, determining whether the data processing amount of the data set is greater than a preset threshold;

[0036] S30: if the data processing amount of the data set is greater than a preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism;

[0037] S40, modifying the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or, modifying the memory recycling mechanism and / or preset parameters of a preset data calculation method according to the second preset memory optimization mechanism, wherein the preset data calculation method includes a plurality of cyclic calculation steps;

[0038] S50. Process the data set of the modified data type according to the preset data calculation method to obtain a calculation result; or, process the data set according to the preset data calculation method with the modified memory recovery mechanism and / or preset parameters to obtain the calculation result; or, process the data set of the modified data type according to the preset data calculation method with the modified memory recovery mechanism and / or preset parameters to obtain the calculation result.

[0039] In this embodiment, the data set may be a collection containing a large number of samples. In particular, the data set may be a collection for cluster analysis. Cluster analysis refers to the analysis process of grouping a collection of physical or abstract objects into multiple classes consisting of similar objects. Cluster analysis is a set of statistical analysis techniques that divides research objects into relatively homogeneous groups. Cluster analysis can be used in scenarios such as dividing customer groups, biological classification, file repair, and data mining.

[0040] The preset threshold can be set according to actual needs. For example, the preset threshold can be set to 100,000. When the data processing volume of the data set is greater than the preset threshold, it means that for the current server, the sample data of the data set has exceeded the highest level that the server can achieve. At this time, if you continue to obtain the calculation results according to the original calculation method, unknown errors will often occur, resulting in the inability to obtain the calculation results.

[0041] In one example, when calculating the silhouette coefficient of a data set with 1 million samples on a server with 20 CPU cores, CPU main frequency 2.39GHz, 256G memory, and memory main frequency 2400MHz, the dimension of each sample in the data set is 10. In one test, the calculation of the silhouette coefficient took as long as 21503.8 seconds. In fact, if the server is replaced with an ordinary home computer, such as a computer with 8G memory, the running time will be greatly extended. You may wait for many days and still not get the result.

[0042] It is necessary to improve the memory usage during data calculation through a memory optimization mechanism to ensure that effective calculation results can be obtained. Here, the first preset memory optimization mechanism can be used to reduce the memory usage of sample data and save the space occupied by each sample in memory to prevent memory errors. The second preset memory optimization mechanism is used to improve the preset data calculation method, which is specifically reflected in two aspects. On the one hand, it is to flexibly set the memory occupied by the distance matrix of the data set so that when calculating the distance matrix, the memory is used more reasonably and the number of operations is reduced. On the other hand, it is reflected in setting a memory recovery mechanism to overcome memory leaks. These two memory optimization mechanisms can be selected one by one or used at the same time.

[0043] The preset data calculation method is a calculation method used to process the data set data, such as a calculation method of the silhouette coefficient. In some cases, it can also be other complex algorithms.

[0044] According to calculations, whether the first preset memory optimization mechanism is used alone or the second preset memory optimization mechanism is used alone, when processing data sets, the memory usage rate is greatly reduced and the frequency of memory errors is greatly reduced. When both memory optimization mechanisms are used at the same time, the memory optimization effect is more obvious.

[0045] In steps S10-S50, a data set to be processed is obtained, and the data set is used for cluster analysis to obtain data samples. It is determined whether the data processing amount of the data set is greater than a preset threshold value to preliminarily evaluate the calculation amount of the data set. If the data processing amount of the data set is greater than the preset threshold value, a first preset memory optimization mechanism and / or a second preset memory optimization mechanism are obtained to adopt appropriate optimization mechanisms to improve the processing capability of the data set. The data types of all samples in the data set are modified according to the first preset memory optimization mechanism, and / or the memory recovery mechanism and / or preset parameters of the preset data calculation method are modified according to the second preset memory optimization mechanism, and the preset data calculation method includes several cyclic calculation steps to execute the corresponding memory optimization mechanism. The data set of the modified data type is processed according to the preset data calculation method to obtain a calculation result; or, the data set is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, the data set of the modified data type is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result, so as to complete the processing of the data set, obtain the required calculation result, reduce errors in large-scale data calculations, ensure that the required calculation results are obtained, and reduce the calculation time.

[0046] Optional, such as Figure 3 As shown, step S40 includes:

[0047] S401, determining a number of data types to be selected according to the data processing amount;

[0048] S402, receiving a data type modification instruction to select a specified data type from the plurality of data types to be selected;

[0049] S403: Modify the preset data type of the samples in the data set to the specified data type according to the data type modification instruction, wherein the data length of the specified data type is shorter than the preset data type.

[0050] In this embodiment, the preset data type of the sample of the data set can be modified to reduce the memory overhead. First, a plurality of different candidate data types can be matched according to the data processing volume of the data set. For example, if the data processing volume of a data set is large, the precision of the matched candidate data type is small, and such candidate data types may be float16 or float8; conversely, if the data processing volume of a data set is small, the precision of the matched candidate data type is large, and such candidate data types may be float64 or float32. The user can select a specified data type from the candidate data types according to actual needs. Then, when the data set is processed, all samples in the data set are stored in the memory as the specified data type.

[0051] In one example, the samples in the dataset are stored in np.array. In Python 3, np.array stores floating-point data types as float64 by default, and each floating-point element occupies 8 bytes. Therefore, the floating-point data type stored in np.array can be changed to float16 through the data type modification instruction, and each floating-point element occupies 2 bytes. In this way, the memory occupied by the dataset samples is only one-fourth of the original memory, or even lower.

[0052] In the previous embodiment, the profile package in the memory_profiler module is used to record the memory usage of the same data set before and after the improvement, as shown in Table 1.

[0053] Table 1 Memory usage of the same data set before and after improvement

[0054]

[0055] It can be seen from Table 1 that by executing the first preset memory optimization mechanism, the memory usage of the data set samples can be reduced and precious memory resources can be saved.

[0056] In steps S401-S403, a plurality of candidate data types are determined according to the data processing amount to match the plurality of candidate data types, so that the user can select the final data type (i.e., the designated data type) therefrom. A data type modification instruction is received to select a designated data type from the plurality of candidate data types to obtain user selection information. According to the data type modification instruction, the preset data type of the sample in the data set is modified to the designated data type, and the data length of the designated data type is less than the preset data type, so as to complete the modification of the data type.

[0057] Optional, such as Figure 4 As shown, step S40 also includes:

[0058] S404, loading a memory detection process adapted to the preset data calculation method;

[0059] S405, when executing the loop calculation step in the preset data calculation method, detecting a memory leak generated by the loop calculation step through the memory detection process;

[0060] S406: Reclaim the memory space occupied by the memory leak through a memory recovery process.

[0061] In this embodiment, the preset data calculation method can be implemented based on the Python algorithm. The Python garbage collection mechanism is lazy, especially the variables in the for loop will not reclaim space immediately after the end, if a new variable appears, a new space will be opened up. Therefore, a memory detection process can be loaded, such as gc.set_debug(gc.DEBUG_LEAK), to check the memory leaks caused by the loop calculation steps, and then use the memory recovery process, such as gc.collect(), to recycle memory and optimize memory space. Since the preset data calculation method contains a large number of loop calculations, the memory space saved by the memory recovery mechanism is very considerable.

[0062] In one embodiment, the peak memory occupancy when the temporary memory is recovered without using the gc algorithm package (including the above-mentioned memory detection process and memory recovery process) and the peak memory occupancy when the temporary memory is recovered using the gc algorithm package are measured, as shown in Table 2.

[0063] Table 2 Memory usage before and after executing the memory recycling mechanism

[0064]

[0065] As can be seen from Table 2, by executing the memory recycling mechanism, the memory usage caused by memory leaks can be reduced and precious memory resources can be saved.

[0066] In steps S404-S406, a memory detection process adapted to the preset data calculation method is loaded to detect the memory usage of the preset data calculation method in real time. When executing the loop calculation step in the preset data calculation method, the memory leak generated by the loop calculation step is detected by the memory detection process to identify redundant objects that have no effect on the processing result. The memory space occupied by the memory leak is reclaimed by the memory recovery process to reclaim valuable memory resources.

[0067] Optional, such as Figure 5 As shown, before step S20, it also includes:

[0068] S21, determining an estimated occupied memory resource value according to the data processing amount of the data set and the preset data calculation method;

[0069] S22. Obtain a current available memory resource value, and determine the preset threshold value according to the estimated occupied memory resource value and the available memory resource value.

[0070] In this embodiment, the preset threshold value can be changed according to different devices. For example, the preset threshold value corresponding to the server configuration of 256G memory is different from the preset threshold value corresponding to the server configuration of 16G memory. The estimated value of occupied memory resources is positively correlated with the data processing volume of the data set. The more samples there are, the higher the estimated value of occupied memory resources. The estimated value of occupied memory resources is positively correlated with the complexity of the preset data calculation method. The higher the complexity, the higher the estimated value of occupied memory resources. The current available memory resource value refers to the computing resources that can be provided by the current hardware resources. In some cases, the preset threshold value can be calculated by a preset formula. For example,

[0071]

[0072] Among them, T is the preset threshold, x is the data processing amount of the data set, s0 is the available memory resource value, s1 is the estimated occupied memory resource value, and ω is the constant factor.

[0073] In steps S21-S22, the estimated occupied memory resource value is determined according to the data processing amount of the data set and the preset data calculation method to calculate the actual memory resource value required to be occupied. The current available memory resource value is obtained, and the preset threshold is determined according to the estimated occupied memory resource value and the available memory resource value to determine a suitable threshold so as to process the data set more flexibly and improve processing efficiency.

[0074] Optional, such as Figure 6 As shown, step S40 also includes:

[0075] S407, determining a number of candidate setting values ​​of the preset parameters according to the data processing amount, wherein the preset parameters are used to set the memory occupied by the distance matrix of the data set;

[0076] S408, receiving a preset parameter modification instruction to select a specified setting value from the plurality of setting values ​​to be selected;

[0077] S409: modify the value of the preset parameter of the preset data calculation method to the specified setting value according to the preset parameter modification instruction.

[0078] In this embodiment, different data processing amounts can be associated with multiple candidate setting values. If the data processing amount is high, the candidate setting value associated with the preset parameter is large, and if the data processing amount is low, the candidate setting value associated with the preset parameter is small. Several candidate setting values ​​that match the current data processing amount can be automatically recommended, and then the user determines the best candidate setting value, which is the designated setting value.

[0079] The user can set the memory occupied by the distance matrix of the data set by entering the preset parameter modification instruction and determine the specified setting value. The preset data calculation method needs to calculate the distance matrix of the data set in the process of calculating the silhouette coefficient of the data set. Taking the calculation of the preset data calculation method in Python as an example, the pairwise_distances_chunked function in the algorithm package sklearn.metrics.pairwise can be loaded and used to calculate the distance matrix of the data set. When the value of the preset parameter is not modified, the default maximum memory setting value (that is, the value of the preset parameter) of the pairwise_distances_chunked function for the distance matrix is ​​1GB. However, for data sets with a data processing volume greater than the preset threshold, the memory space required is much greater than the default maximum memory setting value. Therefore, when the pairwise_distances_chunked function calculates the distance matrix of the data set, it needs to divide the distance matrix into several small sub-matrices. Such processing increases the calculation amount of the distance matrix and increases the calculation time. By modifying the setting value of the pairwise_distances_chunked function, the memory occupied by the distance matrix can be reduced, and at the same time, the data processing volume can be reduced and the calculation speed can be improved.

[0080] In one embodiment, the maximum available memory of the distance matrix is ​​modified, and the calculation time of the distance matrix before and after the improvement is measured, as shown in Table 3.

[0081] Table 3 Calculation time before and after adjusting the maximum available memory of the distance matrix

[0082]

[0083] It can be seen from Table 3 that by modifying the maximum available memory of the distance matrix, the calculation time can be greatly reduced and the calculation efficiency can be improved.

[0084] In steps S407-S409, several candidate setting values ​​of the preset parameters are determined according to the data processing amount, and the preset parameters are used to set the memory occupied by the distance matrix of the data set to match the multiple candidate setting values, so that the user can select the final setting value (i.e., the specified setting value) therefrom. A preset parameter modification instruction is received to select the specified setting value from the several candidate setting values ​​to obtain the user's selection information. According to the preset parameter modification instruction, the value of the preset parameter of the preset data calculation method is modified to the specified setting value to complete the modification of the setting value of the preset parameter.

[0085] Optional, such as Figure 7 As shown, before step S20, it also includes:

[0086] S201, obtaining the sample quantity and sample dimension of the data set;

[0087] S202: Determine the data processing amount of the data set according to the sample quantity and sample dimension of the data set.

[0088] In this embodiment, the data processing capacity of the data set is mainly affected by two aspects, one is the number of samples in the data set, and the other is the sample dimension. The data processing capacity of the data set is positively correlated with the number of samples and the sample dimension. In the actual calculation process, the number of samples will directly affect the memory space occupied by the data set and the memory space occupied by the distance matrix. The sample dimension will have a certain impact on the memory space occupied by the data set, but has a smaller impact on the memory occupied by the distance matrix. If the sample number and sample dimension of the sample set are the same, but the preset data calculation method is different, the data processing capacity may also be different.

[0089] In steps S201-S202, the number of samples and the sample dimension of the data set are obtained to extract key parameters for evaluating the data processing volume of the data set. The data processing volume of the data set is determined according to the number of samples and the sample dimension of the data set, so as to determine whether it is necessary to adjust the preset data calculation method (i.e., the second preset memory optimization mechanism) or modify the data type of the data set sample (i.e., the first preset memory optimization mechanism) according to the data processing volume.

[0090] Optionally, the preset data calculation method is used to calculate the silhouette coefficient of the clustering result of the data set.

[0091] In this embodiment, specifically, the preset data calculation method may be the silhouette coefficient for calculating the clustering result of a data set. The silhouette coefficient combines the cohesion and separation of clustering to evaluate the clustering effect. The value range of the silhouette coefficient is [-1, 1]. The larger the value, the better the clustering effect. The specific definition is as follows:

[0092] Formula:

[0093] Among them, b(i) represents the separation degree, which is used to quantify the average distance from sample i to the samples between clusters; a(i) represents the cohesion degree, which is used to quantify the average distance from sample i to the samples within the cluster; the mean value of s(i) of all samples becomes the silhouette coefficient of the clustering result. When a(i) < b(i), When a(i) = b(i), s(i) = 0; when a(i) > b(i), Therefore, the value range of the silhouette coefficient is [-1, 1]. The larger the value, the better the clustering effect.

[0094] The silhouette coefficient is used to evaluate the effect of unsupervised clustering algorithms and determine the number of clusters in the clustering process. Since the silhouette coefficient has a time complexity of O(n 2 ), therefore, when calculating the silhouette coefficient, it will occupy a large amount of memory resources. When the memory resources of the hardware device cannot meet the calculation requirements of the silhouette coefficient, a memory error will occur, resulting in the failure of the silhouette coefficient calculation.

[0095] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0096] In one embodiment, a memory data optimization device is provided. This memory data optimization device corresponds one-to-one with the memory data optimization method in the above embodiment. As Figure 8 shown, this memory data optimization device includes an acquisition module 10, a judgment module 20, an acquisition mechanism module 30, a calculation optimization module 40, and an acquisition result module 50. The detailed description of each functional module is as follows:

[0097] The acquisition module 10 is used to acquire the data set to be processed, and the data set is used for clustering analysis;

[0098] The judgment module 20 is used to judge whether the data processing amount of the data set is greater than a preset threshold;

[0099] An acquisition mechanism module 30, configured to acquire a first preset memory optimization mechanism and / or a second preset memory optimization mechanism if the data processing amount of the data set is greater than a preset threshold;

[0100] A computing optimization module 40, configured to modify the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modify the memory recycling mechanism and / or preset parameters of a preset data computing method according to the second preset memory optimization mechanism, wherein the preset data computing method includes a plurality of cyclic computing steps;

[0101] The result acquisition module 50 is used to process the data set of the modified data type according to the preset data calculation method to obtain the calculation result; or, to process the data set according to the preset data calculation method with the modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, to process the data set of the modified data type according to the preset data calculation method with the modified memory recovery mechanism and\or preset parameters to obtain the calculation result.

[0102] Optionally, the calculation optimization module 40 includes:

[0103] A data type matching unit, used for determining a plurality of data types to be selected according to the data processing amount;

[0104] A data type selection unit, configured to receive a data type modification instruction to select a specified data type from the plurality of data types to be selected;

[0105] A data type modification unit is used to modify the preset data type of the samples in the data set to the specified data type according to the data type modification instruction, and the data length of the specified data type is shorter than the preset data type.

[0106] Optionally, the calculation optimization module 40 includes:

[0107] A loading process unit, used to load a memory detection process adapted to the preset data calculation method;

[0108] A memory detection unit, configured to detect memory leaks generated by the cyclic calculation step through the memory detection process when the cyclic calculation step in the preset data calculation method is executed;

[0109] The memory recovery unit is used to recover the memory space occupied by the memory leak through a memory recovery process.

[0110] Optionally, the judging module 20 includes:

[0111] Determine a memory resource unit, used to determine an estimated occupied memory resource value according to the data processing amount of the data set and the preset data calculation method;

[0112] A preset threshold determination unit is used to obtain a current available memory resource value and determine the preset threshold value according to the estimated occupied memory resource value and the available memory resource value.

[0113] Optionally, the calculation optimization module 40 includes:

[0114] A setting value matching unit, used to determine a plurality of candidate setting values ​​of the preset parameters according to the data processing amount, wherein the preset parameters are used to set the occupied memory of the distance matrix of the data set;

[0115] A setting value selection unit, used for receiving a preset parameter modification instruction to select a specified setting value from the plurality of setting values ​​to be selected;

[0116] A parameter modification unit is used to modify the value of the preset parameter of the preset data calculation method to the specified setting value according to the preset parameter modification instruction.

[0117] Optionally, the judging module 20 further includes:

[0118] A data parameter acquisition unit, used to acquire the sample quantity and sample dimension of the data set;

[0119] The data processing volume determining unit is used to determine the data processing volume of the data set according to the sample quantity and sample dimension of the data set.

[0120] For the specific limitations of the memory data optimization device, please refer to the limitations of the memory data optimization method above, which will not be repeated here. Each module in the above-mentioned memory data optimization device can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0121] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the memory data optimization method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a memory data optimization method is implemented.

[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0123] Acquire a data set to be processed, where the data set is used for cluster analysis;

[0124] Determining whether the data processing amount of the data set is greater than a preset threshold;

[0125] If the data processing amount of the data set is greater than a preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism;

[0126] Modifying the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modifying the memory recycling mechanism and / or preset parameters of a preset data calculation method according to the second preset memory optimization mechanism, wherein the preset data calculation method includes a plurality of cyclic calculation steps;

[0127] The data set of the modified data type is processed according to the preset data calculation method to obtain a calculation result; or, the data set is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, the data set of the modified data type is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result.

[0128] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0129] Acquire a data set to be processed, where the data set is used for cluster analysis;

[0130] Determining whether the data processing amount of the data set is greater than a preset threshold;

[0131] If the data processing amount of the data set is greater than a preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism;

[0132] Modifying the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modifying the memory recycling mechanism and / or preset parameters of a preset data calculation method according to the second preset memory optimization mechanism, wherein the preset data calculation method includes a plurality of cyclic calculation steps;

[0133] The data set of the modified data type is processed according to the preset data calculation method to obtain a calculation result; or, the data set is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result; or, the data set of the modified data type is processed according to the preset data calculation method with a modified memory recovery mechanism and\or preset parameters to obtain the calculation result.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0135] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0136] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A memory data optimization method, characterized in that: include: Acquire a data set to be processed, where the data set is used for cluster analysis; Determining whether the data processing amount of the data set is greater than a preset threshold; If the data processing amount of the data set is greater than a preset threshold, obtaining a first preset memory optimization mechanism and / or a second preset memory optimization mechanism; Modifying the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modifying the memory recycling mechanism and / or preset parameters of a preset data calculation method according to the second preset memory optimization mechanism, wherein the preset data calculation method includes a plurality of cyclic calculation steps; Processing the data set of the modified data type according to the preset data calculation method to obtain a calculation result; or, processing the data set according to the preset data calculation method with modified memory recycling mechanism and / or preset parameters to obtain the calculation result; or, processing the data set of the modified data type according to the modified memory recycling mechanism and\or the preset data calculation method of the preset parameters to obtain the calculation result; Wherein, before determining whether the data processing amount of the data set is greater than a preset threshold, the method further includes: Determine an estimated occupied memory resource value according to the data processing amount of the data set and the preset data calculation method; Obtaining a current available memory resource value, and determining the preset threshold value according to the estimated occupied memory resource value and the available memory resource value; The preset threshold is calculated by a preset formula, and the preset formula is: Where T is the preset threshold, x is the data processing amount of the data set, is the available memory resource value, To estimate the memory resource usage, ω is a constant factor.

2. The memory data optimization method according to claim 1, characterized in that: The modifying the data types of all samples in the data set according to the first preset memory optimization mechanism includes: Determine a number of data types to be selected according to the data processing amount; receiving a data type modification instruction to select a specified data type from the plurality of data types to be selected; According to the data type modification instruction, the preset data type of the samples in the data set is modified to the specified data type, and the data length of the specified data type is shorter than the preset data type.

3. The memory data optimization method according to claim 1, characterized in that: The step of modifying the memory recycling mechanism and / or preset parameters of the preset data calculation method according to the second preset memory optimization mechanism includes: Loading a memory detection process adapted to the preset data calculation method; When executing the loop calculation step in the preset data calculation method, detecting a memory leak generated by the loop calculation step through the memory detection process; The memory space occupied by the memory leak is reclaimed through a memory reclaim process.

4. The memory data optimization method according to claim 1, characterized in that: The step of modifying the memory recycling mechanism and / or preset parameters of the preset data calculation method according to the second preset memory optimization mechanism includes: Determining a plurality of candidate setting values ​​of the preset parameters according to the data processing amount, wherein the preset parameters are used to set the occupied memory of the distance matrix of the data set; receiving a preset parameter modification instruction to select a specified setting value from a plurality of candidate setting values ​​of the preset parameter; The value of the preset parameter of the preset data calculation method is modified to the specified setting value according to the preset parameter modification instruction.

5. The memory data optimization method according to claim 1, characterized in that: Before determining whether the data processing amount of the data set is greater than a preset threshold, the method further includes: Obtaining the sample quantity and sample dimension of the data set; The data processing amount of the data set is determined according to the sample quantity and sample dimension of the data set.

6. A memory data optimization device, characterized in that: include: An acquisition module, used for acquiring a data set to be processed, wherein the data set is used for cluster analysis; A judgment module, used to judge whether the data processing amount of the data set is greater than a preset threshold; An acquisition mechanism module, configured to acquire a first preset memory optimization mechanism and / or a second preset memory optimization mechanism if the data processing amount of the data set is greater than a preset threshold; A computing optimization module, configured to modify the data types of all samples in the data set according to the first preset memory optimization mechanism, and / or modify the memory recycling mechanism and / or preset parameters of a preset data computing method according to the second preset memory optimization mechanism, wherein the preset data computing method includes a plurality of cyclic computing steps; A result acquisition module, used for processing the data set of the modified data type according to the preset data calculation method to obtain a calculation result; or, processing the data set according to the preset data calculation method with modified memory recycling mechanism and / or preset parameters to obtain the calculation result; or, processing the data set of the modified data type according to the modified memory recycling mechanism and\or the preset data calculation method of the preset parameters to obtain the calculation result; The judgment module is further used to determine the estimated occupied memory resource value according to the data processing amount of the data set and the preset data calculation method; Obtaining a current available memory resource value, and determining the preset threshold value according to the estimated occupied memory resource value and the available memory resource value; The preset threshold is calculated by a preset formula, and the preset formula is: Where T is the preset threshold, x is the data processing amount of the data set, is the available memory resource value, To estimate the memory resource usage, ω is a constant factor.

7. The memory data optimization device according to claim 6, characterized in that: The calculation optimization module includes: A data type matching unit, used for determining a plurality of data types to be selected according to the data processing amount; A data type selection unit, used for receiving a data type modification instruction to select a specified data type from the plurality of data types to be selected; A data type modification unit is used to modify the preset data type of the samples in the data set to the specified data type according to the data type modification instruction, and the data length of the specified data type is shorter than the preset data type.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the memory data optimization method as described in any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the memory data optimization method as described in any one of claims 1 to 5 is implemented.

Images